-
Notifications
You must be signed in to change notification settings - Fork 12
Wilma Issues and Solutions
This page describes the issues Wilma Users faced with, together with the solution of the specific issue. (Reverse order, newest things are on the top.)
This may happen when the generated stub answer is not well formatted. This is the situation when the loaded template file has improper character encoding. Always use UTF-8 character encoding for stub response template files.
In application log you may see the following error message: org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]
This issue happens when the JSON configuration file format is incorrect. Always use UTF-8 encoded files.
After Wilma image is started, it is not possible to reach its UI. When the instance is restarted by using this command:
docker run --publish 1234 --publish 9092 -p 1234:1234 -p 9092:9092 -t epam/wilma
docker said:
docker: Error response from daemon: driver failed programming external connectivity... ... Bind for 0.0.0.0:9092: unexpected error Permission denied.
The issue was that an other program used one of the ports of Wilma. So in case of similar issue, pls check and ensure that no application/program uses the ports: 9092, 1234 - and any other port that is used by Wilma (see its configuration file). In case you cannot free up the ports, you must re-configure Wilma to use free ports.
In a SUT->Wilma->kickbox.io configuration, connection from SUT was failed with:
path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild
The problem was that kickbox.io uses SSL communication and SUT were unable to connect to. The solution was to get the its certificate, add to jks that is used by Wilma (see how to start Wilma with keyStore information here, at Running Wilma section), and the certification to wilma.jks file too (can be found in certificate folder), that is used as trustStore for SUT (see more information here, at Configuring a Java client application to use Wilma as a proxy section). After restart of SUT and Wilma with the updated keyStore and trustStore, the issue disappeared.
In log I see several “XML Parser does not recognize the feature http://xml.org/sax/features/validation” log entries
This is a warning message of a 3rd party lib, nothing else. See details about this issue here: http://stackoverflow.com/questions/31612351/upgrading-saxon-in-servingxml-results-in-xml-parser-does-not-recognize-the-feat It is not harming, please leave with it.
(In the past) Wilma's proxy had a timeout value that is 60000 msec (= 1 minute). Timeout in Wilma (actually in its browsermob-proxy module) causes the situation when Wilma sends -998 status code back to the originator of the request. Please note the after the release of Wilma-1.1.27, the timeout was increased to 4 minutes, and after Wilma 1.2.x, a new ´proxy.request.timeout´ property was introduced in Wilma configuration file, that defaults to 20 seconds. It is up-to-you now to set the default proxy timeout.
use this kind of bash command in messages folder:
find . -name '*req.txt' -exec head -1 {} + | sort | uniq -c | sort -m | less
It has happened that one of our hostname was set to: blah_svc.staging.com, and Wilma did not let the messages targeting this hostname go through.
The error message in Wilma log was:
INFO n.l.b.p.http.BrowserMobHttpClient - java.lang.IllegalArgumentException: Host name may not be null when requesting http://blah_svc.staging.com/api....
In Wilma, this was caused by an apache library, in class org.apache.http.impl.client.RequestWrapper that was unable to figure out the hostname (=null)
Later it turned out that it is not a Wilma problem, rather the used hostname was not standard. See hostname restrictions and rules here: http://en.wikipedia.org/wiki/Hostname
The solution was to change the hostname to a right one (eliminate underscores).
Wilma use system to determine the exact url meanwhile proxy-ing the request. A request that calls "localhost" cannot be understood, as that host is unknown by the DNS (none of the DNS can determine what is the IP of "localhost". As a result Wilma cannot proxy the request.
Therefore if you need to use "localhost", then use the real IP: "127.0.0.1" or the real ip or real name of the local host.
Various error messages we received suddenly from an application, like:
- Error loading trust store from file ../../wilma.jks
- Failure initializing default system SSL context, Keystore was tampered with, or password was incorrect
That leads us to the conclusion, that some http clients are unable to load/use wilma.jks with empty password, meanwhile other clients has no problem with it at all. For this reason a new wilma.jks file is created with non-empty password (that is: vvilma).
So Wilma right now works with the following user agents:
- User-Agent=JAX-WS RI 2.2.4-b01
- User-Agent=Axis/1.4
- User-Agent Jersey/2.2 (HttpUrlConnection 1.7.0_10)
- SystemDefaultHttpClient
Technically Wilma works with all clients those properly use the http(s).proxyHost and http(s).proxyPort System properties.
On the other hand, Wilma does not work with:
- DefaultHttpClient class - in this case see https://issues.apache.org/jira/browse/HTTPCLIENT-1128, and SystemDefaultHttpClient should be used
After this, the application was able to connect to use Wilma as proxy, still the connection to a service did not work. The reason that the connection would be a 2-way-SSL connection, and the jks that is known by Wilma did not contain the certificate of the service.
Resolution: download the certificate of the target, and add to the jks that is used by wilma.
In sum: all new 2-way-SSL connection that is going to be built into the server apps, the server certificate should be uploaded to the jks given to Wilma, in order to build up the connection.
The current version of a client app uses Axis 1.4 to send and receive SOAP web-service messages. Axis 1.4 ignores the standard proxyHost and proxyPort system overrides, and therefore the outgoing XML requests bypass Wilma when only these overrides are added to the client configuration. To solve this, additional system overrides must be specified:
http.proxyHost=<wilma-host>http.proxyPort=<wilma-port>https.proxyHost=<wilma-host>https.proxyPort=<wilma-port>
A client application does not support the standard javax.net.ssl.trustStore System variable. To use it with Wilma, a different process must be followed:
- The trustStore JKS file is configured in the environment specific application configuration, available here:
$CATALINA_BASE/app_config/conf/environment/env_rules.properties - The JKS_FILE and the JKS_PASSWORD properties of the app should be overwritten, with the wilma.jks file and its password.
- Restart the client app.
There is an issue with Wilma that can occur after Wilma is running for multiple days (actually several weeks), and can cause HTTPS request to fail with a Certification Expired exception:
Using curl
export https_proxy=wilma:9092
curl -kv https://url
* SSL connection using ECDHE-RSA-DES-CBC3-SHA
* Server certificate:
* subject: C=US; ST=....; L=....; O=....; OU=Test; CN=p....
* start date: 2012-12-01 08:57:13 GMT
* expire date: 2013-12-07 04:09:13 GMT
* issuer: O=CyberVillians.com; OU=CyberVillians Certification Authority; C=US
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x80057230; line 1164 (connection #0)
From app log
ERROR com.... 2013-12-07 03:04:31,985 [build=...] [ajp-7510-1] [en_US] [WEB_DESKTOP] Calling external system failed. [exceptionClass=c...., user_agent="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; InfoPath.3; MS-RTC LM 8)", edgescape=""] ..
...ExternalSystemUnavailableException: External system is unavailable: ....
...
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Fri Dec 06 22:09:13 CST 2013
at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:273) ~[na:1.7.0_25]
at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:581) ~[na:1.7.0_25]
at sun.security.provider.certpath.BasicChecker.verifyTimestamp(BasicChecker.java:184) ~[na:1.7.0_25]
at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:136) ~[na:1.7.0_25]
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:133) ~[na:1.7.0_25]
... 351 common frames omitted
Cause of the issue:
A restart of Wilma fixes this issue, but it can come up again after Wilma is running for a few weeks.
There is some kind of mechanism built into Wilma's proxy dependency (BrowserMob) that generates certificates dynamically, to be able to proxy HTTPS messages. It needs some more serious investigation getting to know the operation of this mechanism, and to fix this problem because it is part of an external dependency, not Wilma itself. Possible causes:
- The expiration date timestamp is generated when the application starts, and isn't modified later
- Some kind of caching is built in the proxy that prevents Wilma to generate new certificates for a given page after it was proxied once
- Something totally different
Until this investigation is done a workaround is suggested: In case you use httpS connections via Wilma, restart Wilma periodically (e.g.: from a Jenkins job, every week)