Limitations of HTTP Sender using HTTPS URLs

[tom08zehn]

I haven't worked with public endpoints for a while and now I need it.

I have an HTTP Sender destination and need to send to an https:// URL but Mirth throws the following error and queues all messages:

QUEUED: Error connecting to HTTP server [SSLHandshakeException: Received fatal alert: handshake_failure]

Did I forget about a limitation sending to HTTPS endpoints? I thought it should always work but cannot validate SSL certificates... (which is still useful because traffic is encrypted, even if a man-in-the-middle attack cannot be detected). I thought the limitation is that Mirth cannot serve HTTPS as a source, unless you have paid for the SSL manager plugin/extension.

If I'm wrong and Mirth cannot handle HTTPS URLs (without the plugin), then what's a workaround? I only need it for testing, not in production. Would this still work? https://forums.mirthproject.io/forum/mirth-connect/support/14439-http-sender-error?p=84210#post84210

[rjkroll]

Here is a little more info. HTTP Sender via Mirth will work with HTTPS if several conditions are met, otherwise you might have to use a 3rd Party application.

https://gist.github.com/jonbartels/8abd121901eb930f46245d9ef0f5710e

[tom08zehn]

I don't have Java installed separately on my Windows, that's why I'm confused 🤔 maybe the Windows Mirth .exe binaries have Java baked in?

I also checked all my system environment variables and cannot find any Java related entries.

The admin launcher asks during installation what Java VM to install but the Mirth installer does not.

[pacmano1]

You have other tools that use java that might have installed a JRE and pathed set envs for you? A browser installed JRE?

run java -version in a command prompt, check your env vars also

The server installer process will do this if you have no JRE, it does not install one for you.

[tom08zehn]

According to this link the Mirth keystore is used by HTTP destinations (in my case) and Java's keystore is used when invoking HTTP connections using JavaScript code. So I guess I don't need to check Java's keystore...

1. But still, if I wanted to, how do I know what Java RE is used by Mirth under Windows after installation (registry or file)? In my case, the installer did not ask me for the location so it seems like it found it somehow on his own.
2. Do you recommend importing the SSL cert chain into Mirth's default keystore or creating a new one?
3. In my case, the chain has 3 certs: 1 root CA, 1 intermediate CA and the client cert (wildcard). Should I export them one by one (3 files) or as one file including the whole chain?

[pacmano1]

Is this mutual TLS? You mentioned a client cert. That is different than the "client's wildcard server cert"

[ab-mg-23]

I my experience, the most common reason for handshakes to fail is either: 1) No agreed upon SSL/TLS version between the server and the client or 2) No agreed upon cipher suite between the client and the server. Running a network packet capture and comparing a successful connection versus the failed connection should give you more insight on what is happening and where in the handshake the alert is happening.

[tom08zehn]

Hi, I have to continue this discussion because there's still some things that I don't fully understand. I hope you can help me out?

I learned - and I invite you to verify and confirm:

• Java usually uses a keystore and a truststore
• Mirth only uses a keystore and by default uses the Java truststore (link)
• The keystore is only used to hold Mirth's own server cert (private + public key pair) which it presents to clients when they connect to it, e.g. for Admin Launcher, Admin Website and the API -> To my understanding, this is not involved at all when doing outbound HTTPS connections?
• The truststore is used when Mirth connects to third party services and holds CA certs to trust third parties (link)

What works: HTTP Sender connecting to https://www.google.com

What does not work:

• HTTP Sender connecting to our partner service, I cannot publish the URL, so let's name it https://abc.xyz.com
• The CA who signed and issued the SSL webserver cert for abc.xyz.com is already in Java's truststore
• Mirth throws SSLHandshakeException: Received fatal alert: handshake_failure

A difference I noticed: abc.xyz.com uses a wildcard cert that is issued to *.xyz.com

#1 What I did:

• Visited https://abc.xyz.com in Edge/Chrome and both show a valid (green) cert
• Displayed the cert details and downloaded the whole chain (root 1 CA + root 2 [or intermediate?] CA + webserver cert)
• Imported the 2 CA certs to Java's truststore

Result: still same handshake_failure

#2 Then I did: Imported the 2 CA certs + webserver cert to Mirth's keystore
Result: still same handshake_failure

#3 Then I did: Added the following two lines to mcservice.vmoptions (link 1) (link 2) (link 3):

-Djavax.net.ssl.trustStore=NUL
-Djavax.net.ssl.trustStoreType=Windows-ROOT

Same as for Java truststore: The CAs already existed in Windows' CA store but I re-imported the 2 CA certs again.

Result: still same handshake_failure

#4 Then I did:

• Reverted all changes from 1-3
• Switched to JavaScript Writer, using the following code:

var url = "https://abc.xyz.com";
var headers = {};
var body = connectorMessage.getEncodedData();
var charsetEncoding = "UTF-8";

var httpClient = new org.apache.http.impl.client.DefaultHttpClient();
var httpRequest = org.apache.http.client.methods.HttpPost(url);

// Optional: add headers
Object.keys(headers).forEach(function (k) {
	httpRequest.addHeader(k, headers[k]);
});

// Add body (payload)
var entity = new org.apache.http.entity.StringEntity(body, charsetEncoding);
httpRequest.setEntity(entity);

// HTTP response
var httpResponse = httpClient.execute(httpRequest);

// Get status code and response as string
var statusCode = httpResponse.getStatusLine().getStatusCode();
var responseString = org.apache.http.util.EntityUtils.toString(httpResponse.getEntity(), charsetEncoding);

// Close HTTP connection
httpResponse.close();

logger.debug("statusCode: " + statusCode);
logger.debug("responseString: " + responseString);

Result: Works! But why?!?

[tom08zehn]

I found out that I imported the certs wrong. Here's what I did:

1. Installed "KeyStore Explorer" (v5.5) -> run
2. Quick Start -> Open the CA Certificates KeyStore
3. Imported trusted root CAs
4. Saved

The problem is that "KeyStore Explorer" comes with its own JRE and therefore has its own cacerts truststore. You can see it when you click Tools -> KeyStore Properties; the first line gives you File: File: C:\Users\<username>\AppData\Local\Programs\KeyStore Explorer\jre\lib\security\cacerts, which is not the one that Mirth uses.

To find out what JRE/JVM Mirth uses, I found the file C:\Program Files\Mirth Connect\.install4j\inst_jre.cfg. Its content is c:\program files\mirth connect administrator launcher\jre. So I guess Mirth uses C:\Program Files\Mirth Connect Administrator Launcher\jre\lib\security\cacerts as truststore. Can you please confirm?

So I opened C:\Program Files\Mirth Connect Administrator Launcher\jre\lib\security\cacerts in "KeyStore Explorer" (no password). Nice feature: Menu -> Examine -> Examine SSL. Enter the SSL host and port and it opens a window showing the certificate chain:

From there, you can easily import the root CAs. I also installed the server CA but it didn't help.

With and without Mirth restart, it's still the same: handshake_failure

[tom08zehn]

Next, I had the following idea:

1. Connect to https://www.google.com using Mirth's native HTTP Sender -> works
2. Remove root CA "GTS Root R1" from C:\Program Files\Mirth Connect Administrator Launcher\jre\lib\security\cacerts and retry -> still works

3. Restart Mirth -> still works. But Why?

4. I also wanted to remove the intermediate CA "WR2" from cacerts but couldn't find it. Shouldn't this already lead to some TLS error?

Maybe I'm still editing the wrong cacerts? How can I find out what truststore is used by Mirth... ?

[pacmano1]

Mirth should be using the operating systems native Java install. That would not be the jave under the mirth administrator launcher program files folder nor any cache folder.

You are still in the wrong cacerts file I suspect.

[tom08zehn]

I don't have an OS native Java installation. I got a fresh Windows VM without Java being installed and installed Mirth and the Admin Launcher and it "just worked".

As a result, I don't know what JRE/JVM Mirth is using. How do I find out?

[pacmano1]

Please start with a clean windows machine.

Install Amazon Coretto release of Java.

Install Mirth.

Do NOT install the Mirth administrator launcher. Use it on your local workstation, there's no need to have on the server.

The cacerts file you will need to edit will be under the program files directory for Coretto on the new machine.

I'm out on this thread. Good luck.

[tom08zehn]

Unfortunately, I cannot simply install everything from scratch because my Windows VM is a jump host to a 3rd party and the only way to access and troubleshoot its HTTPS interface because my VM is connected to their network via S2S VPN and my IP address is whitelisted.

But the principle should be the same - no matter what Java release and where it's installation path is. Correct?

When I run the following JS code in Mirth:

logger.debug("Java version: " + java.lang.System.getProperty("java.version"));
logger.debug("Java home: " + java.lang.System.getProperty("java.home"));

And the java.home path it shows is c:\program files\mirth connect administrator launcher\jre

[ab-mg-23]

Multiple issues are getting mixed that aren't directly related.

If there is an issue with validating a server certificate to a trust anchor 2 things would show in your WireShark capture:

1. ClientHello is sent to the server
2. Server Response with ServerHello
3. Client sends tls_alert:handshake_failure and closes the connection.

Based on what you have posted with the Wireshark capture the scenario is:

1. ClientHello is sent to the server
2. Server sends tls_alter:handshake_failure because of 1 of 2 possibilities: The client is requesting a TLS version that is outside the versions the server supports or of the cipher suites it sent in the ClientHello, the server does not support any of those cipher suites.

[tom08zehn]

Here's a Wireshark capture. Do you see anything suspicious?

To me it seems like the client presented 37 cipher suites, incl. Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d) and the server responded with OK, let's use "Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)" -> so all good from that end.

Client Hello:

Server Hello:

Client Key Exchange:

Change Cipher Spec, Encrypted Handshake Message:

Change Cipher Spec:

Encrypted Handshake Message:

Application Data (several):

Encrypted Alert:

[tom08zehn]

And this is how it looks like when I'm using Mirth's native HTTP Sender instead of JavaScript Writer:

Client Hello:

Alert:

One difference I see is that the HTTP Sender only offers 17 cipher suites.

[ab-mg-23]

@pacmano1 turns out you were in fact correct, the https.ciphers and https.client.protocols in mirth.properties feed into the configuration of the default HttpDispatcher.

connect/server/src/com/mirth/connect/connectors/http/DefaultHttpConfiguration.java

Lines 62 to 67 in bbeea45

	public void configureSocketFactoryRegistry(ConnectorPluginProperties properties, RegistryBuilder<ConnectionSocketFactory> registry) throws Exception {
	String[] enabledProtocols = MirthSSLUtil.getEnabledHttpsProtocols(configurationController.getHttpsClientProtocols());
	String[] enabledCipherSuites = MirthSSLUtil.getEnabledHttpsCipherSuites(configurationController.getHttpsCipherSuites());
	SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(SSLContexts.createSystemDefault(), enabledProtocols, enabledCipherSuites, NoopHostnameVerifier.INSTANCE);
	registry.register("https", sslConnectionSocketFactory);
	}

The current defaults should all be ciphers that support PFS. @tom08zehn you'll need to add TLS_RSA_WITH_AES_256_CBC_SHA256 to the property and bounce Mirth

[rjkroll]

So this fixes it? Can anyone summarize, then, the correct steps needed to get this to work in the end ? Is is just adding this to the config file and importing the cert in the correct cacerts store?

[pacmano1]

@rjkroll it's been the same answer as it always has been.

1. Understand the endpoint TLS settings (use a tool like testssl already mentioned)
2. adjust mirth.properties as needed (although I have never had to do this)
3. Understand what jde/jdk mirth is using so you know what cacerts file to fix.
4. Attempt connection, if it works cacerts already has the CA cert chain
5. If it does not work import the CA cert chain to cacerts.

[tom08zehn]

So this fixes it? Can anyone summarize, then, the correct steps needed to get this to work in the end ? Is is just adding this to the config file and importing the cert in the correct cacerts store?

1. Edit mirth.properties
2. Add TLS_RSA_WITH_AES_256_CBC_SHA256 to https.ciphersuites = ...
3. Restart Mirth
4. Send HTTPS call using native HTTP Sender

[tom08zehn]

Also see: https://knowledge.digicert.com/general-information/ending-support-for-cbc-ciphers-in-tls-connections-to-our-services

CBC cipher suites are not considered as "strong". Maybe this is the reason why Mirth doesn't support it by default and you have to enable it manually?

@ab-mg-23 How did you know I should add TLS_RSA_WITH_AES_256_CBC_SHA256 - simply because it works with my old JavaScript code using org.apache.http.impl.client.DefaultHttpClient() (above)?

Is there a better answer? I would expect a "Server Hello" before the error occurs to see what cipher suites the server supports, but it didn't happen and the connection just died with a handshake failure.

A problem is that every Mirth installation/version comes with different defaults - so the same connection might work here but not there. This makes it very inconsistent.

@pacmano1 This is really hard to debug... not everybody is an TLS expert - so I appreciate that you still kept an eye on this thread.