Processing of large messages streamed in through TCP Listener

[snarayanan99]

I have a requirement to process large input messages with sizes ranging from 40MB - 60 MB. The Mirth Connect channel has a TCP Listener in the Source that receives the input message. I set the -Xmx4096m in the mcserver.vmoptions file and transformer heap max to 4096m in the mirth.properties as well. The Mirth Connect Administrator Launcher has JVM size as 4GB. The workflow is throwing an out of Memory exceptions even for much smaller message sizes of around 2MB.
The specific error that I am seeing in the logs is "Transformer error ERROR MESSAGE: Error evaluating transformer
java.lang.OutOfMemoryError: GC overhead limit exceeded""

The transformer has a simple for loop

I would appreciate your valuable inputs in this regard

[pacmano1]

This is with or without the attachment handler (which you should likely be using in this case)

I would not expect out of memory problem though at 2Mb assuming your code is not running out of control.

[jonbartels]

I think you absolutely need to use the attachment handler.

• https://forums.mirthproject.io/articles/175046-using-attachments
• https://docs.nextgen.com/bundle/Mirth_User_Guide_4_5_0/page/connect/connect/topics/c_Attachment_Handlers_connect_ug.html

You're getting out of memory exceptions because:

1. Mirth translates HL7 messages in to XML - XML is memory intensive
2. Mirth creates multiple copies of a message - Source, transformed, encoded, sent will each have a copy of your incoming data

Those are not flaws, they are inherently how Mirth works. They're just fine for small messages but large messages will induce failures. The attachment handler eases this memory pressure by reducing the number of copies of large data blobs (both in memory and in the DB).

What is the nature of the HL7 messages? Are they MDM messages with a large file embedded? Simply very, very large HL7 messages with a lot of data?

Can you show the code that your for loop is using?

[snarayanan99]

Thanks much for your inputs. I increased the heap size settings to 8192 (8MB) in the mcserver.vmoptions and the mirth.properties file (Administrator max heap size) and to 4096 (4MB) in mcservice.vmoptions files. This seems to work and I am able to run the mirth channel without errors for input HL7 messages of size 40 MB. I will look into the attachment option as well. Would increasing the heap size to the limits that I have be recommended in a Production environment?

[jonbartels]

Would increasing the heap size to the limits that I have be recommended in a Production environment?

Yes. In general PROD is likely to need more heap since it will get larger messages, more messages, and weird messages that dev/test will not get.

The best practices are:

• Ensure that PROD is monitored using a tool that natively monitors both OS memory usage and JVM level memory usage. Most monitoring tools have a "java agent" that does this easily. Monitor and measure your memory usage.
• use attachment handlers for large data blobs. It saves memory and DB storage.
• memory is usually cheaper than developer time up to a point, so give it a few more GB but don't keep feeding the beast

[pacmano1]

It's worth re-iterating that using the attachment handler is better. You will end up with only one copy in the mirth database of that attachment. And you can discard that attachment in the channel via pruner settings, so you do not have explosive database growth.

[snarayanan99]

Thanks much for your inputs. I did notice that the Out of Memory exception is occurring mostly in the Source Transformer. There is a huge XML message in the OBX segment of the HL7 message. The OOM error is occurring when the Source transformer is attempting to parse this XML.