Web service receiver http status 500 error - javax.xml.soap.SOAPException: Unable to create SAAJ meta-factory

[jhimsworth]

I'm having a problem porting a channel using a web service receiver source to a new server and moving from MC3.2 to 4.4.2. The request (from a Mirth web service sender on a different box) reaches the new server - I see a 'javax.xml.soap.SOAPException: Unable to create SAAJ meta-factory' in the receiver dashboard and logs, but no message is ever seen in the receiver channel dashboard. The sending server sees a http status 500 error (ie. it doesn't appear to be a connectivity issue). It looks like missing java classes, but everything is pretty standard so I'm not sure what.

Original is running Mirth Connect 3.2, java 1.8.0_25 on windows server 2012.
New server is running MC 4.4.2, java 21.0 (JDK not JRE, although 1.8.0_351 is installed), windows server 2019
Web service listener is using the default class com.mirth.connect.connectors.ws.DefaultAcceptMessage

I also encountered this bug, which seems to be a related jax-ws issue: custom-lib folder needs to be empty for web service listener channel to start

There is another thread on this github 5761, but no solution, over a year since replies and they're using openjdk.

I've tried adding saaj-impl.jar as a custom lib as suggested here: https://www.studytonight.com/post/solved-unable-to-create-saaj-metafactory-provider-saajmetafactoryimpl-not-found but no difference - I thought it'd at least give me a different missing dependency, but no difference so not sure that's having any effect.

I've also checked that the java 9+ config is applied to mcserver/service.vmoptions - all done, but I wondered if a new entry along those lines might get jax-ws working in java 9+ ?

Any suggestions for what to try next greatly appreciated.

Full exception is:
ERROR 2024-05-16 13:46:37.064 [pool-101-thread-1] com.sun.xml.ws.transport.http.HttpAdapter: javax.xml.soap.SOAPException: Unable to create SAAJ meta-factory: Provider com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl not found java.lang.Error: javax.xml.soap.SOAPException: Unable to create SAAJ meta-factory: Provider com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl not found at com.sun.xml.ws.api.SOAPVersion.getMessageFactory(SOAPVersion.java:223) ~[?:?] at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSOAPMessage(SAAJFactory.java:292) ~[?:?] at com.sun.xml.ws.api.message.saaj.SAAJFactory.readAsSAAJ(SAAJFactory.java:222) ~[?:?] at com.sun.xml.ws.api.message.saaj.SAAJFactory.read(SAAJFactory.java:211) ~[?:?] at com.sun.xml.ws.message.AbstractMessageImpl.toSAAJ(AbstractMessageImpl.java:236) ~[?:?] at com.sun.xml.ws.api.message.MessageWrapper.readAsSOAPMessage(MessageWrapper.java:176) ~[?:?] at com.sun.xml.ws.handler.SOAPMessageContextImpl.getMessage(SOAPMessageContextImpl.java:85) ~[?:?] at com.mirth.connect.connectors.ws.LoggingSOAPHandler.handleMessage(LoggingSOAPHandler.java:63) ~[?:?] at com.mirth.connect.connectors.ws.LoggingSOAPHandler.handleMessage(LoggingSOAPHandler.java:30) ~[?:?] at com.sun.xml.ws.handler.HandlerProcessor.callHandleMessage(HandlerProcessor.java:297) ~[?:?] at com.sun.xml.ws.handler.HandlerProcessor.callHandlersRequest(HandlerProcessor.java:140) ~[?:?] at com.sun.xml.ws.handler.ServerSOAPHandlerTube.callHandlersOnRequest(ServerSOAPHandlerTube.java:138) ~[?:?] at com.sun.xml.ws.handler.HandlerTube.processRequest(HandlerTube.java:127) ~[?:?] at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1136) ~[?:?] at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:1050) ~[?:?] at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:1019) ~[?:?] at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:877) ~[?:?] at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:419) ~[?:?] at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:868) ~[?:?] at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:422) ~[?:?] at com.sun.xml.ws.transport.http.server.WSHttpHandler.handleExchange(WSHttpHandler.java:113) ~[?:?] at com.sun.xml.ws.transport.http.server.WSHttpHandler.handle(WSHttpHandler.java:97) ~[?:?] at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98) ~[jdk.httpserver:?] at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82) ~[jdk.httpserver:?] at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:101) ~[jdk.httpserver:?] at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:871) ~[jdk.httpserver:?] at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98) ~[jdk.httpserver:?] at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:847) ~[jdk.httpserver:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1583) ~[?:?] Caused by: javax.xml.soap.SOAPException: Unable to create SAAJ meta-factory: Provider com.sun.xml.internal.messaging.saaj.soap.SAAJMetaFactoryImpl not found at javax.xml.soap.SAAJMetaFactory.getInstance(SAAJMetaFactory.java:112) ~[?:?] at javax.xml.soap.MessageFactory.newInstance(MessageFactory.java:150) ~[?:?] at com.sun.xml.ws.api.message.saaj.SAAJFactory.createMessageFactory(SAAJFactory.java:250) ~[?:?] at com.sun.xml.ws.api.message.saaj.SAAJFactory.getMessageFactory(SAAJFactory.java:103) ~[?:?] at com.sun.xml.ws.api.SOAPVersion.getMessageFactory(SOAPVersion.java:221) ~[?:?] ... 30 more

[jonbartels]

This issue sounds familiar. I searched and you already found the hits on GitHub. I did not find any hits on the forums (google for site:forums.mirthproject.io "Unable to create SAAJ") but I didn't spend too long playing with search terms.

I posted to Slack to give some visibility - https://mirthconnect.slack.com/archives/C02SW0K4D/p1716298090048329

[pacmano1]

How about sharing the channel exports? or enough of an export to replicate the issue.

[jhimsworth]

Good point, added in my comment below, thanks.

[jhimsworth]

Thanks. Sample channels to replicate this here: web service receiver issue channel extracts.zip
Test sender channel sends a fixed xml, EPR inbound kiosk is the receiver channel throwing the error. I've cut these down lots to remove code templates and database queries, but issue still replicating, including when run from the same server.

[pacmano1]

https://repo1.maven.org/maven2/javax/xml/ws/jaxws-api/2.3.1/jaxws-api-2.3.1.jar

I had to add this to even start your listener channel (as a custom resource). After that seems OK on Mirth 4.5.

[pacmano1]

Stop using Windows? Sorry, had to.

[jhimsworth]

I think that is a long shot, to put it mildly. I might suggest it though, just to see the reaction...
I see there is a Microsoft build of openjdk though, that might be an easier sell.

[pacmano1]

Ah, so what I tested this on is an instance with quite a few of the platinum plug-ins. The server-lib javax dir has a bunch of libraries not on a clean install which I tried as well and could not get to work.

[pacmano1]

To be precise:

root@mirth45:/usr/local/mirthconnect/server-lib/javax# tree
.
|-- javax.activation-1.2.0.jar
|-- javax.activation-api-1.2.0.jar
|-- javax.annotation-api-1.3.2.jar
|-- javax.inject-2.4.0-b31.jar
|-- javax.json-1.0.4.jar
|-- javax.json-api-1.0.jar
|-- javax.mail-1.5.0.jar
|-- javax.servlet-api-3.1.0.jar
|-- javax.ws.rs-api-2.0.1.jar
|-- jaxb
|   |-- ext
|   |   |-- istack-commons-runtime-3.0.6.jar
|   |   `-- txw2-2.4.0-b180725.0644.jar
|   |-- jaxb-api-2.4.0-b180725.0427.jar
|   `-- jaxb-runtime-2.4.0-b180725.0644.jar
`-- jaxws
    |-- ext
    |   |-- FastInfoset-1.2.13.jar
    |   |-- gmbal-api-only-3.1.0.b001.jar
    |   |-- ha-api-3.1.9.jar
    |   |-- jsr181-api-1.0.jar
    |   |-- management-api-3.2.1.b001.jar
    |   |-- mimepull-1.9.7.jar
    |   |-- policy-2.7.2.jar
    |   |-- saaj-impl-1.0.jar
    |   |-- stax-ex-1.8.jar
    |   `-- streambuffer-1.5.4.jar
    |-- javax.xml.soap-api-1.4.0.jar
    |-- jaxws-api-2.3.0.jar
    |-- jaxws-rt-2.3.0.2.jar
    `-- jaxws-tools-2.3.0.2.jar

[jhimsworth]

Snap. Well, not quite, only difference is I've a slightly older version of javax.annotation-api-1.3.<0/2>.jar and an additional woodstox-core. But I've tried adding the newer version of annotation-api with no effect, and I don't imagine an extra jar on mine is the culprit.

We do have a platinum license I think, plenty of plugins anyway but happy to try any extras that might be related.

C:\Program Files\Mirth Connect\server-lib\javax>tree /f
Folder PATH listing
Volume serial number is 728A-E04F
C:.
│   javax.activation-1.2.0.jar
│   javax.activation-api-1.2.0.jar
│   javax.annotation-api-1.3.jar
│   javax.inject-2.4.0-b31.jar
│   javax.json-1.0.4.jar
│   javax.json-api-1.0.jar
│   javax.mail-1.5.0.jar
│   javax.servlet-api-3.1.0.jar
│   javax.ws.rs-api-2.0.1.jar
├───jaxb
│   │   jaxb-api-2.4.0-b180725.0427.jar
│   │   jaxb-runtime-2.4.0-b180725.0644.jar
│   └───ext
│           istack-commons-runtime-3.0.6.jar
│           txw2-2.4.0-b180725.0644.jar
└───jaxws
    │   javax.xml.soap-api-1.4.0.jar
    │   jaxws-api-2.3.0.jar
    │   jaxws-rt-2.3.0.2.jar
    │   jaxws-tools-2.3.0.2.jar│
    └───ext
            FastInfoset-1.2.13.jar
            gmbal-api-only-3.1.0.b001.jar
            ha-api-3.1.9.jar
            jsr181-api-1.0.jar
            management-api-3.2.1.b001.jar
            mimepull-1.9.7.jar
            policy-2.7.2.jar
            saaj-impl-1.0.jar
            stax-ex-1.8.jar
            stax2-api-3.1.4.jar
            streambuffer-1.5.4.jar
            woodstox-core-5.0.3.jar

[epascuall]

Hello!
I've this same error with Mirth version 4.5.0 and openjdk 21. Any validated solution to this problem?
Thanks, in advance.

[jhimsworth]

Not from me. I've given up on the web service listener and am switching to http listener, but that's obviously no use if you can't get the corresponding changes made on the sending service.

[epascuall]

Thank you, jhimsworth!

[pacmano1]

Well, seems the fix is java 8. Guessing some of the libraries Mirth included break after 8.