Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October #2031

smlambert · 2020-10-27T16:11:14Z

Starting in mid October (around Oct 14th), we see issues running java executable from a mapped drive using the 'nightly' images produced at the project and pushed to Docker hub. This was previously not an issue.

Suspect that something changed in the Docker images as test material and mapped JDK is unchanged when issue is present (so this may get shifted to openjdk-docker repo after additional investigation).

Running Tomee tests using the JDK from within the AdoptOpenJDK Docker image,
https://ci.adoptopenjdk.net/job/Grinder/4315/console you can see that the tests execute and run:

09:19:13  	/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../external/external.sh --clean --tag nightly --version 8 --impl openj9 --dir tomee; } 2>&1 | tee -a "/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16037183519158/TestTargetResult";
09:19:13  docker run --rm  --rm adoptopenjdk-tomee-test:8-jdk-ubuntu-openj9-full 
09:19:13  Using docker image default Java
09:19:13  JAVA_BIN is: /opt/java/openjdk/bin
09:19:13  APPLICATION_NAME=tomee
09:19:13  APPLICATION_TAG=tomee-8.0.2
09:19:13  OS_TAG=ubuntu
09:19:13  =JAVA VERSION OUTPUT BEGIN=
09:19:13  openjdk version "1.8.0_272"
09:19:13  OpenJDK Runtime Environment (build 1.8.0_272-202010231739-b10)
09:19:13  Eclipse OpenJ9 VM (build master-67af03f58, JRE 1.8.0 Linux s390x-64-Bit Compressed References 20201023_805 (JIT enabled, AOT enabled)
09:19:13  OpenJ9   - 67af03f58
09:19:13  OMR      - cb68c30b7
09:19:13  JCL      - bea7d86207 based on jdk8u272-b10)
09:19:13  =JAVA VERSION OUTPUT END=
09:19:13  Build TomEE without running test
09:19:14  [INFO] Scanning for projects...
09:19:14  [INFO] Downloading from apache: https://repository.apache.org/content/repositories/releases/org/apache/apache/21/apache-21.pom
...
...
10:02:12  [INFO] ------------------------------------------------------------------------
10:02:12  [INFO] Reactor Summary for TomEE :: TCK :: MicroProfile 8.0.2:
10:02:12  [INFO] 
10:02:12  [INFO] TomEE :: TCK :: MicroProfile ....................... SUCCESS [  0.375 s]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Config TCK ............ SUCCESS [01:38 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile JWT TCK ............... SUCCESS [01:17 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Fault Tolerance TCK ... SUCCESS [08:59 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Health TCK ............ SUCCESS [ 49.368 s]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Metrics TCK ........... SUCCESS [06:13 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Rest Client TCK ....... SUCCESS [06:54 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Open API TCK .......... SUCCESS [01:56 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Open Tracing TCK ...... SUCCESS [ 40.002 s]
10:02:12  [INFO] ------------------------------------------------------------------------
10:02:12  [INFO] BUILD SUCCESS
10:02:12  [INFO] ------------------------------------------------------------------------

Running same Tomee tests with mapped drive to nightly JDK build, Grinder 4314, unable to determine mapped JDK (JAVA VERSION OUTPUT is empty).

09:18:46  	/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../external/external.sh --clean --tag nightly --version 8 --impl openj9 --dir tomee; } 2>&1 | tee -a "/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16037183262603/TestTargetResult";
09:18:46  docker run --rm -v /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image:/opt/java/openjdk --rm adoptopenjdk-tomee-test:8-jdk-ubuntu-openj9-full 
09:18:47  Using docker image default Java
09:18:47  JAVA_BIN is: /opt/java/openjdk/bin
09:18:47  APPLICATION_NAME=tomee
09:18:47  APPLICATION_TAG=tomee-8.0.2
09:18:47  OS_TAG=ubuntu
09:18:47  =JAVA VERSION OUTPUT BEGIN=
09:18:47  =JAVA VERSION OUTPUT END=
09:18:47  Build TomEE without running test
09:18:47  
09:18:47  tomee_test_j9_0_FAILED
09:18:47

The text was updated successfully, but these errors were encountered:

sophia-guo · 2020-10-27T21:09:15Z

Rerun Grinder [4314] with setting SDK_RESOURCE nightly instead of releases looks fine 4328

sophia-guo · 2020-10-27T21:55:54Z

Also I can't figure out why https://ci.adoptopenjdk.net/view/Test_external/job/Test_openjdk8_j9_sanity.external_s390x_linux_tomee/ didn't run tomee_test_j9? I have pull lastest openjdk-tests and didn't see the testcase was disabled on s390? Curious

sophia-guo · 2020-10-31T15:33:03Z

Currently we mount the test jdk to /opt/java/openjdk, which is also the location of the docker adoptopenjdk images location. That is the the directory’s existing contents are obscured by the bind mount.

The issue only happened when mount latest release jdk to nightly docker image jdk8u272-b10_release is latest released jdk:

sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10_release:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly java -version
sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10_release:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly
root@fef3e0d97733:/# echo $JAVA_HOME
/opt/java/openjdk
root@fef3e0d97733:/# java -version
root@fef3e0d97733:/# /opt/java/openjdk/bin/java -version
root@fef3e0d97733:/#

Mounted with nightly jdk to nightly docker image jdk8u272-b10_release works fine (jdk8u272-b10 is nightly jdk:

sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-202010251721-b10)
Eclipse OpenJ9 VM (build master-6db38961c, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201029_832 (JIT enabled, AOT enabled)
OpenJ9   - 6db38961c
OMR      - 4d32dfda3
JCL      - 8425b0a2 based on jdk8u272-b10)
sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly
root@c4635e09a324:/# echo $JAVA_HOME
/opt/java/openjdk
root@c4635e09a324:/# java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-202010251721-b10)
Eclipse OpenJ9 VM (build master-6db38961c, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201029_832 (JIT enabled, AOT enabled)
OpenJ9   - 6db38961c
OMR      - 4d32dfda3
JCL      - 8425b0a2 based on jdk8u272-b10)

Mounted with latest releases jdk to latest released docker image works fine:

sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10_release:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:jdk8u272-b10_openj9-0.23.0
root@bd8aa8c24e1e:/# echo $JAVA_HOME
/opt/java/openjdk
root@bd8aa8c24e1e:/# java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build openj9-0.23.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201022_822 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 41f03341d5 based on jdk8u272-b10)

Mounted with nightly jdk to latest released docker image works fine:

sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:jdk8u272-b10_openj9-0.23.0 java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build master-6db38961c, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201029_832 (JIT enabled, AOT enabled)
OpenJ9   - 6db38961c
OMR      - 4d32dfda3
JCL      - 8425b0a2 based on jdk8u272-b10)
sophiags-MacBook-Pro:dockermount sophiag$ docker run --rm -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:jdk8u272-b10_openj9-0.23.0
root@3a50f574b38e:/# echo $JAVA_HOME
/opt/java/openjdk
root@3a50f574b38e:/# java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build master-6db38961c, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201029_832 (JIT enabled, AOT enabled)
OpenJ9   - 6db38961c
OMR      - 4d32dfda3
JCL      - 8425b0a2 based on jdk8u272-b10)

This is same for 11 and different platforms.
Looks like things is different from last openj9 release.

Will see if it's same for hotspot.

smlambert · 2020-10-31T18:27:53Z

tomee was only disabled for jdk8 hs s390x (because of no JIT slowness): #1996

@bharathappali @dinogun - Do you have ideas of why we are seeing the behaviour described in #2031 (comment)

bharathappali · 2020-10-31T20:34:58Z

@smlambert @sophia-guo Correct me if i'm wrong with the setup i have done below:

I have downloaded the release build from this link
extracted the tar ball and mounted to /opt/java/openjdk on the adoptopenjdk/openjdk8-openj9:nightly docker image

docker run --rm -v /Users/bharathappali/Downloads/jdk8u272-b10:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly

I have checked the env for options and found these related to java

OPENJ9_JAVA_OPTIONS=-Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,readonly,nonFatal
JAVA_TOOL_OPTIONS=-XX:+IgnoreUnrecognizedVMOptions -XX:+IdleTuningGcOnIdle

I saw this when i tried java -version

root@1fcc996dea2d:/# java -version
root@1fcc996dea2d:/#

I might be wrong but i thought that by default jvm uses scc, and in this case the SCC is generated by a nightly build and used by a release build, so jus gave a try with -Xshareclasses:none and it worked

root@1fcc996dea2d:/# java -Xshareclasses:none -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build openj9-0.23.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201022_822 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 41f03341d5 based on jdk8u272-b10)
root@1fcc996dea2d:/#

The SCC option is given nonFatal which i believe even if JVM can't use/access cache it shouldn't effect the VM startup. but not sure in this case i feel JVM is not coming up. I need to investigate more on this and will update you.

EDIT : I would like to correct the sentence which i mentioned earlier above i feel JVM is not coming up

I wouldn't say that JVM is not coming up as if there is any error in VM startup OpenJ9 output's something like

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Which is not seen in this case.

Thanks and advance.

sophia-guo · 2020-11-02T15:14:38Z

@bharathappali yes, that's the issue we hit. And just update that hotspot doesn't have this issue.

bharathappali · 2020-11-04T08:38:24Z

I'm seeing a strange behavior with this particular experiment of volume mounting the JDK

Appalis-MacBook-Air:~ bharathappali$ docker run --rm -v /Users/bharathappali/Downloads/jdk8u272-b10:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly
root@cd70fd117116:/# java -version

root@cd70fd117116:/# java -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,readonly,nonFatal -version

root@cd70fd117116:/# java -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,printTopLayerStats -version

Current statistics for cache "openj9_system_scc": 

Cache created with:
	-Xnolinenumbers                      = false
	BCI Enabled                          = true
	Restrict Classpaths                  = false
	Feature                              = cr
	Processor Features                   = fpu cx8 cmov mmx sse
	                                       sse2 ssse3 sse4_1 popcnt
	                                       aesni avx

Cache contains only classes with line numbers

base address                         = 0x00007FF8608EC000
end address                          = 0x00007FF8615D5000
allocation pointer                   = 0x00007FF860DAED00

cache layer                          = 0
cache size                           = 13630880
softmx bytes                         = 13630880
free bytes                           = 2481420
Reserved space for AOT bytes         = -1
Maximum space for AOT bytes          = -1
Reserved space for JIT data bytes    = -1
Maximum space for JIT data bytes     = -1
Metadata bytes                       = 162896
Metadata % used                      = 1%
Class debug area size                = 1081344
Class debug area used bytes          = 508368
Class debug area % used              = 47%

ROMClass bytes                       = 4992256
AOT bytes                            = 3750248
JIT data bytes                       = 118356
Zip cache bytes                      = 950640
Startup hint bytes                   = 120
Data bytes                           = 93600

# ROMClasses                         = 1910
# AOT Methods                        = 922
# Classpaths                         = 1
# URLs                               = 0
# Tokens                             = 0
# Zip caches                         = 5
# Startup hints                      = 1
# Stale classes                      = 0
% Stale classes                      = 0%


Cache is 81% full

Cache is accessible to current user = true

root@cd70fd117116:/# java -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build openj9-0.23.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201022_822 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 41f03341d5 based on jdk8u272-b10)

root@cd70fd117116:/# java -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,readonly,nonFatal -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build openj9-0.23.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201022_822 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 41f03341d5 based on jdk8u272-b10)

root@cd70fd117116:/#  java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
Eclipse OpenJ9 VM (build openj9-0.23.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20201022_822 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 41f03341d5 based on jdk8u272-b10)

root@cd70fd117116:/#

When i was running java -version i couldn't see any output.
When i checked printTopLayerStats i see the cache is accessible. I tried removing readonly,nonFatal for the first time it worked fine and there after even with readonly,nonFatal its working fine.

later a simple java -version is working as well.

bharathappali · 2020-11-04T08:44:24Z

And just update that hotspot doesn't have this issue.

@sophia-guo Thanks for sharing this info, This makes it clear that the problem is with SCC (which is OpenJ9 specific) so hotspot is not having any problem.

sophia-guo · 2020-11-04T18:59:52Z

Move to build-docker repo AdoptOpenJDK/openjdk-docker#457

smlambert added the bug label Oct 27, 2020

smlambert mentioned this issue Oct 28, 2020

AQAvit meeting Oct 28, 2020 #2032

Closed

sophia-guo mentioned this issue Nov 4, 2020

Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October AdoptOpenJDK/openjdk-docker#457

Open

sophia-guo closed this as completed Nov 4, 2020

bharathappali mentioned this issue Nov 5, 2020

Unexpected JVM startup behavior when a release build java uses nightly build SCC eclipse-openj9/openj9#11110

Closed

karianna added this to the November 2020 milestone Nov 5, 2020

karianna added the duplicate label Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October #2031

Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October #2031

smlambert commented Oct 27, 2020

sophia-guo commented Oct 27, 2020

sophia-guo commented Oct 27, 2020

sophia-guo commented Oct 31, 2020 •

edited

Loading

smlambert commented Oct 31, 2020

bharathappali commented Oct 31, 2020 •

edited

Loading

sophia-guo commented Nov 2, 2020

bharathappali commented Nov 4, 2020

bharathappali commented Nov 4, 2020

sophia-guo commented Nov 4, 2020

Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October #2031

Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October #2031

Comments

smlambert commented Oct 27, 2020

sophia-guo commented Oct 27, 2020

sophia-guo commented Oct 27, 2020

sophia-guo commented Oct 31, 2020 • edited Loading

smlambert commented Oct 31, 2020

bharathappali commented Oct 31, 2020 • edited Loading

sophia-guo commented Nov 2, 2020

bharathappali commented Nov 4, 2020

bharathappali commented Nov 4, 2020

sophia-guo commented Nov 4, 2020

sophia-guo commented Oct 31, 2020 •

edited

Loading

bharathappali commented Oct 31, 2020 •

edited

Loading