Update E3SM-Project submodule

[xylar]

This merge updates the E3SM-Project submodule from 6b81271377 to 569ed6b730.

This update includes the following MPAS-Ocean and MPAS-Frameworks PRs (check mark indicates bit-for-bit with previous PR in the list):

• (ocn) Make MPAS-Ocean units CF compliant E3SM-Project/E3SM#5027
• (ocn) removed use of Fortran fraction intrinsic from ocn diagnostics E3SM-Project/E3SM#5128
• (ocn) correct nCells to nEdges E3SM-Project/E3SM#5142
• (fwk) [MPAS-Framework] Add a new name_in_output registry attribute to variables E3SM-Project/E3SM#5120
• (ocn) Adds submesoscale parameterization for mpas-ocean E3SM-Project/E3SM#5099
  • comparison with 5120 fails for 17 test cases:
    • all forward runs except with the z-level ice-shelf test case and those with RK4
    • This was documented in Adds submesoscale parameterization for mpas-ocean E3SM-Project/E3SM#5099 (comment)
• (ocn) modified vertical range for KEvertex calc to avoid NaNs E3SM-Project/E3SM#4945
  • comparison with 5099 fails for 4 tests with RK4, as expected
• (fwk) [standalone MPAS] Finalize PIO when finalizing MPAS i/o E3SM-Project/E3SM#5152
• (ocn) [MPAS-Framework] Rename gregorian_noleap to just noleap calendar E3SM-Project/E3SM#5162
• (ocn) Add normalGMBolusVelocity to the gm package E3SM-Project/E3SM#5167
  • comparison with 5162 fails for 17 test cases:
    • all forward runs except with the z-level ice-shelf test case and those with RK4
    • This was documented in Add normalGMBolusVelocity to the gm package E3SM-Project/E3SM#5167 (comment)
• (fwk) Increase printbuf size by one E3SM-Project/E3SM#5174
• (fwk) [MPAS-Framework] New fill value mask feature E3SM-Project/E3SM#5154
• (ocn) Alters MOC calculation in ocean component E3SM-Project/E3SM#5170
• (fwk) Add mpas-framework support for several other attributes E3SM-Project/E3SM#5183
• (ocn) add carbon to conservation check analysis member E3SM-Project/E3SM#5151
• (ocn) move ocean namelist transport_tests to testing section E3SM-Project/E3SM#5186
• (ocn) Fixes Redi Surface Taper double counting E3SM-Project/E3SM#5171
  • comparison with 5186 fails for 13 test cases:
    • baseline comparison for all global ocean forward runs fail (as expected). Differences are significantly larger than machine precision, which I think was also expected.
• (ocn) Enables and modifies the submesoscale eddy parameterization E3SM-Project/E3SM#5172
• (ocn) Fix barotropic thickness on edge in split explicit subcycling E3SM-Project/E3SM#5195
  • comparison with 5172 fails for 18 test cases:
    • 4 vs 8 core validation for baroclinic_channel/decomp_test fails. This was reported in Ocean fails stand-alone decomp test, intel optimized  E3SM-Project/E3SM#5219 though it was originally attributed to 5099, not this PR. I believe this needs to be fixed.
    • baseline comparison for all split-explicit forward runs fail (as expected). Differences are small (1e-7 in temperature, for example) but not machine precision, which I think was expected.
    • No compass testing is documented in the PR
    • We plan to fix this in Move bottomDepthEdge calculation to single loop over all edges E3SM-Project/E3SM#5356 but won't wait for that to merge this PR.
• (ocn) Fix threading issues in submesoscale code E3SM-Project/E3SM#5216
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Add CF-compliant Time coordinate E3SM-Project/E3SM#5202
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Change defunct Fortran line labels to end do E3SM-Project/E3SM#5189
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Replace device_resident and other OpenACC fixes for ocean E3SM-Project/E3SM#5194
  • comparison with 5189 fails for 14 test cases:
    • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
    • baseline comparison fails for all split-explicit and z-star global ocean, ziso, ISOMIP+ and ice-shelf-2d forward runs, which I believe was not expected. In the PR, BFB results were reported for the nightly suite. Differences are small (1e-7 in temperature, for example) but not machine precision.
    • BFB testing in compass was only done with Gnu for the PR: Replace device_resident and other OpenACC fixes for ocean E3SM-Project/E3SM#5194 (comment)
    • However, the test is BFB when we use OpenMPI (as E3SM does by default) instead of Intel-MPI so I think we can call it good.
• (ocn) Add missing variables to omp pragmas from PR #5195 E3SM-Project/E3SM#5226
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195, even though this PR fixed some openmp directives from 4195.
• (fwk) Update to github.io in mpas license website E3SM-Project/E3SM#5143
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Remove unneeded mesh declaration E3SM-Project/E3SM#5201
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Fix out-of-bounds index for vertical mixing with one active layer E3SM-Project/E3SM#5187
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Add zeroing of velocity to ocean init mode E3SM-Project/E3SM#5232
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (fwk) Fix precision for writing field attributes E3SM-Project/E3SM#5251
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Bugfixes for ocean inactive top cells feature E3SM-Project/E3SM#5246
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) Add missing time level index to 'ssh' coupling field E3SM-Project/E3SM#5305
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• (ocn) [Standalone MPAS-Ocean] Add tracers to dam break initialization E3SM-Project/E3SM#5279
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195
• new E3SM-Project 569ed6b730
  • 4 vs 8 core validation for baroclinic_channel/decomp_test fails as in 5195

[xylar]

I'm having trouble tracking down changes so I'm going to write a utility for comparing the results of the pr test suite between subsequent relevant E3SM PR merges and include that in this merge.

[xylar]

@mark-petersen and @jonbob, I would appreciate you having a look at this when you can. I'll comment more on Slack so we can discuss there (or here).

[xylar]

Results from all of these tests are available here:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20221127/update_e3sm_project

I think the subdirectories are straightforward. each has log files that document the validation failures and the size of the differences.

[mark-petersen]

I think there are only two PR results to resolve here:

• Fix barotropic thickness on edge in split explicit subcycling E3SM-Project/E3SM#5195
• Replace device_resident and other OpenACC fixes for ocean E3SM-Project/E3SM#5194

On the first, 5195, failure of baroclinic_channel/decomp_test will be resolved by E3SM-Project/E3SM#5356. Now that we understand it and plan to fix it, I think we can proceed with updating past E3SM-Project/E3SM#5195 in compass.

On the second, I tested on cori and with OPENMP=false the nightly test suite passes bfb comparison with previous master for gnu (both debug and optimized) and intel (both debug and optimized). I need to try with OPENMP=true, but this appears to be a mismatch caused by OPENMP.

[mark-petersen]

Tested intel on chrysalis with OPENMP=true. Comparing before and after E3SM-Project/E3SM#5194 (8b8bbba994 vs 2491723cca):

• intel optimized: all tests match bfb before/after 5194. ocean/baroclinic_channel/10km/decomp_test fails internal comparison on both, as expected.
• intel debug: two threads_tests fail validation and baseline compare. All others pass validation and baseline compare. (I don't know why the thread tests fail).

@xylar, you saw numerous baseline mismatches in the first post above for the 5194 merge, which is different from what I see. I double-checked the commits. I don't understand why we are getting different results.

[mark-petersen]

@xylar, your new testing script looks very useful. Thanks for doing that. Looking in your directory here, I can see the baseline mismatches:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20221127/update_e3sm_project/22_5194_5189/compass_pr.o253780

and I see you are running intel on chrysalis, same as me. I loaded my environment with compass using this commit 6396cfa Date: Thu Dec 8, and my modules and make command are:

Currently Loaded Modules:
  1) cmake/3.19.1-yisciec   3) intel/20.0.4-kodw73g           5) intel-mkl/2020.4.304-g2qaxzf   7) netcdf-c/4.4.1-qvxyzq2         9) parallel-netcdf/1.11.0-b74wv4m
  2) perl/5.32.0-bsnc6lt    4) intel-mpi/2019.9.304-tkzvizk   6) hdf5/1.8.16-se4xyo7            8) netcdf-fortran/4.4.4-rdxohvp

make intel-mpi  USE_PIO2=true OPENMP=true DEBUG=false
make intel-mpi  USE_PIO2=true OPENMP=true DEBUG=true

When I compare the two merge commits:

*   2491723cca (after) Merge branch 'philipwjones/mpas-ocean/rm-device-resident' (PR #5194)
...
*    8b8bbba994 (before) Merge branch 'azamat/cime/shorten-jenkins-paths-for-crusher-amdclang' (PR #5224)

I get bfb match for both intel debug and intel optimized. Are you comparing the merge commits or the branch commits just before the merge? Another difference is that I'm comparing to 8b8bbba994, the previous first-parent commit, and you are comparing to PR 5189, the previous ocean merge. Could that explain the difference in our results? If so, there must be another merge in between 5189 and 5194 that changed the baseline. Could you repeat my test to verify?

[xylar]

@mark-petersen, thank you. I will retest and try to figure out which commit between PR 5189 and 5194 is the actual culprit. I will first retest 5189 vs. 5194 because I just don't see how any of these intermediate PRs could be the cause:

* 2491723cca (22_5194) Merge branch 'philipwjones/mpas-ocean/rm-device-resident' (PR #5194)
* 8b8bbba994 Merge branch 'azamat/cime/shorten-jenkins-paths-for-crusher-amdclang' (PR #5224)
* 27b7d6c5f8 Merge branch 'bishtgautam/datm/jra' (PR #5150)
* a17d4b4fc0 Merge branch 'origin/oksanaguba/homme/spock' (PR #5039)
* 2ad7a59139 Merge branch 'ambrad/hommexx/sl-support-mpi-on-host' (PR #5223)
* f8b135bcf9 (21_5189) Merge branch 'mark-petersen/ocn/harmonic-analysis-enddo' (PR #5189)

[xylar]

@mark-petersen, I'm still baffled. I'm working from a slightly different compass (3b2b18f, which is the branch for #466) I have the same modules:

$ module list

Currently Loaded Modules:
  1) cmake/3.19.1-yisciec           6) hdf5/1.8.16-se4xyo7
  2) perl/5.32.0-bsnc6lt            7) netcdf-c/4.4.1-qvxyzq2
  3) intel/20.0.4-kodw73g           8) netcdf-fortran/4.4.4-rdxohvp
  4) intel-mpi/2019.9.304-tkzvizk   9) parallel-netcdf/1.11.0-b74wv4m
  5) intel-mkl/2020.4.304-g2qaxzf

I have just used environment variables (in the load script) to have:

USE_PIO2=true
OPENMP=true

By not specifying anything, I have DEBUG=false. I just run make clean; make intel-mpi.

I'm seeing the same non-bfb behavior with 5194 vs. 5189 as before. I'm seeing bfb between 5224 and 5189. I am about to test 5194 vs. 5224 but I can't see how that will not be non-bfb. I will update shortly.

[xylar]

Okay, this is truly bizarre! I see BFB between 5194 and 5224, and also between 5224 and 5189 but not between 5194 and 5189. I really don't understand what's going on.

Update: sorry, I jumped the gun. It passed a few baroclinic channel tests cases and I thought that was odd but I now see those were passing before. So I'm seeing the same diffs between 5194 and 5224 as I saw between 5194 and and 5189. They seem to be quite persistent for me.

See:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20221213/e3sm_submodule/22_5194_vs_5189
/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20221213/e3sm_submodule/22_5194_vs_5224

[mark-petersen]

OK. Maybe I made a mistake. I'll test again.

[mark-petersen]

Sorry, it was my mistake. I can reproduce your baseline mismatch between 5224 and 5194, using intel optimized on chrysalis. I used compass suite -p and pointed to the wrong directory the first time.

[xylar]

Things are looking good when I use OpenMPI:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20221213/e3sm_submodule/openmpi_22_5194_vs_5224

[xylar]

@mark-petersen, if you're okay with the BFB results I'm seeing for that last test (with OpenMPI instead of Intel-MPI), I think we can move on. We will bring in E3SM-Project/E3SM#5356 as soon as it's merged but this PR doesn't need to wait for that.

[mark-petersen]

Yes, now that we understand the two non-bfb commits above, 5194 and 5195, we can proceed with this update. Thanks.

[xylar]

Thanks very much for your testing, @mark-petersen and the effort in figuring out these differences. It makes a big difference to me that we understand when things changed and why.