[INSTALL]: Consider updating hdf5 to 1.14.3 and zlib to 1.3, or replace zlib with zlib-ng

[climbfuji]

Package name

hdf5 / zlib

Package version/tag

1.14.3 / 1.3

Build options

current

Installation timeframe

Consider adding to spack-stack-1.6.0 if time permits

Other information

n/a

WCOSS2

• Check this box if and only if your package should be installed on WCOSS2 Cactus and Dogwood (all spack-stack packages will be installed on Acorn). If not, you may disregard the rest of the items below and submit this request.

WCOSS2: General questions

already approved on wcoss2 (hdf5@1.14.0, zlib@1.3)

WCOSS2: Installation and testing

No response

WCOSS2: Technical & security review list

• The code is mature, stable, and production ready
• The code is does not and cannot use the internet, and does not contain URLs (http, https, ftp, etc.) except in comments
• The package does not contain prebuilt binary files that have not been approved by NCO security review
• The code has no publicly disclosed cybersecurity vulnerabilities and exposures (search https://cve.mitre.org/cve/)
• The code is not prohibited by DHS, DOC, NOAA, or NWS
• The code comes from a trusted source. Trusted sources include other NWS, NOAA, or DOC, agencies, or other Federal agencies that operate at a FISMA high or equivalent level. Additionally, trusted sources could be third-party agencies through which there is an existing SLA on file (such as RedHat).
• The code is actively maintained and supported (it continues to get updates, patches, etc.)
• The code is not maintained by a private entity operating in a foreign country (if it is, make a note below)
• There is sufficient documentation to support maintenance
• There are no known security vulnerabilities or weaknesses
• Installing and running the code does not require privileged processes/users
• There are no software dependencies that are unapproved or have security concerns (if there are, make a note below)
• There are no concerns related to SA, SI, and SC NIST control families

WCOSS2: Additional comments

No response

[AlexanderRichert-NOAA]

Thoughts on going to zlib-ng as long as we're at it? I'd be happy to do some testing to see if it's worth the switch.

[climbfuji]

Thoughts on going to zlib-ng as long as we're at it? I'd be happy to do some testing to see if it's worth the switch.

Do you have a feeling for the potential impact of this change, i.e. how much do we believe that it is a drop-in replacement and that it will all work just fine? I am asking because I am looking at the list of planned changes for spack-stack-1.6.0, the resources we have to implement and test all of them, and the plan to roll out the release before Christmas.

[climbfuji]

On Hercules, moving to hdf5@1.14.3 and zlib@1.3 seem to help with the netCDF/hdf5 errors related to the use of quantization in the ufs-weather-model. See the lengthy discussion in ufs-community/ufs-weather-model#2015.

[climbfuji]

@AlexanderRichert-NOAA @ulmononian Since the testing on Hercules has shown that updating to hdf5@1.14.3 doesn't solve the issue with the parallel writes, I suggest to not try to squeeze this into spack-stack-1.6.0. Instead, make the update in 1.7.0 and use the extra time we have until the spack-stack-1.7.0 release to move to zlib-ng. Thoughts?

[AlexanderRichert-NOAA]

zlib-ng for 1.6.0 or 1.7.0? FYI I'll be off for a while starting next Wednesday, so if you want hold off for 1.7.0 that's fine with me (though maybe we have a bunch of testing to do either way...).

[ulmononian]

@climbfuji @AlexanderRichert-NOAA is the move to zlib-ng without risk this close to 1.6.0? is it necessary?

[climbfuji]

@climbfuji @AlexanderRichert-NOAA is the move to zlib-ng without risk this close to 1.6.0? is it necessary?

Sorry for the confusion. I clarified the language above: "use the extra time we have until the spack-stack-1.7.0 release to move to zlib-ng"

[ulmononian]

@climbfuji @AlexanderRichert-NOAA is the move to zlib-ng without risk this close to 1.6.0? is it necessary?

Sorry for the confusion. I clarified the language above: "use the extra time we have until the spack-stack-1.7.0 release to move to zlib-ng"

ah, i misread! yes i think we should hold off on this until 1.7.0.

[climbfuji]

• Update to hdf5@1.14.3 will happen in Update Discover site configs, fix CI buillds, install GEOS-enabled spack-stack on Discover / AWS ParallelCluster #993
• Update to zlib@1.3 or move to zlib-ng: to check

[AlexanderRichert-NOAA]

Did anyone ever test changing zlib->zlib-ng? I can't remember if I did at one point. In any case I'd be happy to test that on its own, then if it does changes thing numerically, we can have UFSWM folks reevaluate RT results along with other changes.

[AlexanderRichert-NOAA]

I ran a couple UFS WM RTs just now with a stack built on the latest version of zlib-ng and they both completed successfully and with results matching baselines. So I vote for moving forward with zlib-ng; I'm happy to make the necessary changes.

[climbfuji]

Update to hdf5 was done, waiting for switch to zlib-ng instead of updating zlib to version 1.3

[edwardhartnett]

Sorry I took so long to respond but we did indeed test zlib-ng and it worked and was about 10% faster. So we should be using it.

[climbfuji]

Sorry I took so long to respond but we did indeed test zlib-ng and it worked and was about 10% faster. So we should be using it.

Thanks, that's good to know. I'll test with JEDI this week and hopefully EPIC can run some tests on their end, too.

[ulmononian]

@RatkoVasic-NOAA have you had a chance to run tests with this (zlib-ng) with the wm or srw?

[RatkoVasic-NOAA]

@RatkoVasic-NOAA have you had a chance to run tests with this (zlib-ng) with the wm or srw?

@ulmononian , no, both are using zlib/1.2.13

[AlexanderRichert-NOAA]

I tested a slew of UFS WM RTs on Hera using zlib-ng, but if you want to set up more tests, you can add a checklist in #1016 to complete before we merge.

[climbfuji]

Closing this as done. We've updated to hdf5@1.14.3 and replaced zlib with zlib-ng instead of updating to zlib@1.3.