Status of Intel compiler/MPI on Cheyenne

[ulmononian]

I noticed in the Cheyenne site/compilers.yaml file that the default compiler/MPI for spack-stack on Cheyenne is intel@19.1.1.217/intel-mpi@2019.7.217, with the note that "[w]eird MPI issues when running single-task jobs (mpiexec -np 1) with latest Intel compiler/mpi library - code simply hangs forever". There is a simlar note in the spack-stack RTD (https://spack-stack.readthedocs.io/en/latest/Platforms.html#ncar-wyoming-cheyenne), with the addition of "for JEDI, job hangs forever in a particular MPI communication call in oops."

I was wondering if this compiler/MPI combination was selected to optimize the build of JEDI environments (and avoid these hangups in OOPS), or if this same issue was already detected when building the ufs-wm or ufs-srw stack environments? I would prefer to use a newer compiler/MPI, but don't want to repeat work that has already proven this to be a dead-end. Perhaps this needs to be opened as an issue?

[climbfuji]

The UFS can use newer versions just fine (it always runs with at least 6 MPI tasks for the global, and usually doesn't run with just one task for the regional either). We did discuss this with UCAR IT and MMM folks, and it was decided that we won't bother on our end, given the remaining lifetime of the system, and stick with the older version for now.

We can use a separate environment for the UFS on Cheyenne and then unify them on Derecho.

[ulmononian]

Thanks for clarifying, @climbfuji. I might test w/ Intel '22 for UFS applications, then. We need to adapt our Jenkins job to automate spack installs and updates on Cheyenne, so I may need to run it from a feature branch for now.

[ulmononian]

@climbfuji follow-up question: were the failures when using newer Intel compiler/MPI encountered when doing the spack install of the spack-stack for the JEDI application OR within OOPS when calling the packages built w/ the newer Intel compiler/MPI? asking for clarification here so i don't just assume all is well if i have a successful install of the ufs-srw-dev environment on cheyenne with Intel 2022 (testing now here: /glade/scratch/bcameron/spack_sandbox/cb_fork w/ https://github.com/ulmononian/spack-stack/blob/feature/cheyenne_update_compiler)

[climbfuji]

I was able to install it cleanly (skylab-1.1.0 at that time, but skylab-2.0.0 would work, too). The errors where only when running mpi jobs with only one MPI task.

[ulmononian]

@ulmononian 10-4. will note this when running SRW WE2E tests and UFS-WM RTs, although like you said, they may never use an MPI job w/ only one task.