Unable to run marine forecast when building with NG-GODAS option

[kbhargava]

Description

I am unable to run a forecast using the ufs_model executable created using the ufs-bundle. Though I am able to run when building ufs-weather-model from it's develop branch. I see the following error:

20240312 214908.614 INFO             PET154 MOM_cap:restartfile = r 
20240312 215136.141 ERROR            PET154 src/addon/NUOPC/src/NUOPC_Base.F90:2108 Invalid argument  - tocn is not a StandardName in the NUOPC_FieldDictionary!
20240312 215136.142 ERROR            PET154 src/addon/NUOPC/src/NUOPC_Base.F90:486 Invalid argument  - Passing error in return code
20240312 215136.142 ERROR            PET154 mom_cap.F90:813 Invalid argument  - Passing error in return code
20240312 215136.142 ERROR            PET154 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2898 Invalid argument  - Phase 'IPDv03p1' Initialize for modelComp 3: OCN did not return ESMF_SUCCESS
20240312 215136.142 ERROR            PET154 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:1331 Invalid argument  - Passing error in return code
20240312 215136.142 ERROR            PET154 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:483 Invalid argument  - Passing error in return code
20240312 215136.142 ERROR            PET154 UFS.F90:386 Invalid argument  - Aborting UFS 
20240312 215136.142 INFO             PET154 Finalizing ESMF

I figured there's some issue with the code in mom_cap.F90. When I do a git status in the directory ufs-bundle/ufs-weather-model/MOM6-interface/MOM6. I see that it points to the correct HEAD but there are some modified files (See output below)

HEAD detached at 10521a921
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   config_src/drivers/nuopc_cap/mom_cap.F90
	modified:   config_src/drivers/nuopc_cap/mom_cap_methods.F90
	modified:   config_src/drivers/nuopc_cap/mom_ocean_model_nuopc.F90
	modified:   pkg/GSW-Fortran (modified content)
	modified:   src/core/MOM.F90
	modified:   src/framework/MOM_horizontal_regridding.F90

I haven't had the time to look more into what is causing this. Most likely comes from here

[climbfuji]

Correct, and this was intentional so that code changes needed for MOM6 wouldn't have to be submitted to the repo. @mark-a-potts added this.

The error you are seeing looks like the same problem you've had in the past, i.e. the contents of fd_nems.yaml not matching what the particular version of the code expects?

[kbhargava]

Correct, and this was intentional so that code changes needed for MOM6 wouldn't have to be submitted to the repo. @mark-a-potts added this.

The error you are seeing looks like the same problem you've had in the past, i.e. the contents of fd_nems.yaml not matching what the particular version of the code expects?

Yes. The fd_nems.yaml generated when I build stand-alone ufs-weather-model works fine with standalone executable but not with the one built by the ufs-bundle. Even if I use the fd_nems.yaml that's generated when I build ufs-bundle, it doesn't work as I think it's build keeping their tagged MOM6 version.

[mark-a-potts]

It could be as simple as needing to change the name from fd_nems.yaml to fd_ufs.yaml. That is the expected name for the ufs-weather-model now. Similarly, nems.configure is now ufs.configure. That said, I don't think the connections back to soca will work correctly without some more work.

[kbhargava]

It could be as simple as needing to change the name from fd_nems.yaml to fd_ufs.yaml. That is the expected name for the ufs-weather-model now. Similarly, nems.configure is now ufs.configure. That said, I don't think the connections back to soca will work correctly without some more work.

Thanks Mark! I have those renamed to ufs instead of nems, doesn't work. I am trying to run just the forecast, no soca components yet. But that is the next step.

[mark-a-potts]

What experiment/test case are you trying to run?

[kbhargava]

What experiment/test case are you trying to run?

I am trying to run a simple 24 hour forecast. The model resolution I am using is quarter degree and I use GFS forcing for the atmosphere. The experiment is part of skylab (see here). Currently, it is set up to use the ufs_model built outside the ufs-bundle following the instructions here. This runs successfully.

I am trying to use the executable built within ufs-bundle. And that is when I get the errors. If I use the same fd_ufs.yaml that I use for the foremer I get the error I mentioned above. If I use the one generated within ufs-bundle, I get a different error complaining that it cannot find Foxx_evap in the NUPC Dictionary.

[travissluka]

FYI, SOCA team has decided to remove the in-core capabilities for running with MOM6. It's not in the AOP for this coming year, EMC will not be using it, and it's clearly causing issues. In the next week or so I'll issue a PR that removes large chunks of code from SOCA, which will allow us to use the authoritative MOM6/FMS repos, at which time these patches can be reverted.

When in-core hofx IS on the agenda for an AOP year, we'll get changes made to MOM6 first before we add the in core hofx capabilities back to SOCA.

[mark-a-potts]

Try changing the UFS_TAG in the ufs-bundle/CMakeLists.txt file to this--

set(UFS_TAG "fb8792b0b82aba888b0b0a236fdad32353bfc84c" )

You might also have to rm the old ufs-weather-model directory under ufs-bundle, but this will pull the specific tag/hash you want for the ufs-wm and build it along with the ufs-bundle. Depending on how old that version of the wm is, you may have to go back to fd_nems.yaml and nems.configure naming conventions....

[kbhargava]

Try changing the UFS_TAG in the ufs-bundle/CMakeLists.txt file to this--

set(UFS_TAG "fb8792b0b82aba888b0b0a236fdad32353bfc84c" )

You might also have to rm the old ufs-weather-model directory under ufs-bundle, but this will pull the specific tag/hash you want for the ufs-wm and build it along with the ufs-bundle. Depending on how old that version of the wm is, you may have to go back to fd_nems.yaml and nems.configure naming conventions....

Thanks for the suggestion, same problem with the patch though. I am able to run with standalone executable using the develop branch of ufs-weather-model. I think I will wait for Travis' work and then try again.

[climbfuji]

@kbhargava Is this somewhere on an HPC that I have access to? I would like to build ufs-bundle myself and try to run your code. Thanks

[kbhargava]

@kbhargava Is this somewhere on an HPC that I have access to? I would like to build ufs-bundle myself and try to run your code. Thanks

I was testing on orion.

[climbfuji]

@kbhargava Is this somewhere on an HPC that I have access to? I would like to build ufs-bundle myself and try to run your code. Thanks

I was testing on orion.

Well, that's good for starters, and if you could please share the run directory that I can copy so that I can drop my executable in there, then I might be able to help.

[kbhargava]

@kbhargava Is this somewhere on an HPC that I have access to? I would like to build ufs-bundle myself and try to run your code. Thanks

I was testing on orion.

Well, that's good for starters, and if you could please share the run directory that I can copy so that I can drop my executable in there, then I might be able to help.

I have the ufs-bundle built at /work2/noaa/jcsda/kritib/jedi if that's what you are asking.

[climbfuji]

@kbhargava I need the run directory for your failing marine forecast so that I can copy it and drop my ufs-bundle executable there. Thanks.

[kbhargava]

Here is the rundirectory for my failed experiment /work/noaa/marine/kritib/jedi/workdir/83f385

[climbfuji]

@travissluka I added you here to help @kbhargava if you can.