Update operational workflow to work with GSI develop branch

[ADCollard]

What new functionality do you need?

Update operational workflow to work with GSI develop branch. This mostly involves changing the links to elements that have been moved out of the GSI repository since GFSv16.3.

What are the requirements for the new functionality?

Able to cycle with v16 workflow but using a version of GSI close to develop. This will not reproduce operations.

Acceptance Criteria

Able to cycle with v16 workflow but using a version of GSI close to develop. This will not reproduce operations.

Suggest a solution (optional)

A new branch has been created in my fork to address this: gfs.v16.GSIdev

[ADCollard]

Changes so far:

• Move the following from from GSI scripts/ directory to global-workflow scripts/ directory.

exglobal_atmos_analysis.sh
exglobal_atmos_analysis_calc.sh
exglobal_diag.sh
exgdas_atmos_chgres_forenkf.sh
exgdas_atmos_verfozn.sh
exgdas_atmos_verfrad.sh
exgdas_atmos_vminmon.sh
exgdas_enkf_ecen.sh
exgdas_enkf_fcst.sh
exgdas_enkf_post.sh
exgdas_enkf_select_obs.sh
exgdas_enkf_sfc.sh
exgdas_enkf_update.sh
exgfs_atmos_vminmon.sh

• Turn off symbolic links from the GSI to global-workflow scripts/ directories.
• Modify checkout.sh to checkout the develop version of GSI (for now - will be a fixed hash eventually)
• Add former gsi.fd scripts to ush:

calcanl_gfs.py
calcinc_gfs.py
getncdimlen
gsi_utils.py

• Add external gsi_utils repository to build scripts.

[ADCollard]

Two initial issues:

1. ncdiag_cat_serial.x is no longer part of the GSI repository. Where has this gone?
2. We need to use an older version of global_anavinfo.l127.txt as we do not want to change this functionality at this stage.

[RussTreadon-NOAA]

ncdiag_cat_serial.x

The source code and build for ncdiag_cat_serial.x resides in NOAA-EMC/GSI-ncdiag.

g-w develop accesses ncdiag_cat_serial.x by loading module ncdiag/1.0.0 This adds /apps/ops/prod/libs/intel/19.1.3.304/cray-mpich/8.1.4/ncdiag/1.0.0/bin to ${PATH}. The ncdiag bin directory contains

-rwxr-xr-x 1 ops.prod prod 1131072 Jul  7  2022 ncdiag_cat_serial.x
-rwxr-xr-x 1 ops.prod prod 1181088 Jul  7  2022 nc_diag_cat.x
-rwxr-xr-x 1 ops.prod prod  913160 Jul  7  2022 test_nc_unlimdims.x

We could add module load ncdiag/1.0.0 to the ecflow scripts which drive the jobs using ncdiag_cat_serial.x. For rocoto runs of ops g-w ncdiag/1.0.0 could be added to the ops g-w modulefiles/module_base.wcoss2.lua.

There's also a ncdiag/1.1.1 and ncdiag-A. I don't know how these differ from ncdiag/1.0.0.

global_anavinfo.l127.txt
NOAA-EMC/GSI-fix at 6882569 contains a global_anavinfo.l127.txt that differs from gfs.16.3.19 only in the addition of an efsoi section

russ.treadon@clogin01:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/develop/fix> diff /lfs/h1/ops/prod/packages/gfs.v16
.3.19/fix/fix_gsi/global_anavinfo.l127.txt global_anavinfo.l127.txt 
71a72,80
> state_vector_efsoi::
> !var     level  itracer source     funcof
>  u       127      0     met_guess    u
>  v       127      0     met_guess    v
>  tv      127      0     met_guess    tv
>  q       127      1     met_guess    q
>  ps        1      0     met_guess    prse
> ::
> 

We could use this global_anavinfo.l127.txt in ops g-w runs. I don't think the efsoi section will adversely impact operational global 4denvar runs of gsi.x.

[KateFriedman-NOAA]

@ADCollard let me know if I can assist with this at all.

[ADCollard]

Thanks @RussTreadon-NOAA and @KateFriedman-NOAA !

I will make the required module changes to bring in ncdiag/1.0.0

I tried an older version of anavinfo and it did not seem to solve my problem. The gdasanalcalc step is failing as extra fields appear to have been added to the sig* files. I am digging to see how to solve this.

[RussTreadon-NOAA]

@ADCollard , you may need to use updated GSI utilities from NOAA-EMC/GSI-utils. The calc_analysis code was update a little over a month ago (see GSI-utils PR #53).

[ADCollard]

@RussTreadon-NOAA Your information about GSI-utils PR #53 was the answer.
I needed to go back to an earlier version of GSI-utils to be consistent with v16.

[ADCollard]

GSI #602 removes use_qsatensmean from the namelist used by enkf.x. This was a bug fix (it wasn't doing anything).

So this option has been removed from exgdas_enkf_update.sh to avoid a failure.

[ADCollard]

Now to turn on some new stuff! Let's start by setting optconv to 0.06 in the GSI SETUP namelist. This inflates the observation error for the water channels of hyperspectral sounders when the Jacobians are large. See GSI #375. This is changed in scripts/exglobal_atmos_analysis.sh.

[ADCollard]

Need to reinstate the observation monitor capability which was moved out of the develop version of the GSI repository into a new GSI-Monitor repository. Check out the version from before the directory structure was changed:
GSI-Monitor #e1f9f21af16ce912fdc2cd75c5b27094a550a0c5

We also need to take the relevant scripts from the global-workflow. This needs to be before all the directory trees were updated. So we went with global-workflow #f159d39a3b28dfcc120cdcdf87d11a611c75061f. The following files were copied:

jobs/JGDAS_ATMOS_VERFOZN
jobs/JGDAS_ATMOS_VERFRAD
jobs/JGDAS_ATMOS_VMINMON
jobs/JGFS_ATMOS_VMINMON
scripts/exgdas_atmos_verfozn.sh
scripts/exgdas_atmos_verfrad.sh
scripts/exgdas_atmos_vminmon.sh
scripts/exgfs_atmos_vminmon.sh
scripts/exglobal_atmos_vminmon.sh
ush/minmon_xtrct_costs.pl
ush/minmon_xtrct_gnorms.pl
ush/minmon_xtrct_reduct.pl
ush/ozn_xtrct.sh
ush/radmon_verf_angle.sh
ush/radmon_verf_bcoef.sh
ush/radmon_verf_bcor.sh
ush/radmon_verf_time.sh
ush/jjob_header.sh
ush/preamble.sh
ush/radmon_diag_ck.sh
ush/rstprod.sh

[ADCollard]

@aerorahul @KateFriedman-NOAA I'm now cycling with this version of the v16 workflow adapted to run the develop version of the GSI. Do you think these changes are acceptable?

Obviously I will eventually need tags for GSI, GSI-Utils and GSI-Monitor rather than checking out develop or particular hashes.

[KateFriedman-NOAA]

@ADCollard Can you pull the g-w release/gfs.v16.3.21 changes into your branch so it already has the WAFS removal updates? That'll help with comparing your branch to that release. Thanks!

[ADCollard]

@ADCollard Can you pull the g-w release/gfs.v16.3.21 changes into your branch so it already has the WAFS removal updates? That'll help with comparing your branch to that release. Thanks!

@KateFriedman-NOAA I have (hopefully successfully) merged release/gfs.v16.3.21 into this branch.