-
Notifications
You must be signed in to change notification settings - Fork 168
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Convert staging job to python and yaml (#2651)
This PR converts the staging job from shell to python and introduces the use of yaml. Changes in this PR: 1. Rename `scripts/exglobal_stage_ic.sh` to `scripts/exglobal_stage_ic.py`. 2. Update `jobs/JGLOBAL_STAGE_IC` to use `.py` script extension. Move `COM*` variable declarations and member loop down into yaml and python respectively. Move `GDATE/gPDY/gcyc` settings up to JJOB from ex-script and replace with newer cycle variables (as done in forecast job). 3. Create `parm/stage` folder to hold newly created `stage.yaml.j2`, which both mimics forecast-only functionality in existing `scripts/exglobal_stage_ic.sh` and adds functionality for cycled mode. 4. Create `ush/python/pygfs/task/stage.py` to house staging job python functions for call from `scripts/exglobal_stage_ic.py`. 5. Remove `stage_ic` job rocoto dependencies from xml. Do not need and removes area of duplicate maintenance. 6. Add cycled staging jobs for gdas and enkf suites. 7. Rename `model_data` to `model` for issue #2686 There will now be distinct `stage_ic` jobs for each `RUN`: `gdasstage_ic`, `gfsstage_ic`, `enkfgdasstage_ic`, `stage_ic` (for gefs). Related work was done to set up new symlink folder structure under supported platform `ICSDIR` folder for use by updated staging job. Resolves #2475 Resolves #2650 Resolves #2686 --------- Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov> Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>
- Loading branch information
4 people
authored
Aug 20, 2024
1 parent
8772e61
commit 659bcbe
Showing
43 changed files
with
765 additions
and
662 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| {% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS_MEM_list[0] | relpath(ROTDIR)) %} | ||
| analysis: | ||
| mkdir: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_ANALYSIS_MEM = COMOUT_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| - "{{ COMOUT_ATMOS_ANALYSIS_MEM }}" | ||
| {% endfor %} | ||
| copy: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_ANALYSIS_MEM = COMOUT_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| {% for ftype in ["abias", "abias_air", "abias_int", "abias_pc", "atminc.nc", "radstat", "ratminc.nc"] %} | ||
| {% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %} | ||
| - ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS_MEM }}"] | ||
| {% endif %} | ||
| {% endfor %} | ||
| {% endfor %} # mem loop | ||
| {% endif %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| atmosphere_cold: | ||
| mkdir: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_INPUT_MEM = COMOUT_ATMOS_INPUT_MEM_list[imem] %} | ||
| - "{{ COMOUT_ATMOS_INPUT_MEM }}" | ||
| {% endfor %} # mem loop | ||
| copy: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_INPUT_MEM = COMOUT_ATMOS_INPUT_MEM_list[imem] %} | ||
| - ["{{ ICSDIR }}/{{ COMOUT_ATMOS_INPUT_MEM | relpath(ROTDIR) }}/gfs_ctrl.nc", "{{ COMOUT_ATMOS_INPUT_MEM }}"] | ||
| {% for ftype in ["gfs_data", "sfc_data"] %} | ||
| {% for ntile in range(1, ntiles + 1) %} | ||
| - ["{{ ICSDIR }}/{{ COMOUT_ATMOS_INPUT_MEM | relpath(ROTDIR) }}/{{ ftype }}.tile{{ ntile }}.nc", "{{ COMOUT_ATMOS_INPUT_MEM }}"] | ||
| {% endfor %} # ntile | ||
| {% endfor %} # ftype | ||
| {% endfor %} # mem loop |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| atmosphere_nest: | ||
| {% set ntile = 7 %} | ||
| {% if EXP_WARM_START == True %} | ||
| mkdir: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_RESTART_PREV_MEM = COMOUT_ATMOS_RESTART_PREV_MEM_list[imem] %} | ||
| - "{{ COMOUT_ATMOS_RESTART_PREV_MEM }}" | ||
| {% endfor %} # mem loop | ||
| copy: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_RESTART_PREV_MEM = COMOUT_ATMOS_RESTART_PREV_MEM_list[imem] %} | ||
| {% for ftype in ["ca_data", "fv_core.res", "fv_srf_wnd.res", "fv_tracer.res", "phy_data", "sfc_data"] %} | ||
| - ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART_PREV_MEM | relpath(ROTDIR) }}/{{ m_prefix }}.{{ ftype }}.tile{{ ntile }}.nc", "{{ COMOUT_ATMOS_RESTART_PREV_MEM }}/{{ m_prefix }}.{{ ftype }}.nest0{{ ntile-5 }}.tile{{ ntile }}.nc"] | ||
| {% endfor %} | ||
| {% endfor %} # mem loop | ||
| {% else %} # cold start | ||
| mkdir: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_INPUT_MEM = COMOUT_ATMOS_INPUT_MEM_list[imem] %} | ||
| - "{{ COMOUT_ATMOS_INPUT_MEM }}" | ||
| {% endfor %} # mem loop | ||
| copy: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_INPUT_MEM = COMOUT_ATMOS_INPUT_MEM_list[imem] %} | ||
| {% for ftype in ["gfs_data", "sfc_data"] %} | ||
| - ["{{ COMOUT_ATMOS_INPUT_MEM }}/{{ ftype }}.tile{{ ntile }}.nc", "{{ COMOUT_ATMOS_INPUT_MEM }}/{{ ftype }}.nest0{{ ntile-5 }}.tile{{ ntile }}.nc"] | ||
| {% endfor %} | ||
| {% endfor %} # mem loop | ||
| {% endif %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| atmosphere_perturbation: | ||
| mkdir: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_ANALYSIS_MEM = COMOUT_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| - "{{ COMOUT_ATMOS_ANALYSIS_MEM }}" | ||
| {% endfor %} # mem loop | ||
| copy: | ||
| {% for mem in range(first_mem, last_mem + 1) %} | ||
| {% set imem = mem - first_mem %} | ||
| {% set COMOUT_ATMOS_ANALYSIS_MEM = COMOUT_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| - ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ m_prefix }}.fv3_perturbation.nc", "{{ COMOUT_ATMOS_ANALYSIS_MEM }}/{{ RUN }}.t{{ current_cycle_HH }}z.atminc.nc"] | ||
| {% endfor %} # mem loop |
Oops, something went wrong.