[develop] Added an option for RRFS external model files used as ICS and LBCS #1089
Conversation
natalie-perlin
requested review from
mkavulich,
gsketefian,
JeffBeck-NOAA,
RatkoVasic-NOAA,
BenjaminBlake-NOAA,
ywangwof,
chan-hoo,
panll,
christinaholtNOAA,
christopherwharrop-noaa,
danielabdi-noaa,
mark-a-potts,
jkbk2004,
willmayfield,
dmwright526,
gspetro-NOAA,
EdwardSnyder-NOAA and
MichaelLueken
as code owners
|
I'll move this work to On-Hold until a WE2E test has been added so that we can properly test this new functionality. |
MichaelLueken
added
enhancement
|
NB: @MichaelLueken - this PR requires an option "RRFS" to be allowed in UFS_UTILS. Current develop branch of ufs-community/UFS_UTILS does have the "RRFS" option enabled, but the version checked out by the SRW does not (a correction needs to be to allow it). How should we proceed with this requirement? |
|
An updated tag could be used for the UFS_UTILS that had this option implemented |
|
I'll check if updating the version of I'll go ahead and try updating the |
|
What version of |
|
It looks like |
Yes - thank you!! I was having troubles finding exact time/version when this change was implemented!. |
|
Using the version of UFS_UTILS associated with PR #902 is causing a failure in the fundamental tests: The
I'll try backing my way through the commits in the UFS_UTILS repository to see which entry is causing issues with |
|
@MichaelLueken - thank you for testing!! Let me look into these errors - look like a data problem. I might need to stage an additional directory in the EPIC space with data that I though was not needed... will get back to you! |
|
@MichaelLueken - What is the location of your test? I was not able to reproduce this error. However, I'm also replacing the explicit format statement for RRFS in the config.yaml file by the format given in the code, and doing some more changes for this PR, but not yet in GitHub. |
…e_RRFS_v1beta for RRFS functionality
removed debugging diagnostics
…test suite, among other coverage tests
Co-authored-by: Michael Lueken <63728921+MichaelLueken@users.noreply.github.com>
natalie-perlin
force-pushed
the
rrfs_ics_lbcs
branch
from
32be6c6
to
d532777
Compare
Updating the documentations files per Gillian Petro's suggestions Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>
|
@MichaelLueken @mkavulich - |
|
A new experiment has successfully completed using the updated code on Orion: All the fundamental tests passed on NOAA AWS (Centos7). One of the tests initially died due to insufficient run time for the "post" tasks; it successfully completed after the run time has been increases for these tasks: After increase of the runtime: Launching fundamental tests on Orion now |
|
Orion fundamental tests have completed successfully: |
| * Obtained and staged by the user from a different source. | ||
| * Pulled from the NOAA High Performance Storage System (:term:`HPSS`) during the workflow execution (requires user access), | ||
| * Obtained and staged by the user from a different source, or | ||
| * Pulled from the `RRFS data bucket (rrfs_a data) <https://registry.opendata.aws/noaa-rrfs/>`_. |
There was a problem hiding this comment.
I think this section is more general and the mention of RRFS specifically should not go here.
There was a problem hiding this comment.
This is included specifically because the new test case for the RRFS is not staged in standard locations, as many other WE2E experiments (data pulled from the SRW App Data Bucket). It is also a ready reference for obtaining readily available initialization data besides the HPSS or other sources ("Obtained and staged by the user from a different source"), for users external to NOAA who may not be aware of actual data sources.
The question of where to get the data to initialize the UFS/SRW model is one of the most frequently asked questions (sometimes directly sent to Maoyi), so it looks that including this info in the section on "Input and Output" is likely very beneficial to users.
There was a problem hiding this comment.
I think this still needs to be more generic. There are several different models that can be pulled by SRW from the NODD buckets. It is not just RRFS.
...uites_community/config.grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta.yaml
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
It looks like data has been removed from https://registry.opendata.aws/noaa-rrfs/ bucket for 20240605. It only has data for 00, 06, 12, and 18 hours. This is causing the new WE2E test to fail:
INFO: Retrieved files will be placed here:
/work/noaa/epic/mlueken/ufs-srweather-app/expt_dirs/grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta/2024060517/RRFS/for_ICS/mem001
DEBUG: Looking for fhr = 0
DEBUG: Looking for files like ['rrfs.t{hh}z.prslev.f{fcst_hr:03d}.conus.grib2']
DEBUG: They should be here: https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.{yyyymmdd}/{hh}/control/
INFO: Getting file: https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240605/17/control/rrfs.t17z.prslev.f000.conus.grib2
DEBUG: Target path: /work/noaa/epic/mlueken/ufs-srweather-app/expt_dirs/grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta/2024060517/RRFS/for_ICS/mem001
DEBUG: Running command:
wget -q -c -T 15 -t 2 https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240605/17/control/rrfs.t17z.prslev.f000.conus.grib2
INFO: Command 'wget -q -c -T 15 -t 2 https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240605/17/control/rrfs.t17z.prslev.f000.conus.grib2' returned non-zero exit status 8.
DEBUG: Retrieved status: False
DEBUG: Some files were not retrieved: ['https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.20240605/17/control/rrfs.t17z.prslev.f000.conus.grib2']
If the data has been staged, please change the test so that the new test will pull the data that has been staged on disk. If the data hasn't been staged, please update the DATE_FIRST_CYCL and DATE_LAST_CYCL to use one of the available cycle hours available for the date.
...uites_community/config.grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta.yaml
Show resolved
Hide resolved
Updating a description for RRFS data available from the AWS S3 NOAA-RRFS bucket Co-authored-by: Christina Holt <56881914+christinaholtNOAA@users.noreply.github.com>
…S bucket; use staged data for the new test with RRFS as ICS/LBCS
|
After all the updates, a new test grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta has successfully completed on Orion: |
...uites_community/config.grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta.yaml
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Thank you so much for working through the concerns I had with the changes in the PR!
The fundamental WE2E tests successfully passed on Orion:
----------------------------------------------------------------------------------------------------
Experiment name | Status | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2 COMPLETE 16.47
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240 COMPLETE 11.77
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot COMPLETE 42.97
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024091 COMPLETE 63.05
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240910084 COMPLETE 40.20
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024091008425 COMPLETE 76.82
----------------------------------------------------------------------------------------------------
Total COMPLETE 251.28
as well as the Orion coverage WE2E tests:
----------------------------------------------------------------------------------------------------
Experiment name | Status | Core hours used
----------------------------------------------------------------------------------------------------
custom_ESGgrid_SF_1p1km_20240910102634 COMPLETE 469.23
deactivate_tasks_20240910102636 COMPLETE 2.12
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me COMPLETE 1984.34
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_ COMPLETE 1107.76
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240 COMPLETE 418.20
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202409101 COMPLETE 28.81
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240910102 COMPLETE 1028.28
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta_ COMPLETE 24.09
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_ COMPLETE 73.74
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2 COMPLETE 808.48
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202409 COMPLETE 75.64
2020_CAD_20240910102655 COMPLETE 83.09
----------------------------------------------------------------------------------------------------
Total COMPLETE 6103.78
The new RRFS WE2E test successfully passes.
Approving PR now.
MichaelLueken
added
the
run_we2e_coverage_tests
|
The tests have been submitted, but I can relaunch the Orion tests and they should finish over night along with the rest of the automated Jenkins testing. Please go ahead and remove the |
…FS_v1beta.yaml Removing settings for "global" including ensemble size, from the new test with RRFS as ICS/LBCS
|
The Derecho and Jet coverage WE2E test suites were closed for running beyond the 8 hour time limit. These coverage tests are now being run manually on both machines. The rest of the Jenkins tests successfully passed. Once the manual runs on Derecho and Jet are complete, this PR will get merged to develop. |
|
The WE2E coverage tests successfully completed on Derecho: and Jet: Moving forward with merging this work now. |
DESCRIPTION OF CHANGES:
(updated 09/06/2024)
An option to use RRFS model output (control) files are added as initial and lateral boundary conditions, ICS and LBCS.
RRFS_a data for the test was retrieved from the NODD website ((https://registry.opendata.aws/noaa-rrfs/)), pressure-level grib2 files from the control directory, RRFS forecasts interpolated into 3-km regular grid.
A new test has been added
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1betawith RRFS input files for the event on 06/05/2024 with the tornadoes reported in Maryland.RRFS data location:
https://noaa-rrfs-pds.s3.amazonaws.com/rrfs_a/rrfs_a.{yyyymmdd}/{hh}/control/files are in the format
rrfs.t{hh}z.prslev.f{fcst_hr:03d}.conus.grib2where {yyyymmdd} are 4-digit year, 2-digit month, and 2-digit day of the forecast cycle, and {hh} is a 2-digit hour of the forecast cycle (forecast start), and {fcst_hr:03d} is a 3-digit forecast hour.
Browsing the bucket could be done at the site: browse the bucket:
https://noaa-rrfs-pds.s3.amazonaws.com/index.html#rrfs_a/
For this PR, RRFS input data uses are interpolated into a regular 3-km grid, these files need older sfs_data v1.
The sfc_data v2 that contains rotated u,v fields or fractional grids will be needed to use a newer UFS_UTILS version and tag. This would allow use of full RRFS input files, i.e. on a native grid with no remapping into regular grids; these files are ~6GB per file, and also require higher-version of packages (g2) that are not present in a spack-stack v1.5.1 or 1.6.0.
In order to bypass the limitation of currently used UFS_UTILS hash version where the the use of RRFS ICS and LBCS is not yet considered, the scripts exregional_make_ics.sh and exregional_make_lbcs.sh treat the "RRFS" option in the way similar to "HRRR", i.e.:
The following needs to added to config.yaml file to use RRFS ICS/LBCS option:
An example of a config.yaml file that is use for the test is shown below. RRFS data are accesses from the AWS.
Type of change
TESTS CONDUCTED:
Conducted a test for RRFS_CONUScompact_25km grid, setting ICS and LBCS to "RRFS" option, running on NOAA AWS cloud. One-, two-, and 3-ensemble member experiments.
A new test configured, config.grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_v1beta , which could be launched on all the platforms.
[As a recent update of Sept. 2024, ] RRFS ICS/LBCS input data for the test is retrieved from the AWS.
These data are staged in standard location for EPIC project on NOAA Tier 1 systems as well:
NOAA Cloud: /contrib/EPIC/UFS_SRW_data/develop/input_model_data/RRFS/
Derecho: /glade/work/epicufsrt/contrib/UFS_SRW_data/develop/input_model_data/RRFS/
Hera: /scratch1/NCEPDEV/nems/role.epic/UFS_SRW_data/develop/input_model_data
Gaea: /gpfs/f5/epic/world-shared/UFS_SRW_data/develop/input_model_data/RRFS/
Jet: /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/input_model_data/RRFS/
Orion/Hercules: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/input_model_data/RRFS/
A directory that uses forecast cycle date stamp for the test, ./2024060517, has 10 files:
rrfs.t17z.prslev.f000.conus.grib2
rrfs.t17z.prslev.f001.conus.grib2
rrfs.t17z.prslev.f002.conus.grib2
rrfs.t17z.prslev.f003.conus.grib2
rrfs.t17z.prslev.f004.conus.grib2
rrfs.t17z.prslev.f005.conus.grib2
rrfs.t17z.prslev.f006.conus.grib2
rrfs.t17z.prslev.f007.conus.grib2
rrfs.t17z.prslev.f008.conus.grib2
rrfs.t17z.prslev.f009.conus.grib2
DEPENDENCIES:
DOCUMENTATION:
A new option for "RRFS" used as ICS and LBCS may need to be documented.
ISSUE:
In preparation for RRFS integration tasks, option to use "RRFS" model file as ICS and LBCS was added.
CHECKLIST
LABELS (optional):
A Code Manager needs to add the following labels to this PR:
CONTRIBUTORS (optional):
@christinaholtNOAA