Orion: Migration to Rocky9 OS #2694

KateFriedman-NOAA · 2024-06-17T18:10:21Z

What new functionality do you need?

Support for Rocky9 on Orion

Use: /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/gsi-addon-env-rocky9

What are the requirements for the new functionality?

System and components build/run on Orion Rocky9

Acceptance Criteria

System build and runs without issue related to OS

Suggest a solution (optional)

No response

The text was updated successfully, but these errors were encountered:

GeorgeGayno-NOAA · 2024-06-17T19:11:59Z

@KateFriedman-NOAA - UFS_UTILS will be updated under this issue: ufs-community/UFS_UTILS#963

RussTreadon-NOAA · 2024-06-18T11:07:26Z

GDASApp issue #1159 documents Orion Rocky 9 updates.

KateFriedman-NOAA · 2024-06-21T19:36:26Z

UWM issue: ufs-community/ufs-weather-model#2332

GeorgeGayno-NOAA · 2024-06-21T19:39:38Z

I see multiple stacks:

/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/modulefiles/Core
/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/gsi-addon-env-rocky9

Which should we use?

aerorahul · 2024-06-21T19:44:33Z

I see multiple stacks:

/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/modulefiles/Core

/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/gsi-addon-env-rocky9

Which should we use?

Based on
https://github.com/ufs-community/UFS_UTILS/blob/65b530560c0a1620982d1857fdb36d65be17b867/modulefiles/build.orion.intel.lua#L5
I think /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env-rocky9/install/modulefiles/Core

DavidHuber-NOAA · 2024-06-25T15:07:23Z

Opened issue NOAA-EMC/UPP#983 for the UPP.
Opened issue JCSDA/spack-stack#1158 to fix a CRTM-fix file problem in the spack-stack installation, a prerequisite for the GSI NOAA-EMC/GSI#754.

Point to the Rocky 9 spack-stack. Fixes #963. Related to NOAA-EMC/global-workflow#2694

RussTreadon-NOAA · 2024-06-26T15:35:03Z

Changes to build and run gsi.x and enkf.x on Orion Rocky 9 committed to RussTreadon-NOAA/GSI:feature/orion_rocky9. ctests show significant increase in gsi.x and enkf.x wall times. See GSI PR #764 for details.

RussTreadon-NOAA · 2024-06-28T20:16:55Z

GDASApp and GSI updates

As of GDASApp PR #1180 and GSI PR #764 build and run on Orion Rocky 9.

Note that gsi.x and enkf.x run approximately 2x slower on Orion Rocky 9 and Orion Centos 7. This slowdown has been reported via

RDHPCS ticket #2024062754000098
spack-stack issue #1166

KateFriedman-NOAA · 2024-07-01T11:47:05Z

Branches for prepobs and fit2obs:

https://github.com/KateFriedman-NOAA/prepobs/tree/feature/orion_rocky9
https://github.com/KateFriedman-NOAA/Fit2Obs/tree/feature/orion_rocky9

Installs on Orion for testing (unable to install under glopara space on Orion):

/work/noaa/global/kfriedma/git/prepobs/feature-orion_rocky9
/work/noaa/global/kfriedma/git/fit2obs/feature-orion_rocky9

Will need to update g-w modulefile path for prepobs once installed:
https://github.com/NOAA-EMC/global-workflow/blob/develop/modulefiles/module_base.orion.lua#L48

Will need to update fit2obs_ver to 1.1.2:
https://github.com/NOAA-EMC/global-workflow/blob/develop/versions/run.spack.ver#L35
...and have it installed everywhere else as well. I can assist with installs elsewhere.

DavidHuber-NOAA · 2024-07-01T13:14:44Z

The UFS weather model was updated to Rocky 9 on Orion at hash e784814.

DavidHuber-NOAA · 2024-07-01T13:19:13Z

Unfortunately, this does not include the UPP, so that will need to be incorporated at a later hash. I will help with the UPP today.

JessicaMeixner-NOAA · 2024-07-01T13:22:55Z

The UFS weather model was updated to Rocky 9 on Orion at hash e784814.

PR #2729 has the updates for this. I believe after I merge in the updates from 2700 are merged to develop and then to this PR branch the issue with the CI tests will be resolved.

DavidHuber-NOAA · 2024-07-01T13:24:54Z

The UPP is waiting on a new spack-stack installation of g2tmpl/1.12.0 (spack-stack #1164).

aerorahul · 2024-07-01T17:49:49Z

#2741 will partially address this issue.
Remaining items after #2741 will be for:
UPP @WenMeng-NOAA is aware
UFSWM #2729 is in progress

KateFriedman-NOAA · 2024-07-02T14:14:02Z

Updated prepobs for Orion Rocky9 is available here on Orion:

/work/noaa/global/kfriedma/glopara/git/prepobs/v1.0.2

Have also installed new v1.0.2 on both WCOSS2s and Jet. Will install on Hera when it's back from monthly maintenance tomorrow.
The following updates are needed within g-w:

Update these lines:
https://github.com/NOAA-EMC/global-workflow/blob/develop/modulefiles/module_base.orion.lua#L48
https://github.com/NOAA-EMC/global-workflow/blob/develop/modulefiles/module_base.hercules.lua#L46

to:

prepend_path("MODULEPATH", "/work/noaa/global/kfriedma/glopara/git/prepobs/v1.0.2/modulefiles")

Update prepobs_run_ver=1.0.2 here:
https://github.com/NOAA-EMC/global-workflow/blob/develop/versions/run.spack.ver#L32

KateFriedman-NOAA · 2024-07-02T14:31:15Z

Updated Fit2Obs install on Orion is available here: /work/noaa/global/kfriedma/glopara/git/Fit2Obs/v1.1.2

Have also installed in official global group account spaces on both WCOSS2s and Jet. Will install on Hera when it is back from monthly maintenance.

For this issue the following changes are needed in g-w:

Temporarily change these lines to use the Orion install in my space ahead of Walter's return:

https://github.com/NOAA-EMC/global-workflow/blob/develop/modulefiles/module_base.orion.lua#L51
https://github.com/NOAA-EMC/global-workflow/blob/develop/modulefiles/module_base.hercules.lua#L49

Fit2Obs version needs to be updated to fit2obs_ver=1.1.2 here:

https://github.com/NOAA-EMC/global-workflow/blob/develop/versions/run.spack.ver#L35

DavidHuber-NOAA · 2024-07-02T16:18:21Z

UPP PR: NOAA-EMC/UPP#987
UFS PR that would be nice to include the UPP into: ufs-community/ufs-weather-model#2326

KateFriedman-NOAA · 2024-07-08T16:44:43Z

The new prepobs/v1.0.2 also ran successfully on Orion.

Awaiting cycled tests on Hera/WCOSS2/Orion to get far enough to invoke Fit2Obs.

- set the prepobs MODULEPATH to where the install will be when set back to official glopara space Refs NOAA-EMC#2694

RussTreadon-NOAA · 2024-07-09T10:29:45Z

GSI update
dclock timers where added to a copy of GSI develop at 529bb796. The code was compiled on Dogwood, Hera, Hercules, and Orion. The global_4denvar ctest was run on each machine. Tabulated below are key timings

machine	total `read_obs` time (s)	`read_satwnd` time (s)	total `gsi.x` wall time (s)
dogwood	169.988	120.264	403.800888
hera	194.612	137.553	466.024682
hercules	164.571	116.393	444.405860
orion	608.372	435.489	964.591370

Comparison of timings across the machines show timings for Hercules, Hera, and Dogwood are comparable. Orion is the outlier. Reading bufr files takes significantly longer on Orion following the Rocky 9 upgrade. The reason(s) for this increased wall time remain, at present, unknown.

For additional details see

GSI issue #771
spack-stack issue #1166

Refs NOAA-EMC#2694

KateFriedman-NOAA · 2024-07-11T13:49:36Z

Short cycled atmos-only low-res testing of new prepobs and fit2obs versions have completed successfully on WCOSS2-Dogwood, Hera, Orion, and Hercules. Will initiate PR to update prepobs and fit2obs versions in develop.

RussTreadon-NOAA · 2024-07-11T13:54:02Z

@KateFriedman-NOAA , where are log files from the Hera, Orion, and Hercules tests? I would like to compare wall times across the machines. GSI ctests indicate that gsi.x runs noticeably slower on Orion (GSI issue #771). debufr tests also show this executable to run slower on Orion (NCEPLIBS-bufr issue #608).

Refs NOAA-EMC#2694

KateFriedman-NOAA · 2024-07-11T14:13:56Z

@RussTreadon-NOAA Yes, please see the following:

Hera:

/scratch1/NCEPDEV/global/Kate.Friedman/expdir/testprepobs
/scratch2/NCEPDEV/stmp1/Kate.Friedman/comrot/testprepobs

Orion:

/work/noaa/global/kfriedma/expdir/testprepobs
/work/noaa/stmp/kfriedma/comrot/testprepobs

Hercules:

/work/noaa/global/kfriedma/expdir/testprepobsherc
/work/noaa/stmp/kfriedma/comrot/testprepobsherc

I also ran this same test on WCOSS2-Dogwood if that would be helpful.

I had to increase the eobs walltime on Orion (15mins->30mins) but otherwise left walltimes as is and did not have any other timelimit failures.

KateFriedman-NOAA · 2024-07-11T14:18:47Z

@RussTreadon-NOAA In case it's helpful, here is the WCOSS2-Dogwood test:

/lfs/h2/emc/global/noscrub/kate.friedman/expdir/devcycprepobs
/lfs/h2/emc/ptmp/kate.friedman/comrot/devcycprepobs

RussTreadon-NOAA · 2024-07-11T15:18:03Z

Thank you @KateFriedman-NOAA .

gsi.x wall times are considerably higher on Orion compared to other platforms.

Orion

orion-login-2:/work/noaa/stmp/kfriedma/comrot/testprepobs/logs$ grep "wall" */gdasanal.log
2021122100/gdasanal.log: 0: The total amount of wall time                        = 1908.889531
2021122106/gdasanal.log: 0: The total amount of wall time                        = 2038.964334
2021122112/gdasanal.log: 0: The total amount of wall time                        = 1848.529683
2021122118/gdasanal.log: 0: The total amount of wall time                        = 1904.650164
2021122200/gdasanal.log: 0: The total amount of wall time                        = 1883.218843
orion-login-2:/work/noaa/stmp/kfriedma/comrot/testprepobs/logs$ grep "wall" */enkfgdaseobs.log
2021122100/enkfgdaseobs.log: 0: The total amount of wall time                        = 901.077139
2021122106/enkfgdaseobs.log: 0: The total amount of wall time                        = 884.285354
2021122112/enkfgdaseobs.log: 0: The total amount of wall time                        = 933.364648
2021122118/enkfgdaseobs.log: 0: The total amount of wall time                        = 1084.572618
2021122200/enkfgdaseobs.log: 0: The total amount of wall time                        = 983.077523

Hera

Hera(hfe03):/scratch2/NCEPDEV/stmp1/Kate.Friedman/comrot/testprepobs/logs$ grep wall */gdasanal.log
2021122100/gdasanal.log: 0: The total amount of wall time                        = 1116.593922
2021122106/gdasanal.log: 0: The total amount of wall time                        = 1293.681421
2021122112/gdasanal.log: 0: The total amount of wall time                        = 1131.990532
2021122118/gdasanal.log: 0: The total amount of wall time                        = 1168.874366
2021122200/gdasanal.log: 0: The total amount of wall time                        = 1103.230691
Hera(hfe03):/scratch2/NCEPDEV/stmp1/Kate.Friedman/comrot/testprepobs/logs$ grep "wall" */enkfgdaseobs.log
2021122100/enkfgdaseobs.log: 0: The total amount of wall time                        = 407.097608
2021122106/enkfgdaseobs.log: 0: The total amount of wall time                        = 432.787526
2021122112/enkfgdaseobs.log: 0: The total amount of wall time                        = 434.973986
2021122118/enkfgdaseobs.log: 0: The total amount of wall time                        = 532.029862
2021122200/enkfgdaseobs.log: 0: The total amount of wall time                        = 443.266375

Interestingly Orion, Hercules, and WCOSS2 (Dogwood) run the prep step with 4 streams. Hera runs with 1 stream. Thus, Orion prep step wall times are compared with Hercules and Dogwood. Wall times for various executables in the 2021122100 gfsprep job have larger wall times on Orion compared to Hercules and Dogwood. This comment applies to gdasprep and other cycles.

Orion

orion-login-2:/work/noaa/stmp/kfriedma/comrot/testprepobs/logs$ grep "wall" 2021122100/gfsprep.log
The total amount of wall time                        = 2.378960
The total amount of wall time                        = 58.736572
The total amount of wall time                        = 0.683556
The total amount of wall time                        = 2.369928
The total amount of wall time                        = 59.592429
The total amount of wall time                        = 0.772289
The total amount of wall time                        = 2.356385
The total amount of wall time                        = 58.627895
The total amount of wall time                        = 0.618208
The total amount of wall time                        = 2.378561
The total amount of wall time                        = 59.753184
The total amount of wall time                        = 0.767621
The total amount of wall time                        = 1.102821
The total amount of wall time                        = 72.996897
The total amount of wall time                        = 6.940320
The total amount of wall time                        = 0.701914
The total amount of wall time                        = 1.628848
The total amount of wall time                        = 115.640379

Hercules

hercules-login-3:/work/noaa/stmp/kfriedma/comrot/testprepobsherc/logs$ grep "wall" 2021122100/gfsprep.log
The total amount of wall time                        = 1.891698
The total amount of wall time                        = 44.080027
The total amount of wall time                        = 0.417157
The total amount of wall time                        = 1.888169
The total amount of wall time                        = 44.332188
The total amount of wall time                        = 0.416644
The total amount of wall time                        = 1.905027
The total amount of wall time                        = 44.460939
The total amount of wall time                        = 0.397002
The total amount of wall time                        = 1.885472
The total amount of wall time                        = 45.027571
The total amount of wall time                        = 0.226408
The total amount of wall time                        = 0.437389
The total amount of wall time                        = 21.519730
The total amount of wall time                        = 2.531554
The total amount of wall time                        = 0.176123
The total amount of wall time                        = 0.583526
The total amount of wall time                        = 42.280456

WCOSS2 (Dogwood)

russ.treadon@dlogin05:/lfs/h2/emc/ptmp/kate.friedman/comrot/devcycprepobs/logs> grep "wall" 2021122100/gfsprep.log                  
The total amount of wall time                        = 0.936697
The total amount of wall time                        = 23.490267
The total amount of wall time                        = 0.217773
The total amount of wall time                        = 2.307517
The total amount of wall time                        = 24.180469
The total amount of wall time                        = 0.219624
The total amount of wall time                        = 2.280056
The total amount of wall time                        = 24.032663
The total amount of wall time                        = 0.212903
The total amount of wall time                        = 0.984444
The total amount of wall time                        = 23.692733
The total amount of wall time                        = 0.217461
The total amount of wall time                        = 1.772865
The total amount of wall time                        = 20.959409
The total amount of wall time                        = 2.440648
The total amount of wall time                        = 0.141279
The total amount of wall time                        = 0.587792
The total amount of wall time                        = 40.554083

KateFriedman-NOAA · 2024-07-11T17:18:32Z

Thanks for comparing these walltimes @RussTreadon-NOAA ! Interesting that the prep jobs also run slower on Orion. I'm guessing it's a similar reason to the GSI slowness.

RussTreadon-NOAA · 2024-07-11T17:34:33Z

@KateFriedman-NOAA , issue #2759 reports an interesting finding from your Hera and Hercules parallels. No metplus files are generated. The Dogwood test generates metplus files. Shouldn't this be true for the Hera and Hercules parallels?

KateFriedman-NOAA · 2024-07-11T18:17:32Z

@KateFriedman-NOAA , issue #2759 reports an interesting finding from your Hera and Hercules parallels. No metplus files are generated. The Dogwood test generates metplus files. Shouldn't this be true for the Hera and Hercules parallels?

Good catch @RussTreadon-NOAA ! We'll look into it.

RussTreadon-NOAA · 2024-07-11T18:40:28Z

@KateFriedman-NOAA , issue #2759 reports an interesting finding from your Hera and Hercules parallels. No metplus files are generated. The Dogwood test generates metplus files. Shouldn't this be true for the Hera and Hercules parallels?

Good catch @RussTreadon-NOAA ! We'll look into it.

See g-w issue #2759 for an update - found a way to generate stats but not the correct solution.

KateFriedman-NOAA · 2024-07-16T19:39:30Z

EMC_verif-global was updated for Orion Rocky9. See NOAA-EMC/EMC_verif-global#127.

Update global-workflow to use new hash: NOAA-EMC/EMC_verif-global@df296f4

malloryprow · 2024-07-17T13:55:37Z

EMC_verif-global was updated for Orion Rocky9. See NOAA-EMC/EMC_verif-global#127.

Update global-workflow to use new hash: NOAA-EMC/EMC_verif-global@df296f4

@KateFriedman-NOAA Could you update the EMC_verif-global hash again? NOAA-EMC/EMC_verif-global@0d9e0b6

KateFriedman-NOAA · 2024-07-17T14:02:17Z

WIll do, thanks for the updated hash @malloryprow !

Refs NOAA-EMC#2694

KateFriedman-NOAA · 2024-07-17T17:16:14Z

@DavidHuber-NOAA is testing the latest EMC_verif-global hash to resolve issue #2759. That hash also includes updates for Orion Rocky9, so it will take care of both things in a single PR. FYI @aerorahul

KateFriedman-NOAA added feature New feature or request triage Issues that are triage labels Jun 17, 2024

This was referenced Jun 17, 2024

Orion: Migration to Rocky9 NOAA-EMC/prepobs#29

Closed

Orion: Migration to Rocky9 NOAA-EMC/Fit2Obs#24

Closed

WalterKolczynski-NOAA removed the triage Issues that are triage label Jun 17, 2024

RussTreadon-NOAA mentioned this issue Jun 20, 2024

Update for JCB policies and stage DA job files with Jinja2-templates #2700

Merged

2 tasks

GeorgeGayno-NOAA mentioned this issue Jun 24, 2024

Port to Orion Rocky 9 ufs-community/UFS_UTILS#966

Merged

3 tasks

GeorgeGayno-NOAA added a commit to ufs-community/UFS_UTILS that referenced this issue Jun 26, 2024

Port to Orion Rocky 9 (#966)

3ef2e6b

Point to the Rocky 9 spack-stack. Fixes #963. Related to NOAA-EMC/global-workflow#2694

RussTreadon-NOAA mentioned this issue Jun 28, 2024

Updates to build and run on Orion Rocky 9 NOAA-EMC/GSI#764

Merged

4 tasks

This was referenced Jul 1, 2024

Updates for Rocky9 OS on MSU Orion NOAA-EMC/prepobs#30

Merged

Updates for Orion Rocky9 OS NOAA-EMC/Fit2Obs#25

Merged

KateFriedman-NOAA mentioned this issue Jul 2, 2024

Update (partially) global-workflow for orion+rocky9 #2741

Merged

7 tasks

RussTreadon-NOAA mentioned this issue Jul 3, 2024

investigation of unexpected behavior of ctest rrfs_3denvar_rdasens NOAA-EMC/GSI#766

Open

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jul 8, 2024

Update commented out prepobs MODULEPATH for Orion

e3f7297

- set the prepobs MODULEPATH to where the install will be when set back to official glopara space Refs NOAA-EMC#2694

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jul 11, 2024

Update prepobs/fit2obs paths for Orion Rocky9

bb34783

Refs NOAA-EMC#2694

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jul 11, 2024

Update prepobs install path for new v1.0.2

1540921

Refs NOAA-EMC#2694

KateFriedman-NOAA mentioned this issue Jul 11, 2024

Updated prepobs and fit2obs versions for Orion Rocky9 #2758

Merged

RussTreadon-NOAA mentioned this issue Jul 11, 2024

gfsmetp jobs do not generate stats files on Hera and Hercules #2759

Closed

KateFriedman-NOAA mentioned this issue Jul 16, 2024

Update Orion modulefile after Rocky 8 upgrade NOAA-EMC/EMC_verif-global#127

Closed

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jul 17, 2024

Update EMC_verif-global hash to 0d9e0b6a

c14ebee

Refs NOAA-EMC#2694

This was referenced Jul 17, 2024

Set METplus process count in config.metp; add verif-global support for Rocky 9 #2774

Merged

Test Orion and Update Doxygen Path #2785

Closed

CoryMartin-NOAA mentioned this issue Aug 16, 2024

Use spack-stack python on Hercules NOAA-EMC/GDASApp#1133

Closed

DavidHuber-NOAA self-assigned this Aug 29, 2024

DavidHuber-NOAA mentioned this issue Aug 29, 2024

Reenable Orion Cycling Support #2877

Merged

19 tasks

WalterKolczynski-NOAA closed this as completed in #2877 Sep 7, 2024

WalterKolczynski-NOAA closed this as completed in ac93a9b Sep 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Orion: Migration to Rocky9 OS #2694

Orion: Migration to Rocky9 OS #2694

KateFriedman-NOAA commented Jun 17, 2024

GeorgeGayno-NOAA commented Jun 17, 2024

RussTreadon-NOAA commented Jun 18, 2024

KateFriedman-NOAA commented Jun 21, 2024

GeorgeGayno-NOAA commented Jun 21, 2024

aerorahul commented Jun 21, 2024 •

edited

Loading

DavidHuber-NOAA commented Jun 25, 2024

RussTreadon-NOAA commented Jun 26, 2024 •

edited

Loading

RussTreadon-NOAA commented Jun 28, 2024

KateFriedman-NOAA commented Jul 1, 2024 •

edited

Loading

DavidHuber-NOAA commented Jul 1, 2024

DavidHuber-NOAA commented Jul 1, 2024

JessicaMeixner-NOAA commented Jul 1, 2024 •

edited

Loading

DavidHuber-NOAA commented Jul 1, 2024

aerorahul commented Jul 1, 2024

KateFriedman-NOAA commented Jul 2, 2024

KateFriedman-NOAA commented Jul 2, 2024

DavidHuber-NOAA commented Jul 2, 2024

KateFriedman-NOAA commented Jul 8, 2024

RussTreadon-NOAA commented Jul 9, 2024

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 11, 2024 •

edited

Loading

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 16, 2024

malloryprow commented Jul 17, 2024

KateFriedman-NOAA commented Jul 17, 2024

KateFriedman-NOAA commented Jul 17, 2024

Orion: Migration to Rocky9 OS #2694

Orion: Migration to Rocky9 OS #2694

Comments

KateFriedman-NOAA commented Jun 17, 2024

What new functionality do you need?

What are the requirements for the new functionality?

Acceptance Criteria

Suggest a solution (optional)

GeorgeGayno-NOAA commented Jun 17, 2024

RussTreadon-NOAA commented Jun 18, 2024

KateFriedman-NOAA commented Jun 21, 2024

GeorgeGayno-NOAA commented Jun 21, 2024

aerorahul commented Jun 21, 2024 • edited Loading

DavidHuber-NOAA commented Jun 25, 2024

RussTreadon-NOAA commented Jun 26, 2024 • edited Loading

RussTreadon-NOAA commented Jun 28, 2024

KateFriedman-NOAA commented Jul 1, 2024 • edited Loading

DavidHuber-NOAA commented Jul 1, 2024

DavidHuber-NOAA commented Jul 1, 2024

JessicaMeixner-NOAA commented Jul 1, 2024 • edited Loading

DavidHuber-NOAA commented Jul 1, 2024

aerorahul commented Jul 1, 2024

KateFriedman-NOAA commented Jul 2, 2024

KateFriedman-NOAA commented Jul 2, 2024

DavidHuber-NOAA commented Jul 2, 2024

KateFriedman-NOAA commented Jul 8, 2024

RussTreadon-NOAA commented Jul 9, 2024

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 11, 2024 • edited Loading

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 11, 2024

RussTreadon-NOAA commented Jul 11, 2024

KateFriedman-NOAA commented Jul 16, 2024

malloryprow commented Jul 17, 2024

KateFriedman-NOAA commented Jul 17, 2024

KateFriedman-NOAA commented Jul 17, 2024

aerorahul commented Jun 21, 2024 •

edited

Loading

RussTreadon-NOAA commented Jun 26, 2024 •

edited

Loading

KateFriedman-NOAA commented Jul 1, 2024 •

edited

Loading

JessicaMeixner-NOAA commented Jul 1, 2024 •

edited

Loading

KateFriedman-NOAA commented Jul 11, 2024 •

edited

Loading