-
Notifications
You must be signed in to change notification settings - Fork 25
2021 MARBL Dev team meetings
Michael Levy edited this page Oct 19, 2021
·
42 revisions
- More on MOM vs POP comparison runs
- Keith pointed out iron fields didn't look right,
FESEDFLUXwas likely culprit- found unit issue in vertical component of
FESEDFLUXfile - also had issue mapping from
gx1v7->tx0.66v1because POP mask was leading to unwantedNaNvalues on MOM grid -
/glade/work/mlevy/cesm_inputdata/fesedflux_total_reduce_oxic_tx0.66v1.c211019.ncand/glade/work/mlevy/cesm_inputdata/feventflux_5gmol_tx0.66v1.c211019.ncresolve those two issues, hopefully represent reasonable forcing files
- found unit issue in vertical component of
- Max has a notebook for quick-look at MOM output
- Question (from Mike): when should we be looking at output that was remapped to static vertical coordinate (
h_bgc_*), and when should we stick with output on native vertical grid (hm_bgc_*)
- Question (from Mike): when should we be looking at output that was remapped to static vertical coordinate (
- Keith pointed out iron fields didn't look right,
- Started MOM vs POP comparison runs
-
1850_DATM%JRA-RYF0304_SLND_CICE_POP2%ECO_DROF%JRA-RYF0304_SGLC_SWAVvs1850_DATM%JRA-RYF0304_SLND_CICE_MOM6%MARBL_DROF%JRA-RYF0304_SGLC_SWAV - 20 year integration, default output
- Two POP cases (
pop_controlis completely out-of-the-box,pop_no_mcogsetslmcog = .false.inuser_nl_pop) - Two MOM cases (
mom_oobis completely out-of-the-box,mom_hybrid_zsets up hybrid z-coordinate - POP runs have both finished, been converted to time-series, and are on
campaign -
mom_oobis running ~4 SYPD, and will finish tonight or tomorrow -
mom_hybrid_zwas running <3 SYPD, and I want to find a better PE layout before letting it run - Output will end up in
/glade/campaign/cesm/development/omwg/projects/MOMMARBL_vs_POPECO
-
- cesm2-marbl update?
- Fixed a few more bugs with
bot_flux_to_tendimplementation, I think that code really is behaving the way we want - Github Actions for CI seems to be working, testing on latest Ubuntu (20.04) instead 14.04 [gfortran v9 instead of 4.8]
- Three POP updates in
cesm2_3_alpha07a(cesm2_3_beta05is about to be released, andbeta06will set default coupler to NUOPC instead of MCT; then back to regular development)- Update to latest MARBL tag (slightly answer-changing due to
other_remincalculation withbot_flux_to_tend) - Get GreenPlanet support on
mainso that Keith M's group can use latest version of model instead of CESM 2.2 offshoot I set up - Minor testing update for CGD systems (use
izumiinstead ofhobartfor machine name)
- Update to latest MARBL tag (slightly answer-changing due to
- Talked with Keith about some useful diagnostics to look at re: how carbonate solver treats bottom layer, hoping for live demo of what I have so far
- Anyone with diagnostics updates?
- Slow but steady -- looks like tracers aren't going off the rails, and good discussion on reducing some noise in tracer concentration in vanishing layers
- Have a PR (#386) that fixes a bug in
bot_flux_to_tend, I think it's ready to go - Set up #388 to look at moving from Travis CI to Github Actions
- Fixed a bug in how we were using
openmpiin the stand-alone driver while I was at it - Getting a little nervous about how many PRs have been merged since CI last ran
- Fixed a bug in how we were using
- Diag updates from Max? (NUOPC vs MCT comparison is pretty cool)
- Pinatubo
- added more vars to zonal averages
- I had set up five ensemble members for the 20th C portion with original LENS forcing; now running RCP portion (2006-2025) of those cases
- Finally happy with bottom flux to tendency code
- compute weights by hand rather than relying on
tracer_vertdiff() - need to clean up some variable names and add some comments
- compute weights by hand rather than relying on
- Next step for me is looking at NUOPC cap? Or add driver-level budget checks? (I think carbon inventory came up in a meeting last month w/ Keith and Andrew)
- I think #381 is ready to merge
- previous hold up with incrementing
max_bracket_grow_it, but newbot_flux_to_tendvalues in MARBL driver let us go back to old value - Only other change is new error message when
sum(bot_flux_to_tend*dz) - 1is not close to zero (I added some output from the column, but maybe I should include a line in the log describing what each column means)
- previous hold up with incrementing
- High-res dashboard
- Anderson's VM is back up and running (to access, you need to be on the NCAR network, either in the building or via VPN)
- Issues Keith and I ran into on Monday with panelify were resolved by merging #53
- A couple of small bugs to work out: #54 and #55
- Lots of discussion on improving user interface / experience, but at least for now the dashboard is functional
- Zonal Averager
- Max was trying to use COSIMA's xhist-based zonal averager but results don't match
za- I don't think
xhistweights each cell's contribution by how much of its area is actually in the lat band - Instead, if the cell center is in the band then it is treated as the whole cell being in the band
- I don't think
- Max was trying to use COSIMA's xhist-based zonal averager but results don't match
- I met with Alper and Gustavo a couple weeks ago to talk about MOM strategy
- NUOPC cap is pretty stable in
dev/NCARbranch, so I should be okay to update sandbox tocesm2_3_alpha05b - Still not clear why MOM doesn't run well on
izumi, but I'll let Alper figure that out sincecheyenneis up again
- NUOPC cap is pretty stable in
- High-res dashboard works
- In addition to latest
(hires-marbl)environment, requiresjupyterlab>=3,hvplot, andipywidgetsin(base)environment (if that's where you launchjupyter lab) - I can demo looking at output on webext from my laptop
- In addition to latest
- Keith and Andrew helped me iron out last issue with computing
bot_flux_to_tend(needed to enforce minimum diffusivity at bottom of columns viaMIX_BOUNDARY_TRACER_ALEparameter)- See this notebook for some plots on the effect of this parameter, but basically there was no mixing at the bottom with
MIX_BOUNDARY_TRACER_ALE=Falseso there were extremely large tracer concentrations accumulating in the bottom millimeters
- See this notebook for some plots on the effect of this parameter, but basically there was no mixing at the bottom with
- I was hoping to use
izumito test updating my branches to the latest CESM beta tag, but two issues- I can't get a
CMOM(no BGC) compset to run on that machine (nagdoesn't build FMS,intelcrashes when initializing T & S) - I forgot to copy the MARBL initial condition / forcing files off glade before the NWSC power-down
- I can't get a
- Last hold-ups on my branch are purely engineering (aforementioned need to merge latest tags; get NUOPC cap working)
- I'd like to get to the latest CESM beta, but then I'll need help with the scientific validation step
- I'd like to merge #381 (GCM provides bottom flux for redistributing sinking particulates that hit the sea floor)
- Only hold-up is that I bumped
max_bracket_grow_itfrom 3 to 5 for the iterative solver in the co2calc module before realizing MOM6 wasn't providing much diffusivity at the bottom of some columns... Keith points out that we should check if that increase is still necessary before merging, but that requires cheyenne compute nodes - Also, I'm not sure that Travis is still running the CI... not a huge issue because I can run the tests locally
- Only hold-up is that I bumped
- Once MARBL is updated, I'll need to make a new POP tag passing in the proper
bot_flux_to_tendweights (1/dzin the bottom layer, 0 elsewhere)
- High-res diagnostics
- Max, Anderson and I talked about Max's PR with the dashboard notebook (#44) yesterday
- incoming commit to that PR works on local machine and casper (via ssh tunneling), but not CISL's Jupyter Hub... need Jared to update environment for the Hub to work; this will be good for browsing images on glade
- Anderson will get the VM up and running again (there will probably be a PR to the High-Res repo first), which will be good for browsing images on webext
- Max, Anderson and I talked about Max's PR with the dashboard notebook (#44) yesterday
- Open discussion on MARBL driver in MOM
- Keith and I talked yesterday, wanted to look at vertical profiles of specific columns where vanishing layers are still giving unexpected results
- Analysis updates?
- cesm2-paper submitted, moving plot / table generation to new repo
- High-res: any update on viewing plots?
- I generated new figures for years 18 - 34, but I don't think the dashboard is still available in the cloud and I can't get it to work in Jupyter Hub
- Keith asked me to hold off on running additional years until he can look at how the run is going, but we were counting on the dashboard to allow him to navigate the new plots
- Any general topics to cover following CESM Workshop?
- Same three hold-ups with the MARBL driver in MOM
- Need more flexibility in how we read river fluxes (in progress; Keith and I are meeting to talk about it more on Thursday and I hope that's more code review and less co-programming / design)
- Issue with MARBL bottom fluxes and vanishingly thin layers
- Talked with Keith and Andrew about it, will proceed with plan to have GCM diffuse a unit bottom flux into an empty column and then passing resulting weights to MARBL to actually convert flux to tendency
- Keith found a bug in parentheses when passing in
sfc_fluxto vertical diffusion routine, so I'll need to incorporate a fix for that (and also update units of argument I'm passing to the function)
- NUOPC cap isn't passing fluxes well
- This is on hold while I address the other two issues; it sounds like there are still some NUOPC bugs being worked out, so it's been tough to figure out if the issue is in my cap or some other component.
- Question from Cheryl re: CISO + cocco in a 3-degree paleo run (tried to modify old CESM+CISO restarts that predate switch to
Ctotfor isotopes to account for addition ofDOCrto pools passed to ciso module, seeing large negative DOC values in deep ocean)
- June 15, 2021: CESM Workshop (meeting postponed a week)
- June 1, 2021: Mike out of town
- High-res
- I see we have more computer time, I need to start running again
- No updates on diagnostics from me; anyone else?
- Pinatubo
- Ran 5 additional ensemble members
- Finally seem to have Gary's
PyReshaperscripts working again, will convert to time series (and also compute zonal averages) this week
- MARBL paper
- MARBL jupyter book (Max)
-
Bottom flux generation PR
- I have verified that we are properly accounting for everything we remove from interior tendency by adding it to bottom flux
- Keith L and I think there are still terms in the interior tendency that should move to bottom fluxes, but we need feedback from Keith M
- The denitrification process is messy, and there are probably more terms to move from denitrif to bottom_fluxes(no3_ind)
- The is also the issue of scaling denitrification to prevent losing all the NO3 at the bottom
- Best way to get Keith M on a zoom call?
- Will require small update once bottom fluxes are computed correctly; have a branch where I have been doing all my testing
- Not much to report - mostly working on other projects, so only a little progress towards goals mentioned last time (I met with Keith and Andrew and have a solid plan for the river flux code)
- Time series generation tool
- Question from Max: what should the tool do with
F_PERPandF_PARAin thepop.hvfile? I assume we should just tell uses to save the history file if they want access to those two [time-invariant / 3D] fields?
- Question from Max: what should the tool do with
- Intake catalog generation update? (Max sent me this notebook from Anderson)
- Running into unexpected test failures in PR #377 (adding bottom fluxes)
- Meeting tomorrow with Keith and Andrew
- My sandbox has the first round of NUOPC updates, but there are a few more on the way (I think Chris is making a new alpha tag today)
- MARBL is close to having bottom fluxes available
- Goal for tomorrow: reading river fluxes correctly (e.g. infrastructure that sort of imitates the share-stream read, lets us read first slice, last slice, or align our run with a set of years)
- High-res diagnostics
- PR #42 (png generation) is ready for review -- preferences between asynchronous and walking through via zoom?
- Run for Amanda and Galen: changing xkw_coeff
- What value for the parameter? (
a=0.251->a=0.33_, converted froms^2/m^2 cm/hrtos/cm?) - Run details: code-base (SMYLE?), run length, compset, etc
- What value for the parameter? (
- Updated my sandbox to use latest
dev/ncar(pretty big infrastructure refactor; updated nuopc support)- Uncovered a bug introduced since
beta01, so made my first PR back to GFDL - Apparently there's now continuous integration for merges into
dev/ncarand my branch does not pass (some lines too long, maybe some doxygen issues as well) - Working on implementation to read river forcing that mimics share stream in CESM (thanks to Keith and Andrew for helping develop plan)
- Uncovered a bug introduced since
- Ready to tackle #360 (returning bottom fluxes), but looking through the code I don't know how to proceed
- Created a branch that introduces
marbl_instance%bottom_fluxes(:) - Lots of
k == kmtlogic in compute_particulate_terms - which terms for which variables should be stored in
bottom_fluxes? - Potential need to pass
bottom_fluxesto other routines as well? (e.g. ifsed_denitrifgets changed, that affects output ofcompute_denitrif()) - Discuss here? Or set up a separate meeting later this week?
- Created a branch that introduces
- High-res diagnostics
- Just a few tasks left for png-generation PR (#42; hopefully it will be finished before Jackie Robinson Day)
- Allow user-specified file size (TBD: by dimension? By pre-defined
small,medium,large?) - Fix how
isel_dictinformation is reported in the metadata; knowingisel_dict = {'z_t': 0}isn't as useful as knowingreduced_dims = {'z_t': 500.00}(or something similar)
- Allow user-specified file size (TBD: by dimension? By pre-defined
- A few other PRs are open, probably waiting for the png-generation work to be merged
- Next steps
- Refactor
utilsto clearly sort functionality into POP-specific (may not have any yet? Not sure if adjusting time to be midpoint of bounds is a CESM-wide issue or just a POP thing), CESM-specific (e.g. producing history / time series file names based on CESM conventions), and GCM-agnostic (plotting) - Option to use
intake-esmwithCaseClass(will need to talk to Anderson and Max about generating catalogs for our data in campaign / updating catalog for 004 as run progresses) - Generate more plots for 004 (lat-lon plots are just from 0001, time series runs through 0017, but we have data through 0034
- Refactor
- Just a few tasks left for png-generation PR (#42; hopefully it will be finished before Jackie Robinson Day)
- Pinatubo runs
- I still need to generate zonal averages for some fields -- should I use both the Fortran tool and the Python tool so we can compare?
- Max and I talked about the python tool yesterday, and had some questions for Keith (is it important to use
lat_aux_gridor is something like a uniform 0.25 degree spacing okay)
- Max and I talked about the python tool yesterday, and had some questions for Keith (is it important to use
- Matt: we still need to find a home for ~20 TB of the CAM output
- I need to start a new round of runs
- I still need to generate zonal averages for some fields -- should I use both the Fortran tool and the Python tool so we can compare?
- I have plumbed in river fluxes, though the time interpolation options don't currently support all the different use cases share stream can manage (have emailed Andrew and Gustavo to ask for help finding the right equivalent in MOM; have some work-arounds in mind, but none are reasonable enough to put in writing)
- Still to do before handing off code for scientific validation, roughly sorted from what I anticipate being easiest to hardest
- Add a few timers around the MARBL calls
- Update my branch to include Mariana's updates to NUOPC, then finish bringing in forcing fields in NUOPC cap
- Fix aforementioned issue reading river fluxes (or at least hardwire in the appropriate choice for whatever compset will be used for validation)
- Update MARBL to return bottom flux instead of applying it to kmt (need to get the "KMT kludge" out of the MARBL driver in MOM)
- Would like to merge PR #346: storing more data from YAML file describing MARBL parameters in python data structure, to allow us to generate tables for papers
- Previously mentioned update to how MARBL treats bottom fluxes
- Lower priority, but I'd like to address #146 (make it easier for GCM to modify MARBL domain information) once MOM validation is underway
- March 23, 2021: Matt and Mike out of town
- Hi-res stuff
- Run is still stalled
- PR #42 (generating PNGs instead of inline plots)
- I haven't done anything recently, though I do have two tasks laid out in the PR (letting user specify figure size and also Keith's suggestion about adding
*argsand**kwargsto the argument list of routines that raiseNotImplementedError) - Anderson and Max, can we talk about the dashboard? I'd like to see how this code plays with the prototype
- Still some open questions (things like image file names and JSON structure), but I think we should wait until a prototype is in place to see what is actually needed
- I haven't done anything recently, though I do have two tasks laid out in the PR (letting user specify figure size and also Keith's suggestion about adding
- Taking advantage of NWSC being down to work in stand-alone MARBL
- Kristen wants to add an option to allow zooplankton mortality to vary due to ice frac / shortwave (#374)
- Can work on issues raised by MARBL driver (setting domain variables in a single call, and I think there are a few others documented elsewhere)
- Mariana found a bug: the NUOPC driver was causing some "uninitialized memory" errors we didn't see from MCT. Keith, I'd like to set up a review for #372 soon (I still need to verify it's bit-for-bit in POP; I did check that the stand-alone test is unchanged with
cheyenne_inteland alsogfortranon my laptop
- Still making steady progress (when the machine is up); met w/ Keith and Andrew last week and now MOM's tracer module can provide vertical integrals in diagnostic output
- Still waiting for the greenlight from UCI to bring GreenPlanet support in to CESM tags
- Hi-res: Nothing on my end, anyone else?
- Pinatubo: I have some tasks to do following Feb 8th meeting but I haven't started yet
- Inching closer to being ready to review the PR to bring in MARBL driver
- Removed
_USE_MARBL_TRACERSCPP macro to mimic what was done with_USE_GENERIC_TRACER - Created
aux_mom_MARBLtest list (including aCMOMtest that does not build MARBL at all) - Spent a few days fixing bugs (mostly found by Keith) so driver runs with gnu and with
DEBUG=TRUE
- Removed
- Still need to get MARBL forcings out of NUOPC cap
- Other tasks that I don't think are necessary for initial PR
- Addressing some of the MARBL issues meant to improve MOM driver (e.g. make it easier to update
domainevery time step) - Add
ALT_CO2diagnostics to output when appropriate (when testing and whenatm_alt_co2is not the same asatm_co2) - More flexibility in forcings, such as allowing
atm_alt_co2to differ fromatm_co2
- Addressing some of the MARBL issues meant to improve MOM driver (e.g. make it easier to update
- Anything to do re: Nikki L's grad student (Mary M)? We could invite her to this meeting in two weeks to hear more about her project
- Mariana mentioned in CSEG meeting that she had come across an un-allocated memory issue in MARBL while testing the NUOPC cap in POP; will meet with her and Alper tomorrow to learn more (unclear if it's in MARBL or the POP driver)
- High-res diagnostics:
- PR #42 (producing png files instead of filling notebooks) is feature-complete but probably needs clean-up. It's no longer in draft, but I left a comment outlining some things to do (and another comment that came to mind a few days later)
- I can walk through the parent-child classes that I hinted at two weeks ago, they're fully implemented
- At this point I'd like some input on which direction to go rather than adding features that I think might be useful
- Getting "full" diagnostic output (hooray!) but things don't look so great (boo!)
- E.g. will show
STF_O2from POP and MOM6 - Current PE layout is taking ~30 minutes to run January, so a little over 4 SYPD
- Still missing some fields that need to be computed by MOM (vertical integrals, and maybe surface values?)
- E.g. will show
- My branch requires a few tweaks outside
components/mom:- Adding
-D_USE_MARBL_TRACERStoFFLAGSfor mom (incime/config/cesm/machines/config_compilers.xml) - Updating
max_num_axis_setsandmax_output_fieldsin FMS- These are namelist settings, so I just need to talk to Alper about how to modify them via MOM's
buildnml - I think it makes sense to keep
max_output_fields = 300by default, but increase it to 600 for standard runs and [something bigger] for tests that are outputting every MARBL variable
- These are namelist settings, so I just need to talk to Alper about how to modify them via MOM's
- Adding
- Movement in git and github to use
mainas the default branch name (rather thanmaster); I'd like to switch our repos as well-
marbl-ecosys/MARBL is using
stableanddevelopmentas the two main branches, so not an issue - I'd like to put together a list of our repos that would effected as well as a plan for making the switch (I think github's web interface can change the name and update open PRs, etc; users might need to update forks and would definitely need to update local clones)
-
marbl-ecosys/MARBL is using
- High-res analysis tools
- Anderson and I started talking about generating PNG plots
- Lots to still figure out but the framework is coming together; open questions include
- How should we organize files
- What metadata should we include
- How should we maintain dates (in filenames as well as metadata)
- Lots to still figure out but the framework is coming together; open questions include
- Anderson and I started talking about generating PNG plots
- Turns out MOM is already reading
marbl_inand setting parameters accordingly; I don't know why I thought that was still an open issue - Alper's framework for generating
diag_tableis easy to work with- Currently have pulled streams containing MARBL diagnostics into separate JSON file and verified output is unchanged
- Next step will be generating that separate JSON file on the fly based on MARBL's scripts
- Keith, Andrew, and I talked about refactoring the changes I made in the MCT driver so they can easily be applied to the NUOPC driver; I think that's a lower priority than getting more output into the history files, but I have a plan ready for when I have time.
- Hi-res
- Matt's working with CISL on getting us more time on Cheyenne
- Nothing new from me on the run or diagnostics. Anderson? Keith?
- LENS without Pinatubo
- I have [separate] scripts to do the following:
- Build the 20thC case and run from 1990 - 2005 (2x 8 year runs)
- Build the RCP 8.5 case and run from 2006 - 2025 (4x 5 year runs)
- Convert output to time series
- Transfer the time series output, log files, restarts (2006-01-01, used for the RCP initial conditions; 2026-01-01, end of the run), and
pop.ddiagnostic files to campaign - Sanity checks -- make sure each component / frequency has the right number of time series files on campaign, etc
- Seven ensemble members have been run, but six of them are missing some (8) daily CICE history fields
- Two of the ensemble members (including the one with the complete CICE history) are on campaign already
- Planning to run the remaining 22 members before re-running the six members to get additional output
- I'd like to verify that we have everything we need on campaign before wiping my scratch space, but I need more disk space before launching another batch of runs
- Who's the best person to talk to in order to ensure nothing is missing?
- I have [separate] scripts to do the following:
- I think the ndep climatology is being interpolated correctly, but still need to test it
- Next step will be adding flexibility in what MARBL / BGC variables are output
- Currently I've hardcoded some BGC fields into the
diag_table.yaml - Met with Gustavo, Alper, Keith, and Frank -- have a good plan for converting MARBL's
ecosys_diagnosticsfile into a seconddiag_table.yamlfile and then MOM can build full diagnostic table from two files instead of just one
- Currently I've hardcoded some BGC fields into the
- After that, need to make sure parameters set in
user_nl_marblmake it into MARBLput()calls; it's been a while, but I might still be stuck on how to read a text file in MOM so that each task can access contents line by line (e.g. how do I read the file on master and broadcast lines to other tasks?). If this is still an issue, it'll be a good thing to take up with Andrew - At that point, it would be good to have someone other than me (Keith? Gustavo?) run some tests to make sure things look okay scientifically. There will still be code clean-up for me to do, but I think that will mostly be bit-for-bit refactoring.
- Part of the code clean-up for MOM will involve updating the MARBL API to make it easier to do things like update the vertical grid column-by-column