AK K64G Example Ensemble Simulation
"Paging Dr. Kenny Loggins! You are entering the Danger Zone!"
This advanced demo illustrates how to execute a single-pixel ELM v2 simulation at an NGEE-Arctic study site (i.e. Utqiagvik, Council, Kougarok, Teller). This example illustrates how to configure, run, edit then re-run ELM ensemble simulations. We will start with the Kougarok site but the same approach can be used at any of the four main sites (i.e. Utqiagvik, Council, Kougarok, Teller). The initial demo will demonstrate how to run a shorted ensemble experiment as well as how to setup a proper ensemble simulation that you can run at a later time. As usual, the simulations are controlled by a wrapper script that utilizes the Offline Land Model Testbed, developed and maintained by Dan Ricciuto (ricciutodm@ornl.gov). OLMT controls the creation, configuration, and execution of each ensemble simulation. At the completion of each simulation, the wrapper calls the "ncrcat" command to concatenate all transient simulation years and model variables into a new output file called "ELM_output.nc" within each ensemble member directory. Once complete, the ensemble output will be analyzed in a Jupyter notebook using the provided Jupyter-Lab container environment.
Part 1: Running a short ensemble simulation to demonstrate setting up, running, and viewing ensemble output
First, you will need to download the ELM container designed to run ensembles. You can copy and paste the following code in your terminal window
docker pull fasstsimulation/elm-builds:elm_v2-for-ngee-serial_multiarch
To perform a quick ensemble, you can copy and paste the following code in your terminal window
docker run -t -i --hostname=docker --user modeluser -v elmdata:/inputdata -v elmoutput:/output fasstsimulation/elm-builds:elm_v2-for-ngee-serial_multiarch /scripts/OLMT_ens_docker_example.sh --site_name=kougarok --ad_spinup_years=10 --final_spinup_years=20 --transient_years=5 --num_ens=6 --num_groups=3
This will execute an ensemble with the following options:
case_prefix=OLM_ens (default, can be changed. We will explore changing this below)
site_name=kougarok
ad_spinup_years=10
final_spinup_years=20
transient_years=5
num_ens=6
num_groups=2
This translates to 10 years of accelerated decomposition spinup, 20 years of standard spinup, and a 5 (1850-1855) year transient simulation per ensemble member. The additional options are the number of parameter ensembles (6) and the number of MPI groups (2) to run these cases.
The default file to control parameter options (param_list_example_kougarok) is located in the /scripts folder in the ELM container. We can view this later, for now here is a copy of what that file contains
flnr 11 0.07 0.145
flnr 12 0.07 0.145
vcmaxha 11 65000 75000
vcmaxha 12 65000 75000
jmaxha 11 45000 57000
jmaxha 12 45000 57000
roota_par 11 4.0 10
roota_par 12 6 15
mbbopt 11 1 13
mbbopt 12 1 13
slatop 11 0.009 0.045
slatop 12 0.009 0.045
Once you start the ensemble simulations and after the model builds and the cases start executing, you will see a new "UQ" folder show up in your "elmoutput" docker volume. Inside you will see the three core case groups show up as the execute, AD spinup, spinup and transient
You may also note some warning messages during the model execution that are safe to ignore, e.g.
When the simulation completes you will see a message similar to this show up in your terminal screen
You will also see that the transient directory in your elmoutput/UQ folder will contain 6 ensemble members, as well as a copy of the file that controlled the parameter perturbation experiment used in this short demo
If you have not already done so, create a new docker volume called "elmwork" either in the Docker Desktop app or by typing docker volume create elmwork in your open terminal
To view the output from the initial ensemble output, you need to start the elmlab_3.3.2 container (if it's not already running on your host machine). If you plan to edit the ensemble run and restart, you should open the elmlab_3.3.2 container with the additional elmwork volume mapped to the internal container directory "work"
docker run -p 8888:8888 -v elmdata:/home/jovyan/inputdata -v elmoutput:/home/jovyan/output -v elmwork:/home/jovyan/work/ fasstsimulation/fasst_simulation_tools:elmlab_3.3.2
Once the container loads and you open the browser tab to enter into the Jupyter-Lab environment, you can navigate to the output location fo the new ensemble run. Do this by opening the output folder in the directory tree on the left
You can then navigate to the ensemble output by going to /output/cime_run_dirs/UQ/
The transient results are in OLMT_ens_AK-K64G_ICB20TRCNPRDCTCBC, where you will see a series of sub-directories (e.g. g00001, g00002, ...)
You will also see the parameter settings file used for the ensemble simulation.
To plot the results of this first ensemble simulation, navigate back to home by clicking the folder icon in the path browser
Once back, navigate into the scripts folder and open the "plot_ELM_ensembles.ipynb" notebook
Once you open the "plot_ELM_ensembles.ipynb" notebook, you can start analyzing the ensemble simulation results. Note that this notebook has an option to output the figures as a png and pdf file in the work/figures folder. The default behavior is to save the figure files, you can turn this off by changing generate_plots=False, or changing "True" to "False" (with the capital F)
Also note that like the other notebooks, you will select which simulation output you want to review. You do this in step 3 where you select the ensemble output folder you wish to review, e.g.
And then in step 4 you will see a list of all ensemble sub-directories printed to the screen
Once you select the output you wish to plot, in step 5 the notebook will load the output from each sub-directory into new objects labeled "output_0, output_1, ...:"
Using these output objects, a series of standardized plots will then generate ensemble plots including all ensemble member output read into Python in step 5, for example a plot of Gross Primary Productivity (GPP)
Using this provided notebook, you can modify the plots, add new variables or change the way the ensemble results are plotted. Just dont forget to save a copy of your new notebook to your host computer before logging out of the Jupyter-Lab environment!
You can revise the previous simulation experiment by increasing the length of the spinup and transient runs. For example, you could modify the spinup and transient run length. The below example will run a 200 year AD and 600 final spinup, and simulate then full transient period (1850-2014) for a similar 6-ensemble simulation
docker run -t -i --hostname=docker --user modeluser -v elmdata:/inputdata -v elmoutput:/output fasstsimulation/elm-builds:elm_v2-for-ngee-serial_multiarch /scripts/OLMT_ens_docker_example.sh --site_name=kougarok --ad_spinup_years=200 --final_spinup_years=600 --num_ens=6 --num_groups=3
You may also want to experiment with changing the parameter options, including what ELM parameters are considered in the perturbation experiment, what PFTs to target for each site, as well as the parameter ranges. To do this, if not already running, start the jupyter-lab container environment. However in this case you want to also include the "elmwork" volume, e.g.
docker run -p 8888:8888 -v elmdata:/home/jovyan/inputdata -v elmoutput:/home/jovyan/output -v elmwork:/home/jovyan/work/ fasstsimulation/fasst_simulation_tools:elmlab_3.3.2
After you have run the default example ensemble case you can copy the file that controls the parameter options from that case directory to the "work" directory in the jupyter-lab environment. To do this navigate, go to the output/cime_run_dirs/UQ/OLMT_ens_AK-K64G_ICB20TRCNPRDCTCBC and copy the file labeled param_list_example_kougarok Go back to the root of the "work" directory and paste the file
Once you copy the param_list_example_kougarok file to "work" you can open the file in the viewer by "double-clicking" the file. It will open in a new tab on the right, e.g.
You may want to right-click and "duplicate" this file, and rename it e.g. param_list_example_kougarok_v2 so that you can keep a copy of the original file as well as your modified files.
You can also add new parameters to the ensemble experiment. For example, you could add in some parameters that control canopy radiation transfer, e.g.
Once you have made the edits you want to include in your new ensemble run, make sure to save the new parameter file either by going to "File/Save File" or by doing command+s or ctrl+s on Mac or Windows, respectively
After you have saved your new parameter file in the /work directory, you can start a new ensemble simulation using that edited file. For example, to start a new ensemble simulation using a file called param_list_example_kougarok_v2 you can do:
docker run -t -i --hostname=docker --user modeluser -v elmdata:/inputdata -v elmoutput:/output \
-v elmwork:/example_inputs fasstsimulation/elm-builds:elm_v2-for-ngee-serial_multiarch /scripts/OLMT_ens_docker_example.sh \
--site_name=kougarok --case_prefix=OLMT_ens2 --ad_spinup_years=10 --final_spinup_years=20 \
--transient_years=5 --num_ens=6 --num_groups=2 --param_list=/example_inputs/param_list_example_kougarok_v2
where we have the options: case_prefix=OLMT_ens2 (to save the results in a new ensemble folder and not overwrite the previous simulation) site_name=kougarok ad_spinup_years=10 final_spinup_years=20 transient_years=5 num_ens=6 num_groups=2 param_list=/example_inputs/param_list_example_kougarok_v2
Importantly the --param_list=/example_inputs/param_list_example_kougarok_v2 flag locates the file you just edited and uses it for the ensemble experiment
Once you hit enter and start the new run, you will see the new cases build and the ensemble members execute the simulations. Once complete you can go back into your Jupyter-Lab environment and you should see the new output path as an option in the dropdown menu. Using this new set of ensemble outputs, you should be able to create new plots of the results. If you save the figures, you can also compare the new run with the old
Using this same approach, we can also re-create the earlier SLA and flnr parameter experiments but using the OLMT parameter ensemble perturbation capabilities. In this case we will make another version of the parameter file but only retain the SLA and flnr rows
In the "work" folder, make another copy of the original parameter file, and give it a new name like param_list_example_kougarok_flnr_sla In that file edit so that the contents look like the example below and save the file
flnr 11 0.07 0.145
flnr 12 0.07 0.145
slatop 11 0.009 0.045
slatop 12 0.009 0.045
After you create the new parameter file, you can start a new ensemble simulation the same way you setup the previous runs. In this case, you create another new case_prefix to store the new results, e.g. --case_prefix=OLMT_ens_sla_flnr. An example of the full run command is
docker run -t -i --hostname=docker --user modeluser -v elmdata:/inputdata -v elmoutput:/output \
-v elmwork:/example_inputs fasstsimulation/elm-builds:elm_v2-for-ngee-serial_multiarch /scripts/OLMT_ens_docker_example.sh \
--site_name=kougarok --case_prefix=OLMT_ens_sla_flnr --ad_spinup_years=30 --final_spinup_years=60 \
--transient_years=20 --num_ens=4 --num_groups=2 --param_list=/example_inputs/param_list_example_kougarok_flnr_sla
Once the new simulation is complete, you can then review the output using the same plot_ELM_ensembles.ipynb notebook but opening the new ensemble location (set earlier when creating the runs), e.g. OLMT_ens_sla_flnr_AK-K64G_ICB20TRCNPRDCTCBC
For example, the variation in Net Ecosystem Exchange (NEE) given the parameter options for flnr and SLA. In this case, flnr alters the value of Vcmax which is a key physiological parameter controlling carbon fluxes
Similar to above, but for LAI (which is partially controlled by variation in SLA)
Congratulations, at this point you should be a pro at setting up ensembles! Try experimenting with different parameters, parameter options, or spinup or transient run lengths to see how the results change. And don't forget that if you change the notebooks and want to save a copy for later you need to download a copy to your host computer!
There are several options you can tweak for running ensemble simulations. These are:
--case_prefix which allows you to change the output path, e.g. --case_prefix=OLMT_ens_kougarok_v1
--site_name which allows you to change which site to run the model, e.g. --site_name=council
--ad_spinup_years which allows you to change the accelerated spinup length, e.g. --ad_spinup_years=200
--final_spinup_years which allows you to change the length of spinup, e.g. --final_spinup_years=600
--transient_years which allows you to change the length of the transient simulation period, e.g. --transient_years=50 or you can just remove this so that the simulations cover the full transient period, 1850-2014
--num_ens which allows you to change the number of ensemble members, e.g. --num_ens=50
--num_groups which controls the number of MPI groups, e.g. --num_groups=5
--param_list which controls which parameter file to use to control the parameter perturbation experiment, e.g. --param_list=/example_inputs/param_list_example_kougarok_sla_and_flnr