update capacity input database to AEO-NEMS 2026 by atpham88 · Pull Request #13 · ReEDS-Model/ReEDS_Input_Processing

atpham88 · 2026-08-16T19:51:56Z

Summary

This PR updates ReEDS capacity input database with AEO-NEMS 2026 and EIA860M version June 2026, and also overhauls the NEMS processing script to make it more clear and easier to use as well as remove redundant steps.

Technical details

Implementation notes

Changes made to to this PR include:

Remove a_inheritance.py: This script is no longer necessary. We now directly merge AEO-NEMS to EIA860M using the final output file structure established in Clean up input plant database ReEDS#164.
b_aeo_cleaning.py --> a_data_cleaning.py: Improve the process of cleaning AEO and EIA860M data to resolve issues saw in prior version of capacity database used in ReEDS (such as applying repower year as new start year).
- Units with statuses (OA) Out of service but expected to return to service in next calendar year , (OS) Out of service and NOT expected to return to service in next calendar year and (SB) Standby/Backup: available for service but not normally used are considered operating and available for dispatch.
c_geospatial_mapping.py --> b_geospatial_mapping.py.
d_hydro_classification.py --> c_hydro_classification.py: Remove any mention of state and resource regions (we only use FIPS from now on to capacity input database).
e_additional_inputs.py --> d_additional_inputs.py:
- Remove several manual retirement data adjustment for nuclear power plants (For example Palisades nuclear restarting in 2026 is now captured in new version of EIA860M and no longer need manual adjustment here).
- Plant max ages are now read directly from main ReEDS repo from inputs/plant_characteristics/maxage.csv and only include one value for each plant type regardless of its size. lfill-gas, pumped-hydro, csp-ns, pvb_pv and pvb_battery are missing from maxage.csv in the ReEDs repo and are assigned values of 100, 100, 30, 30, 30 years respectively.

Additional changes

Rename all subfolders to lower case for consistency.
Add intermediate_outputs folder within the outputs folder to store outputs from each step of the process.

Issues resolved

Several units in current capacity input database have wrong start years (and thus wrong retirement years) due to not having their start years updated to their repowered years. This issue is fixed in this PR.
Units with out of service and standby statuses previously are not considered operating and not available for dispatch but now they are. This results in some extra 16GW of capacity available for dispatch compared to last version.

Relevant sources or documentation

Output verification:

Plot national- and FIPS-level changes in online and retire capacity by tech for each year between 2010 and 2050.
- Planned online:

Planned retire:

Existing capacity map:
- Current fleet

- Updated fleet

Rerun ReEDS model with updated capacity input database (in ReEDS main repo): Update capacity input database to AEO-NEMS 2026 ReEDS#190.

Documentation: did you update the README for a given a folder?
Yes
ReEDS Documentation: do any parts of the ReEDS documentation need to be updated because of the changes in this pull request?
No. Updated README for this input processing only.

… available for dispatch

…ison

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

bsergi

A few thoughts, mostly on processing structure that aren't necessarily directly related to the update itself.

For the plots, could you structure so that the old and new capacity plots are side-by-side and use the same y-axis? I think that would facilitate comparsion. You already include the different plot which covers similar information here, but just a thought for going forward.

bsergi · 2026-08-26T15:39:44Z

+* `d_additional_inputs.py`, which includes `d1_set_retire_years.py`, `d2_fix_upgrades.py`, and `d3_merge_psh_dbs.py`: This script handles updated retirement years that are outdated or missing in AEO-NEMS and EIA860M, fix upgrades, and handles other additional adjustments. Any manually adjusted retirement years should be included in `d1_set_retire_years.py`.
+* `e_comparison_plotting.py`: This script generates comparison figures between previous version of NEMS and the newly updated version for validation.

 # Input files and params to run run.sh


Would you add some details to this section on where you get the input files? If we download them from somewhere it would be good to include links.

bsergi · 2026-08-26T15:41:15Z

Would you add a bit more to the overview about the general process? My rough understanding is that we start from AEO file, clean it, and then supplement with the 860M.

I added a bit more details in overview and the steps

bsergi · 2026-08-26T15:43:15Z

 coal_plant_retirement='EIA860_2025ER_CoalRetirements.csv'

-# Average lithium battery duration:
+# Assumed average duration for planned storage units:


what is this based on on? would be good to document where this assumption comes from.

This is now removed and storage_duration is calculated based on battery units that came online within 5 years of current year.

bsergi · 2026-08-26T15:45:27Z

+    return nems_eia860_final
+
+def addHeatrates(nems_eia860):
+    # Add in heat rates for planned units (AEO inputs):


+1 to this comment. It also might be better to store these as an input file and read them in.

bsergi · 2026-08-26T15:50:13Z

+    # Main directory
+    dir = os.getcwd()                                                                   
+
+    # Key parameters:


Right now if the script is launched using run.sh it will ignore the command line arguments and use the testing ones below. Before merging you should uncomment these and recomment the ones below.

Alternatively, you could set up the main function to take arguments and then either call directly using testing argument or pass command-line arguments to it from a if __name__ == '__main__': (this is the approach we've been using for a number of other python modules).

bsergi · 2026-08-26T15:53:54Z

+    # Specify scrubber or unscrubber coal units
+    aeo_data.loc[aeo_data['reeds_tech'].notna(), 'tech'] = aeo_data['reeds_tech']
+    aeo_data.loc[(aeo_data['tech'].str.contains("coal", na=False)) &
+                 (aeo_data['T_SYR']<=1969),'tech'] = 'coalolduns'


It might be overkill but could you put this in variable with a descriptive name? I think that's usually more informative than a number with no context.

I added two variables coaluns and coalcrs with more descriptions for both now.

bsergi · 2026-08-26T16:00:31Z

+    # Then, collapse on T_PID and T_UID
+    aeo_data_mult_g = aeo_data_mult_g.groupby(['T_PID','T_UID','TVIN'],
+                                            as_index=False).agg(
+                                                {'ctt':'first','wst':'first','THRATE':'mean','TC_SUM':'sum',


might suggest storing this as a global dictionary or input file and then reading in here. if you do keep in the code it might be cleaner to read with one assignment per line even though it's more lines. alternatively, there might be a way to use 'first' as the default and then only specify the ones that are different.

{
'ctt':'first',
'wst':'first',
'THRATE':'mean',
'TC_SUM':'sum',
'TC_NP':'sum',
'TC_WIN':'sum',
'battery_duration':'mean',
'T_SYR':'min',
'T_RYR':'first',
'tech':'first',
'EFDcd':'first',
'ECPcd':'first',
'T_PNM':'first',
'T_PCA':'first',
'TRFURB':'first',
'T_VOM':'mean',
'T_FOM':'mean',
'T_SMO':'first',
'T_RMO':'first',
'T_CCSROV':'first',
'T_CCSF':'first',
'T_CCSV':'first',
'T_CCSHR':'first',
'T_CAPAD':'first',
'T_CCSCAPA':'first',
'T_CCSLOC':'first',
'sector':'first',
'TCOUNT':'sum',
'T_LONG':'first',
'T_LAT':'first',
'status':'first',
'nems':'first'
}

I cleaned this up a little bit. first is now used as default and columns that are 'sum' and 'mean' are specified.

bsergi · 2026-08-26T17:53:39Z

ReEDS has switched to using EPSG:5070 instead of ESRI:102008 for the crs, so you might consider updating that here. It shouldn't change the actual lat/lon values, but it might affect how things get placed into counties.

Thanks for catching. I updated everything to 5070 now. Previously I had some weird issue where I got point(inf,inf) with 5070 so had to use the ESRI one (but I stilled used the 5070 for nearest distance calculation). But that did not seem to be an issue with projections and can be fixed by preloading projection data by conda install -c conda-forge proj-data, which is a note I added to README.

… value

atpham88 added 8 commits August 5, 2026 15:59

update to 860M version June 2026

ea0e484

Merge branch 'ap_clean_up_NEMS' into ap_update_NEMS

b154cd5

remove a_inheritance & rename

115df1a

improve aeo & eia data cleaning and merging processes

096d2da

improve geospatial mapping

052b76c

clean up

ca10173

clean up retirement year assignment

65c5d10

update documentation and running instruction

9f7f52f

atpham88 commented Aug 16, 2026

View reviewed changes

Comment thread nems_database_processing/d2_fix_upgrades.py Outdated

atpham88 and others added 13 commits August 16, 2026 14:54

Delete maxage.csv

7094ac1

set subfolder names to lower case

a9747e9

more folder name clean up

6f1e95c

Delete nems_database_processing/inputs/AEO_NEMS directory

4d51f22

Delete nems_database_processing/inputs/Coal_Retirements directory

00c3b12

Delete nems_database_processing/inputs/EIA860M directory

f28ff48

Delete nems_database_processing/inputs/Inheritance directory

f5144a9

Delete nems_database_processing/inputs/ORNL_EHA directory

511e19d

allow operating units that have standby/out of service statuses to be…

6a0a615

… available for dispatch

Merge branch 'main' into ap_update_NEMS

2887277

fix issues with upgrades

c8b448a

clean up

73a333a

Update e_comparison_plotting.py

fc7f63d

atpham88 requested review from bsergi and wesleyjcole August 20, 2026 16:18

atpham88 mentioned this pull request Aug 20, 2026

Update capacity input database to AEO-NEMS 2026 ReEDS-Model/ReEDS#190

Open

24 tasks

atpham88 added 3 commits August 20, 2026 11:35

add message for upgrade error

dd2e810

Update e_comparison_plotting.py

db6ba98

clean up descriptions

4e4f336

wesleyjcole reviewed Aug 25, 2026

View reviewed changes

remove inheritance and read NEMS file from ReEDS main repo for compar…

398ca93

…ison

atpham88 and others added 8 commits August 25, 2026 17:50

remove older data files

b421a01

add checks for empty rows in EIA860M

124e4e2

move main functions to the end

11ce7f7

Apply suggestions from Wesley

19ec2c5

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

fix issues and clean up

a4afc6b

Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

add MAT exempt link source and remove pdf file in inputs folder

d970927

clean up

a2fcead

remove OM change switch in d_additional_inputs.py

959cd9c

bsergi reviewed Aug 26, 2026

View reviewed changes

atpham88 added 11 commits August 27, 2026 10:58

calculate storage duration for planned units instead of using assumed…

3e2ed7d

… value

remove hard coded heat rates and replace with calculated values

8857a9c

reorganize agg_dict for groupby

bbbed2c

use EPSG:5070 projection for geospatial mapping

207c02d

add more description for scrubbed and unscrubbed coal categories

0704bf4

update README to include more process description and input data details

c3fa3ef

add links to inputs in README

4497416

Update README.md

f4f3563

Update README.md

2ef8621

Update a_data_cleaning.py

e08c448

check for T_LONG and T_LAT being 0 to adjust as well

11e331f

Uh oh!

Conversation

atpham88 commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Technical details

Implementation notes

Additional changes

Issues resolved

Relevant sources or documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bsergi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

atpham88 commented Aug 16, 2026 •

edited

Loading