Parallel efficiency

[Weiming-Hu]

This is the time profiling from 4 nodes, 72 processes on each node:

****************** Simple Clock Summary ******************
Total wall time: 5807.40820813179s
Reading file: 2727.614988088608s
Sun position calculation: 658.7170560359955s
Scenario 00000 simulation: 1468.309131860733s
Scenario 00000 simulation for analogs: 596.2754669189453s
Scenario 00000 simulation: 140.38229155540466s
Scenario 00000 simulation for forecasts: 45.32916855812073s
Scenario 00000 simulation: 133.69862699508667s
Scenario 00000 simulation for analysis: 37.081478118896484s
*************** End of Simple Clock Summary **************

This is the time profiling from 2 nodes, 72 processes on each node:

****************** Simple Clock Summary ******************
Total wall time: 7323.613418340683s
Reading file: 1980.4284970760345s
Sun position calculation: 1295.2253940105438s
Scenario 00000 simulation with analogs: 3146.107565164566s
Scenario 00000 writing simulation for analogs: 274.17069029808044s
Scenario 00000 simulation with forecasts: 273.43497824668884s
Scenario 00000 writing simulation for forecasts: 51.14760994911194s
Scenario 00000 simulation with analysis: 278.81663608551025s
Scenario 00000 writing simulation for analysis: 24.282047510147095s
*************** End of Simple Clock Summary **************

The overall wall time is improved when more processes are created. And the computation parts (e.g. various stages for simulation) show good parallelization efficiency. But the file I/O overhead is significant.

The significant file I/O could be related to parallel access to NetCDF files. All processes are accessing the same file which could slow down the process.

A better model for parallel file I/O would be a hierarchical structure where only a handful of processes carry out the file I/O.