CanopyStructure Issues when running with third canopy layer

[ckoven]

Hi All,
In running ELM-FATES with three potential canopy layers and the "strict PPA" (i.e. fates_comp_excln=-1) canopy structure algorithm in global runs, I ran into some issues that I want to document here. The initial problem showed up as an error in the two-stream code, but the error was because the scattering elements were not being constructed correctly, and in discussion with @rgknox we realized it was because the canopy was somehow constructed with a third canopy layer even though the second canopy layer was empty. I'm not sure how this could happen unless somehow all the cohorts in the second layer on some patch all died suddenly. But what we realized is that, should such a situation arise somehow, there wouldn't be a way to get to a canopy structure that made sense.

So I tried two things to alleviate the situation. The first and simplest was to change the order of the promotion loop to work upwards from the base of the canopy rather than downwards from the top. So in principle, this means that a cohort could be promoted more than once in a timestep. That alleviated the problem and allowed me to continue my run. So probably we should make a PR with at least this change.

But the deeper problem is that there isn't actually a check that, for all canopy layers above the lowest, that the canopy is exactly full. There is a check that the canopy is not overfull here. But it doesn't check that a given canopy layer is underfull if it isn't the bottom most canopy layer. So I tried making such a check here. The problem is that when I ran it, the model crashed because somewhere there was a patch that couldn't meet the criteria. I tried upping the number of allowable iterations from 10 to 15, and reduced the tolerances, but neither worked. And I am not sure how tight the tolerances need to be for this. So, for now at least, I am putting that change aside.

It seems like a problem that there isn't any check on canopy layers above the base being full, so we should add it to the list of problems to solve that doing this doesn't seem to work right now.