Validation thresholds in compareScenarios

[pweigmann]

Purpose of this PR

Validation thresholds can now be included into line plots in compareScenarios via the following changes:

• this PR
  • add validationConfig as optional argument of a compareScenarios profile which is handed to piamPlotComparison::compareScenarios.
• piamPlotComparison PR: Validation thresholds in compareScenarios pik-piam/piamPlotComparison#26
  • checks if a validationConfig is selected for a cs call, tell remind2 to run validation. Information about the validationConfig is shown under the file reference table in the beginning of the document.
• remind2 PR: Validation thresholds in compareScenarios pik-piam/remind2#836
  • executes the validation in preprocessing.R, no changes to plot calls
• piamValidation PR: Validation thresholds in compareScenarios pik-piam/piamValidation#76
  • supplies the thresholds via the new function getThresholdBands() which is the outsourced version of how the colored background areas were calculated in linePlotThresholds() before.
• mip PR: Validation thresholds in compareScenarios pik-piam/mip#154
  • adjustments to showLinePlots() and the new function addThresholdBands() plots the colored areas below line plots if variable, unit region and period match.

It boils down to using the default REMIND thresholds in the AMT validationConfig (via the new validationConfig parameter) on the runs that were selected for a compareScenarios execution, together with their historical data for checks that are relative to reference data. It has been implemented in a way which avoids needing to make changes to the plotting calls in the cs RMDs by checking if a given variable has thresholds available that can be plotted.

For now, the REMIND scripts use NULL as default, so it won't yet run the validation, until further testing has been done.

Example:

Type of change

Indicate the items relevant for your PR by replacing ◻️ with ☑️.
Do not delete any lines. This makes it easier to understand which areas are affected by your changes and which are not.

Parts concerned

• ◻️ GAMS Code
• ☑️ R-scripts
• ◻️ Documentation (GAMS incode documentation, comments, tutorials)
• ◻️ Input data / CES parameters
• ◻️ Tests, CI/CD (continuous integration/deployment)
• ◻️ Configuration (switches in main.gms, default.cfg, and scenario_config*.csv files)
• ◻️ Other (please give a description)

Impact

• ◻️ Bug fix
• ◻️ Refactoring
• ☑️ New feature
• ◻️ Change of parameter values or input data (including CES parameters)
• ◻️ Minor change (default scenarios show only small differences)
• ◻️ Fundamental change of results of default scenarios

Checklist

Do not delete any line. Leave unfinished elements unchecked so others know how far along you are.
In the end all checkboxes must be ticked before you can merge.

• I executed the automated model tests (make test) after my final commit and all tests pass (FAIL 0)
• I adjusted the reporting in remind2 if and where it was needed
• I adjusted the madrat packages (mrremind and other packages involved) for input data generation if and where it was needed
• My code follows the coding etiquette
• I explained my changes within the PR, particularly in hard-to-understand areas
• I checked that the in-code documentation is up-to-date
• I adjusted forbiddenColumnNames in readCheckScenarioConfig.R in case the PR leads to deprecated switches
• I updated the CHANGELOG.md correctly (added, changed, fixed, removed, input data/calibration)

Further information (optional)

A test cs is available here: https://cloud.pik-potsdam.de/index.php/f/30284389

Note, that most variable do not have validation thresholds. Find a list of which variables are checked how here:

https://pik-piam.github.io/piamValidation/articles/configs/validationConfig_REMIND.html

[simonkrogmann]

How much does this increase CS2 time (both with and without validation thresholds)?

[pweigmann]

How much does this increase CS2 time (both with and without validation thresholds)?

Would expect it to be 30s to 5min max, depending on config and number of runs being compared. No change expected if config is NULL as no validation is run.

[simonkrogmann]

And another thing: maybe test if this breaks the AMTs or even integrate it validationConfig if you find it useful there. Last time I touched this code I broke them. (coordinate with @tobiasdiez on his refactoring of modelstats)

[simonkrogmann]

put that directly in the definition of the args aboce

[pweigmann]

that would break backwards compatibility, I'd rather keep it here

[simonkrogmann]

I think adding it to the definition of args directly above does not change the behavior

[pweigmann]

yes it does, if the argument is NULL, it would still create args$validationConfig if created in the list, while args$validationConfig <- NULL will not create it. Older versions will not expect that arg and throw an error in the first case.

[simonkrogmann]

Ah, I learned another weird quirk of R today