MONAN-analysis

Project to centralize code for analyses of simulations with the MONAN model.

Structure

MONAN-analysis/
├── README.md             
├── pyproject.toml                # makes packages inside src/ installable
├── requirements.txt              # lists libraries needed to run repo
├── .gitignore                    # lists files to be ignored by git
├── src/                          # source code (installable packages)
│   └── monan_analysis/           # namespace for monan_analysis package
│       ├── __init__.py           # python looks for this file when importing modules
│       ├── config.py             # general settings shared across multiple analyses
│       ├── utils.py              # reusable general-purpose routines
│       ├── io.py                 # reusable routines for reading input and saving output
│       ├── plots.py              # reusable routines for plotting
│       ├── preprocess.py         # reusable routines for data preprocessing
│       └── stats.py              # reusable routines for calculating statistics
├── analyses/                     # ready-to-use analysis code
│   ├── analysis1/
│   │   ├── analysis1_main.py     # exemplary main script for analysis1
│   │   ├── analysis1_aux.py      # exemplary auxiliary script for analysis1
│   │   └── analysis1_config.py   # exemplary config with settings specific to analysis1
│   └── analysis2/
│       ├── analysis2_main.py.py  # exemplary main script for analysis2
│       ├── analysis2_aux.py      # exemplary auxiliary script for analysis2
│       └── analysis2_config.py   # exemplary config with settings specific to analysis2
├── exploratory/                  # exploratory (preliminary) scripts
└── unittests/                    # unit tests to ensure code is behaving as expected
    └── test_monan_analysis/      # unit tests for monan_analysis package

---

Installation guide

1. Clone the repository

git clone https://github.com/GAMA-INPE/MONAN-analysis.git
cd MONAN-analysis

2. Set up conda environment

2.1) If using an HPC system, load Anaconda (or other conda) module

module load anaconda

2.2) Create conda environment to install project packages

conda create -n gama

2.3) Activate conda environment

conda activate gama

2.4) Install python inside that conda environment

conda install python=3.12

2.5) Make sure python is installed in the conda env

which python

If the above points to your recently created conda environment, proceed to step 3.

3. Install project-specific dependencies

All project-specific dependencies (external and internal packages under src/) are listed in requirements.txt. Depending on whether you have the external dependencies already installed, you can follow two options:

a) If you do not yet have the external dependencies, you can install those dependencies plus the internal project packages by running

python -m pip install -r requirements.txt

b) If you already have the needed external dependencies and want to install only the internal project packages under src/, you can do this by running

python -m pip install -e .

inside your conda environment.

After installation by option a) or b), you should be able to import in particular internal modules from the packages under src/ in your analysis scripts. You can test this by running from the project's root

python exploratory/import_test/example_analysis.py

If the internal packages have been correctly installed, this test should return

this is a function imported from the utils.py module.
this is a function imported from the io.py module.
this is a function imported from the plots.py module.
this is a function imported from the stats.py module.

---

Quick user & developer guide

The core idea behind this project is that analysis scripts, although developed for performing a particular type of investigation, involve also many repetitive tasks that can be organized in general routines to be reused in the future.

Workflow for analysis pipelines

To account for this, the suggested workflow for performing analyses in this repository is the following:

1. Before starting your analysis, check out the general functions and settings already available in our packages under src/. They may be useful for your particular purposes.

2. In exploratory/, develop your initial analysis scripts and generate your results without worrying too much about organization.

3. As the scripts developed in 2. get more mature, transfer them to analyses/. Here, the goal is to have well tested and documented analysis pipelines that can be readily understood and employed by other users. For a clean code, try to organize the main analysis script as a wrapper (https://en.wikipedia.org/wiki/Wrapper_function; it could be e.g. analyses/analysis1/amalysis1_main.py) that calls subroutines for each task performed. Subroutines that are specific to that particular analysis can be locally defined under analyses/ (e.g. analyses/analysis1/analysis1_aux.py). Subroutines of a more general nature should be defined under src/monan_analysis in one of the available modules depending on its purpose (utils.py, io.py, plots.py, stats.py, etc.). A similar procedure can be employed for analysis settings: in defining the parameters/paths/any user configuration for the analysis, try using separate config files (settings specific to the analysis under e.g. analyses/analysis1/amalysis1_config.py, and general settings under src/monan_analysis/config.py). Such files have the advantage that they can be saved together with the analysis results, so that one has easily at hand all user input employed for obtaining them.

The goal of all this is to integrate the created general functions and settings into our src/ packages, which are installable, and will therefore be readily available for any future analysis script.

As explained in 1), during steps 2) and 3) the user can take advantage of the already defined functions and settings in our packages under src/. To call them, one just needs to import the respective package as usually done in python, e.g., to import a plotting function do

from monan_analysis.plots import example_function_plots

Technical workflow for developing the project

For developing the project, the following technical workflow is suggested:

1. Preferably, new ideas for implementations should be added to issues (https://github.com/GAMA-INPE/MONAN-analysis/issues), where they can be discussed with the team.

2. Once you plan to implement one of those ideas, assign the issue to yourself and comment something like "I'm taking up this issue" in the respective issue page to avoid different people working on the same issue.

3. Open new branch (name abbreviation/feature, e.g. gtm/repo-organization) for working on that implementation.

4. Work on the code in that branch.

5. Once the code is ready for merging, open a pull request with a basic description of the implementation (mentioning the issue it addresses, if any), and assign reviewers.

6. After reviewer's approval, and if the unit tests are passing, merge into main branch.