Skip to content

Allow grouping of input files into output files#4

Open
joshuatorrance wants to merge 4 commits into
mainfrom
3-yearly-output
Open

Allow grouping of input files into output files#4
joshuatorrance wants to merge 4 commits into
mainfrom
3-yearly-output

Conversation

@joshuatorrance

@joshuatorrance joshuatorrance commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

This PR adds a new feature to allow for the grouping of input files into output files. E.g. 12 monthly files into 1 yearly file per variable.

  • Code
  • Tests
  • Docs

Closes #3

@joshuatorrance joshuatorrance self-assigned this Jul 21, 2026
@joshuatorrance
joshuatorrance marked this pull request as ready for review July 21, 2026 05:58
@joshuatorrance
joshuatorrance requested a review from blimlim July 21, 2026 05:58
@blimlim

blimlim commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Thanks @joshuatorrance for putting this together. The implementation with the wild group is really neat. I haven't looked through the code changes/tests in detail yet, but have started by running the new code on a year of ESM1.6 output in /g/data/tm70/sw6175/development/esm1p6/test_splitnc/hist-r1-output000. This has brought up a couple small catches related to the data:

Encodings
Combining daily outputs, and combining subdaily outputs fails for both the atmosphere and sea ice. Because of the different month lengths, the original_shape, chunksizes and preferred_chunks values in the encodings aren't always the same across the files. E.g comparing encodings in two of the 3hrly output files from the atmosphere:

fld_s03i237 original_shape
file1: (248, 145, 192) file2: (224, 145, 192)
--------
fld_s05i215 chunksizes
file1: (124, 73, 96) file2: (112, 73, 96)
--------
fld_s05i215 preferred_chunks
file1: {'time_0': 124, 'lat': 73, 'lon': 96} file2: {'time_0': 112, 'lat': 73, 'lon': 96}
--------

I think we could drop the original_shape value without issue. What would the best way be to deal with the chunksizes and preferred_chunks values?

Inconsistent surface altitude
The 3D atmosphere variables come with a 2D surface_altitude coordinate attached. Due to how it's been calculated in the monthly mean files, it has rounding level differences in different files:
download-65

This causes an issue with xr.open_mfdataset:

 xarray.structure.merge.MergeError: conflicting values for variable 'surface_altitude' on objects to be combined. You can skip this check by specifying compat='override'.

Not sure what the best way around this is, perhaps we could catch this error and use compat='override' when it occurs, or have a preprocessing step that rounds surface_altitude when it is present?

Dropping surface_altitude from the datasets, no other issues come up when merging.

@joshuatorrance

Copy link
Copy Markdown
Collaborator Author

Inconsistent surface altitude

This change should handle the inconsistent surface_altitude fields.
74ec035

It adds an extra preprocessing step to just round off 'surface_altitude' to 4 decimal places. I've piggybacked this behaviour off of the --use-esm1p6-filenames to keep things simple.

@joshuatorrance

Copy link
Copy Markdown
Collaborator Author

Encodings

original_shape, chunksizes and preferred_chunks are now also being excluded from the saved encodings:
0547716

None of these are important for splitnc. If we end up with more encoding causing problems we can consider only including the encodings we need, probably just coordinates but I'd have to confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generating yearly output with splitnc

2 participants