Coverage nexp masks by martinkilbinger · Pull Request #797 · CosmoStat/shapepipe

martinkilbinger · 2026-07-09T14:10:59Z

Summary

Implements creation of coverage healsparse masks encoding the exposures on which PSF information is avialable.

The current code, implemented in utilities does the following steps:

Read CCD list and download corresponding FITS exposure images headers (header_downloader.py)
Extract CCD corner coordinates using WCS from header (field_corners_extractor.py)
Create healsparse mask (coverage_map_builder.py)

This can be integrated into a workflow. We also need to implement obtaining the list of valid CCDs with PSF model.

Reviewer Checklist

Reviewers should tick the following boxes before approving and merging the PR.

…asses

…o coverage

# Conflicts: # docs/source/pipeline_canfar.md # pyproject.toml

- Extract _expnum_from_path, _parse_header_to_wcs, _megacam_field_corners as module-level helpers. process_single_header (static, for multiprocessing) and the instance methods now share one implementation. - Remove dead get_wcs_from_header and get_megacam_field (duplicated logic that had already been inlined into process_single_header). - Fix re.search(r'(\d+)\.txt') missing its subject string. The new helper also replaces the hard-coded p[end-6:end] exposure-number extraction in run(), so exposure numbers of any length work. - Declare verbose in params_default so the contract is local instead of relying on cs_util.args.parse_options to inject it. Addresses review feedback on #699.

- Replace hsp_map + 0 with hsp_map.copy() (healsparse supports .copy since at least 1.x; more explicit intent). - Declare verbose in params_default; currently populated only by cs_util.args.parse_options auto-injection. - Fix typo: "Install te cs_util" -> "Install the cs_util". Addresses review feedback on #699.

- coverage_plotter, header_downloader: declare verbose explicitly in params_default. - coverage_plotter: comment acknowledging that FootprintPlotter._regions is a private cs_util attribute; refactor to public accessor when cs_util exposes one. - header_downloader: expand the vos:cfis/pitcairn comment to flag it as UNIONS/CFIS-specific and document override. - build_and_plot_coverage_maps.sh: explain the BUILD_NSIDE=131072 choice (matches UNIONS bit-mask pixel scale, ~0.1"), and note the CoverageMapBuilder default of 2048 for lighter use. Addresses review feedback on #699.

Resolve pyproject.toml conflict: the coverage branch forked before develop's uv/pyproject modernization, so take develop's file as the base and layer in the coverage additions — healsparse + skyproj runtime deps, the plot optional-deps group, plot added to the dev extra, and the six coverage/get-ccds console entry points. All other files auto-merged. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW

The coverage feature adds healsparse (map building) and skyproj (plotting) to pyproject. healsparse was already resolved transitively via cs_util; this pins skyproj 2.5.0 and records both as shapepipe's own direct dependencies so 'uv lock --check' passes. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW

martinkilbinger · 2026-07-14T05:43:58Z

Update on workflow, the 0th step is:

Get CCDs with valid PSF: get_ccds_with_psf using CcdPsfHandler (needs updating for v2.0)

And field_corner_extractor.py is indeed not doing the CCD-wise masking, but only looking at the entire MegaCam field, which is wrong. This needs fixing.

cailmdaley · 2026-07-14T11:07:16Z

want me to take care of these issues @martinkilbinger ?

martinkilbinger · 2026-07-15T09:13:55Z

want me to take care of these issues @martinkilbinger ?

yes please :-)

There was an earlier question, not sure whether this has been answered, but I confirm that here we want single-CCD-based masks, to account for available/missing PSF. Exposure-count-based masks, or coverage masks, are done elsewhere.

The v>=1.4 path built the full set of exposure single-HDU IDs but never removed the CCDs whose PSF model is missing, so every candidate CCD was reported as valid. Subtract the missing set so the result is the CCDs that actually have a valid PSF, and report the three counts.

Replace the whole-MegaCam-field corner extraction with per-CCD extraction: for each HDU's WCS, build the 4 corners from the CCD pixel bounds (read from pixel_shape, i.e. NAXIS1/NAXIS2) using pixel edges so the quadrilateral covers the full CCD area. Emit one row per CCD keyed on the '<expnum>-<ccd_idx>' ID. Add an optional --ccd_list/-l option: when given, only CCDs in that file (the output of get_ccds_with_psf) are written, so the coverage mask is built from single CCDs with a valid PSF. Resume still keys the done-set on exposure number, now parsed from the CCD-ID first column. Drop the dead _megacam_field_corners whole-field path; keep _parse_header_to_wcs and the multiprocessing structure.

…ards Parse the new per-CCD row format (string CCD ID in column 0, then 8 float corner columns). Add an RA-wrap guard that unwraps polygons straddling the RA=0 seam onto a common branch, and a pole guard that skips polygons reaching |dec| >= 89 deg with a warning (CCD footprints never reach there). Since the CCDs of one exposure do not overlap, accumulating value 1 per CCD polygon still counts exposures-with-valid-PSF per pixel.

Copy each header to a '.part' file in the same directory and os.rename on success. An interrupted transfer can no longer leave a partial file that resume treats as complete; the temp file is removed on failure.

Fix the console-script name typo (get_ccd_with_psfs -> get_ccds_with_psf) so it matches the docs and script. Harden the build/plot script with 'set -euo pipefail' and document the per-CCD chain (step 0 get_ccds_with_psf feeding extract_field_corners --ccd_list). Update the canfar docs: restore the coverage helpers as shipped, align filenames (ccds_with_psf_<version>.txt, exp_ra_dec_<version>.txt), and describe the per-CCD workflow including step 0.

…ownload Rewrite the coverage tests for the per-CCD design: handler missing-CCD subtraction; per-CCD corner extraction and pixel-shape error; --ccd_list filtering end to end; builder parsing of the new row format; RA-wrap unwrap (a CCD straddling RA=0 fills a small patch, not the complement); pole-guard skip; and atomic-download rename behaviour with a mocked copy. Drop the tests that pinned the removed whole-field convention.

Two correctness fixes in the per-CCD extractor: - Image dimensions: header_downloader fetches fpack tile-compressed headers where NAXIS1/NAXIS2 describe the compressed binary table (byte width, row count), so WCS.pixel_shape reports e.g. (8, 4612) instead of the true (2080, 4612). Read ZNAXIS1/ZNAXIS2 when ZIMAGE is set, falling back to NAXIS1/NAXIS2. _parse_header_to_wcs now returns (wcs, shape) pairs since the WCS drops the Z-keywords; _ccd_corners takes the shape explicitly. - Resume: key the done-set on individual CCD IDs, not exposure numbers. A write interrupted mid-exposure previously left a partial exposure that a -r rerun skipped forever (silent undercount); an expanded --ccd_list rerun skipped exposures whose new CCDs were never written. get_done_ccds reads the present CCD IDs and run() drops only those, so partial exposures are completed and new CCDs added, never duplicating a row.

- header_downloader.get_exposures: wrap the text-format loadtxt in np.atleast_1d so a single-line CCD list (0-d array) does not crash on iteration. - coverage_run: update the extract/build runner docstrings from whole-field 'field corner coordinates' to per-CCD language. - pipeline_canfar docs: add a runnable plot_coverage_map example (the SGC region command) covering the flags build_and_plot_coverage_maps.sh uses.

Extend the coverage tests for the confirmed findings: - fpack-compressed header fixture; assert ZNAXIS dims are recovered and corners span the true CCD width, not the 8-byte compressed NAXIS1. - resume run() path: a new exposure is appended without duplicating existing rows, and a partial exposure (interrupted mid-write) is completed rather than skipped. - the nexp contract on pixel values: two overlapping CCDs from different exposures give value 2 in the overlap and 1 outside. - single-row builder file (atleast_1d/2d guards); single-line get_exposures; RA-wrap strengthened to a positional-parity check (seam CCD vs the same CCD shifted +10 deg agree in pixel count); get_ccds_with_psf subtraction pinned against the real get_all_shdus.

cailmdaley · 2026-07-16T00:02:46Z

Done — the branch now builds single-CCD-based masks. The chain:

get_ccds_with_psf → valid-PSF CCD IDs. The v ≥ 1.4 path now actually subtracts the missing-CCD set (it previously returned every CCD as valid), and -V v2.0 is accepted (see below).

On "needs updating for v2.0": following your #812 note that the patch concept is removed in v2.0, -V v2.0 is now patch-less — the handler reads summary/missing_job_32_all.txt and exp_numbers.txt relative to the run root instead of iterating P1..P9 (a new -p option restores an explicit P1..P<n> layout if ever needed). The job-32 numbering and summary filenames are unchanged on develop, so those are kept as-is — flag if the v2 runs will change the summary conventions themselves.
extract_field_corners --ccd_list <step-1 output> → one row of corners per valid CCD, from each HDU's own WCS (image dims read from ZNAXIS — the .fz headers' NAXIS are the compressed-table dims). This is where the PSF-validity filter lives; without -l all CCDs are extracted.
build_coverage_maps → one polygon per CCD, with RA-wrap and pole guards. CCDs of one exposure don't overlap, so pixel values still count exposures with valid PSF.

Robustness along the way: extractor resume is keyed on individual CCD IDs (interrupted runs and expanded CCD lists complete cleanly), and header downloads are atomic, so partial files can't poison a resumed run.

57 tests cover the chain — including the nexp value contract (overlapping CCDs from two exposures → 2 in the overlap, 1 outside) — and pipeline_canfar.md documents the full flow with aligned filenames. CI green. Over to you for review.

Update (2026-07-20): get_ccds_with_psf -V now defaults to the patch-less v2.0 (1ffcd4b1), matching the convention adopted in #812's collate_star_cat.py; v1.x layouts are selected explicitly with -V v1.6. Tests still green (45 passed, in-container).

— Claude (Fable) on behalf of Cail.

…v1.6) Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

… map The v2 footprint is still growing (>5000 deg2 per the 2026-07 Paris meeting, new Dec<30 region); the newest patch definitions (auxdir/CFIS/tiles_202510) include P10. Default v2.0 to 10 patches and add an explicit -p/--n_patch override so the handler tracks evolving patch layouts without code changes. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

Martin (PR #812 review): the patch concept is removed in v2.0. For -V v2.0 the handler reads summary/missing_job_32_all.txt and exp_numbers.txt relative to the run root instead of iterating P1..P9; -p restores an explicit P1..P<n> layout when needed. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

…-less) Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

cailmdaley · 2026-07-16T01:16:51Z

@martinkilbinger please take a look, should be good now!

cailmdaley · 2026-07-16T08:53:34Z

i think this closes #836?

Align with the collate_star_cat convention (PR #812): v2.0 (patch-less) is now the default catalogue version. Older, patched layouts are selected explicitly with -V. Help text and the pipeline_canfar.md invocations updated; the v2.0 call drops the flag, v1.x shows -V v1.6. Tests set the version explicitly, so none pinned the old default. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WqNFhq9uGpDpprYeuGeb8k

Lifted from Martin's PR CosmoStat#704 (martinkilbinger:contributors), split off from the unrelated coverage-mask work that now lives in CosmoStat#797. Co-authored-by: Martin Kilbinger <[email protected]> Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DJEQwNRApVAQnt53fkyXLZ

martinkilbinger and others added 25 commits January 9, 2026 14:18

removed tmp files

668e05d

Merge branch 'p9' of github.com:martinkilbinger/shapepipe-1 into p9

de045f5

Moved get_ccds script to pipeline runner and library script

38c5819

class to download eposure headers (for coverage)

d6144ad

coverage mask: scripts and plotter; using cs_util

bc43799

coverage code: added Mike Hudson as co-author

487166c

documentation (in pipeline_canfar.md) on coverage mask scripts and cl…

d0d5488

…asses

Merge remote-tracking branch 'origin/coverage' into coverage

bc8b006

CCD PSF Handler: Added version for v1.3

f4d00d0

fixed coverage plots

271ff40

header downloads (coverage): account for existing downloads

7f99f5a

Merge branch 'coverage' of github.com:martinkilbinger/shapepipe-1 int…

77fe037

…o coverage

Merge remote-tracking branch 'upstream/develop' into coverage

be10deb

Merge branch 'develop' into coverage

728f413

canfar-changes

e10111b

Merge remote-tracking branch 'origin/coverage' into coverage

ac94c16

Merge remote-tracking branch 'origin/develop' into coverage

6d10285

# Conflicts: # docs/source/pipeline_canfar.md # pyproject.toml

PR: remove non-functioning coverage pipeline function

fd61120

PR: coverage_map_builder; moved import into try block

fc37cbf

PR: field_corners_extractor.py; fixed exp number extraction

afcbae6

PR: added comment

96cc4f5

PR: added comment

5215788

martinkilbinger requested a review from cailmdaley July 9, 2026 14:10

martinkilbinger self-assigned this Jul 9, 2026

martinkilbinger added the enhancement New feature or request label Jul 9, 2026

cailmdaley and others added 2 commits July 11, 2026 14:05

This was referenced Jul 12, 2026

Update contributors list #809

Merged

Contributors #704

Closed

cailmdaley mentioned this pull request Jul 15, 2026

Build ShapePipe per-CCD failure mask #836

Open

cailmdaley added 10 commits July 16, 2026 01:30

Merge remote-tracking branch 'origin/develop' into coverage

d2bdf28

fix(coverage): download headers atomically via temp-file rename

96cd62b

Copy each header to a '.part' file in the same directory and os.rename on success. An interrupted transfer can no longer leave a partial file that resume treats as complete; the temp file is removed on failure.

cailmdaley and others added 4 commits July 16, 2026 02:21

feat(coverage): accept catalogue version v2.0 (9-patch footprint, as …

15401d9

…v1.6) Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

docs: state the patch convention per catalogue version (v2.0 is patch…

46214c7

…-less) Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr

This was referenced Jul 16, 2026

Exposure-level star/defect masking strategy #839

Open

HSM: measure PSF/star shapes directly in sky coords (use_sky_coords) #812

Open

cailmdaley mentioned this pull request Jul 21, 2026

Coverage mask #699

Open

12 tasks

cailmdaley linked an issue Jul 21, 2026 that may be closed by this pull request

[NEW FEATURE] Create coverage masks for UNIONS #700

Open

cailmdaley mentioned this pull request Jul 21, 2026

[NEW FEATURE] Create coverage masks for UNIONS #700

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Coverage nexp masks#797

Coverage nexp masks#797
martinkilbinger wants to merge 44 commits into
developfrom
coverage

martinkilbinger commented Jul 9, 2026 •

edited by cailmdaley

Loading

Uh oh!

martinkilbinger commented Jul 14, 2026

Uh oh!

cailmdaley commented Jul 14, 2026

Uh oh!

martinkilbinger commented Jul 15, 2026

Uh oh!

cailmdaley commented Jul 16, 2026 •

edited

Loading

Uh oh!

cailmdaley commented Jul 16, 2026

Uh oh!

cailmdaley commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

martinkilbinger commented Jul 9, 2026 • edited by cailmdaley Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reviewer Checklist

Uh oh!

martinkilbinger commented Jul 14, 2026

Uh oh!

cailmdaley commented Jul 14, 2026

Uh oh!

martinkilbinger commented Jul 15, 2026

Uh oh!

cailmdaley commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cailmdaley commented Jul 16, 2026

Uh oh!

cailmdaley commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martinkilbinger commented Jul 9, 2026 •

edited by cailmdaley

Loading

cailmdaley commented Jul 16, 2026 •

edited

Loading