Skip to content

feat(array): georeferenced arrays, SRID zones, and GeoTIFF COPY - #427

Merged
farhan-syah merged 3 commits into
mainfrom
fix/eo-raster-import
Oct 18, 2026
Merged

farhan-syah merged 3 commits into
mainfrom
fix/eo-raster-import

Conversation

@farhan-syah

Copy link
Copy Markdown
Member

Summary

Arrays gain a georeference, polygon zones drawn in a real SRID, and GeoTIFF bulk load through COPY.

What changes

Georeferenced arrays

  • CREATE ARRAY ... GEOREF (SRID n, X dim, Y dim [, TRANSFORM (gt0, .., gt5)]). With TRANSFORM, the dims are INT64 pixel indices, and the pixel centre is index + 0.5. Without it, the dims hold world coordinates.
  • ALTER ARRAY <name> SET GEOREF (...) and DROP GEOREF. The georef is outside schema_hash, so data, segments and the Data Plane open check are unchanged.
  • DESCRIBE <array> shows dims, attrs, tile extents and the georef.
  • ARRAY_ELEMENTWISE refuses two arrays with different georefs.

Zones in an SRID

  • ARRAY_AGG and ARRAY_SLICE take an EWKT zone: 'SRID=4326;POLYGON((...))'. zone_dims NULL means the georef axes.
  • Membership is exact per cell: cell centre → array CRS → zone SRID → point-in-polygon. A densified, padded dim box prunes tiles and narrows the slice.
  • ARRAY_SLICE accepts 2 to 6 arguments (5 is refused). The zone travels to every shard on the distributed read path.
  • ARRAY_CELL_POINT() and ARRAY_CELL_FOOTPRINT() return a cell's geometry in the array SRID.
  • A zone drawn clockwise is accepted. Rings are oriented before validation.

GeoTIFF load

  • COPY <array> FROM '<file>.tif' WITH (FORMAT geotiff [, BANDS (a, SKIP, b)] [, NODATA n] [, TIME '...']).
  • New crate nodedb-raster on tiff 0.11.3, with deflate and LZW only. Every decode allocation is bounded before it happens. The first image directory is checked before tiff reads it.
  • A file refused at open commits 0 rows. A file cut short is 22P04.
  • New fuzz target geotiff_decoder with a seed corpus, added to the fuzz CI matrix.

Behavior changes

  • Server-path COPY (all formats, FROM and TO) requires superuser (42501). It reads regular files only through one handle (58030). Any writer could previously read any file the server user could read.
  • COPY ... FROM STDIN WITH (FORMAT geotiff) is 0A000.
  • ALTER ARRAY ... SET (audit_retain_ms = n) refuses negative values. The retention floor is checked again against the committed entry when the ALTER applies.
  • Cached plans over an array are keyed on the array's catalog version. Before this, a plan cached before an ALTER, DROP or re-CREATE kept running with the old schema.
  • New SQLSTATE 22P04. New SqlError::FeatureNotSupported maps to 0A000.

Lite

The Lite workspace does not build against this branch until these sites change:

  • error/from_sql.rs: add SqlError::FeatureNotSupported to the refused arm.
  • MetaOp::AlterArray.schema_msgpack (physical_visitor/adapter/meta.rs).
  • AlterArrayVisitArgs (adapter/visitor/trait_impl.rs, adapter/visitor/ddl.rs).
  • array_slice takes ArraySliceVisitArgs. ArrayOp::Slice gains zone_msgpack (visitor/array/query.rs, physical_visitor/adapter/array.rs).
  • CreateArrayVisitArgs.georef, and ArraySchema literals gain georef.

Measured

  • Zone membership costs about 2,510 instructions per tested cell (release, UTM 47N pixel grid, WGS 84 zone).
  • GeoTIFF load allocates 2 times per kept pixel and 0 per nodata pixel.
  • Three GeoTIFFs written by tifffile load with row counts and sums equal to tifffile's own read.

Testing

  • Full suite: stage 1 22,681/22,681, stage 2 512/512.
  • Workspace clippy (all features), the nodedb-client feature sets, cargo deny, every static gate, wasm32 check, and a replay of all 7 fuzz targets.
  • 3 minutes of real fuzzing on geotiff_decoder: 512,565 inputs, no crash.
  • End-to-end test: a 256×192, 3-band tiled GeoTIFF on EPSG:32647 loads with COPY 49052. ARRAY_AGG over a WGS 84 polygon matches an independent UTM inverse plus point-in-polygon.

Add nodedb-raster, which reads GeoTIFF rasters with the deflate and
LZW codecs only and refuses JPEG, fax and WebP compression. Decode
capacity is bounded before allocation. A fuzz target and its CI job
cover the decoder.
Add geometry_from_ewkt with an optional SRID prefix, and orient_rings
to normalise exterior and interior ring winding.
Arrays can declare a georeference, which the SQL parser, planner,
catalog and cell-geometry functions carry through to region and zone
slicing, including the distributed read path. COPY into an array loads
GeoTIFF rasters through nodedb-raster.

Server-path COPY now requires superuser, reads regular files only and
refuses a pre-existing temporary output file. The planner records array
catalog versions as unleased descriptors so an ALTER invalidates cached
plans, and array retention is tracked in the catalog.

The surrogate round-trip integration tests are split into a module
directory.
@farhan-syah farhan-syah added the run-ci Opt this PR into the full test suite; re-add to force a re-run label Oct 18, 2026
@farhan-syah
farhan-syah merged commit 9ac9c35 into main Oct 18, 2026
14 checks passed
@farhan-syah
farhan-syah deleted the fix/eo-raster-import branch October 18, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci Opt this PR into the full test suite; re-add to force a re-run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant