Repository navigation
negpy_core: CPU kernels in Rust, no Numba, no cv2 in the app - #1489
Merged
Merged
Conversation
A pure-Rust PyO3 extension (abi3-py313) built by maturin. The core functions build without Python, so cargo test needs no interpreter; the python feature adds the PyO3 wrappers. The setuptools package search narrows to negpy and negpy.*, so it skips the crate folder.
percentiles() calls negpy_core.percentiles: a 16-bit radix histogram, a gather of the buckets that hold the ranks, then a select in each. Results are bit for bit np.percentile (test_percentiles adds a 1M signed array, heavy repeats and a strided view). On 23 sample scans, autocrop detection and measure_base_log give identical results. Best of N, 7800X3D, float32: 19 MP (1, 99): 152 ms -> 7.9 ms (16 threads), 23.6 ms (1 thread) 1.7 MP preview: 16.9 ms -> 0.70 ms, 2.1 ms (1 thread) 20k strip (3 qs): 0.20 ms -> 0.08 ms 60k strip (3 qs): 0.65 ms -> 0.10 ms
make rust runs cargo fmt --check, clippy -D warnings and cargo test; make all and make format include it. The Makefile unexports ARCHFLAGS, because maturin reads it on every OS. CI adds the Rust checks, cargo-deny (licenses, a ban on crates that compile C) and a four-OS native job that runs the Rust tests and the ported-function oracles. CI also runs on mz/core. release.yml installs the pinned toolchain and takes a publish input, so a test build makes no release. flake.nix builds negpy-core from the vendored Cargo.lock (not tested: no nix here).
- More than 255 histogram buckets wrapped the u8 slot index onto the empty-slot marker and panicked into Python; slots are u32. - The wrapper rejects a q outside [0, 100], as np.percentile does. - make rust runs cargo from negpy_core/, so rustup uses the pinned toolchain; the CI native job does the same and installs the Qt libraries the pytest session fixture needs on Linux. - Docs state the signed-zero exception and the pkg-config ban.
The test job installs mesa-vulkan-drivers and pins wgpu to Vulkan, so the GPU tests and WGSL parity tests run in CI instead of skipping.
scripts/core_manifest.py scans negpy/ for functions that call cv2, a numpy order statistic or are Numba kernels, and writes negpy_core/manifest.toml: 232 entries with a seeded phase, the calls they make and the manifest functions they depend on. Phase, status, owner and note are kept by hand across rescans. test_core_manifest fails when the file is stale, so a merge from main that adds a hot function shows up.
scripts/replay.py records real calls of the manifest functions while frames run through the preview load and the CPU pipeline (inputs, result, one-thread time called alone), then replays them through the current code bit for bit, or times them under a machine-wide lock. Three sample frames give 35 phase-2 functions and all replay the same. The manifest scan also takes functions built on numpy reductions, so the measure_base_log chain is in it (273 entries).
negpy_core.resize_area copies OpenCV's area resize for float32: the general path (computeResizeAreaTab weights, the horizontal pass per source row, then the vertical sum) and the integer-scale path, where a 2x2 one- or four-channel image takes the 4-lane SIMD order over the columns the vector loop reaches and the scalar order elsewhere. test_core_resize compares it with cv2 bit for bit (31 cases). _normalize_detection_input uses it: the phase 2 replay over 33 real frames gives 40/40 functions identical. replay.py record takes --edits, so each frame runs with its saved edit from a copy of edits.db. 5668x3945x3 -> 1600x1114: cv2 32.3 ms / Rust 27.6 ms on one thread; cv2 5.3 ms / Rust 10.7 ms on 16.
This reverts commit 9d5f082.
negpy_core.gaussian_blur, sep_filter2d and gaussian_kernel copy OpenCV's sepFilter2D engine for float32 as opencv-python's AVX2 build runs it: the 8-lane vector region with fused multiply-adds, then the scalar loops in the arithmetic GCC compiled them to (contracted or not, per loop, found by experiment and kept as rules). The Gaussian kernel follows getGaussianKernelBitExact, with OpenCV's softdouble exp. Borders: reflect-101, replicate, constant, reflect. Symmetric kernels and Sobel's 3-tap antisymmetric one; others raise. test_core_filter compares with cv2 bit for bit (366 cases). The cv2 oracles now run on Linux x86-64 only: opencv-python on macOS arm64 differs from it in the last bits (CI showed resize_area failing there while Windows and Intel macOS matched). test_core_same_bits holds output hashes every release OS must match. The passes run in strips of rows with AVX2+FMA code picked at runtime. 1600x1059, interleaved medians: one thread 1.3-1.5x slower than cv2; 16 threads 2-3.4x faster (5x5x3 2.78 -> 1.32 ms, sigma 2 2.25 -> 0.65).
Every float32 cv2.GaussianBlur and 3x3 cv2.Sobel in geometry, skew, batch autocrop, normalization and exposure analysis now calls negpy_core.gaussian_blur or the new negpy_core.sobel (sepFilter2D with getDerivKernels' kernels, 10/10 identical to cv2.Sobel). The uint8 calls stay on cv2 until the fixed-point path is ported. The phase 2 replay over 33 real frames gives 40/40 functions identical. The manifest scan keeps functions that call negpy_core, so a function half moved to Rust stays tracked.
negpy_core.morphology_ex mirrors cv2.morphologyEx (erode, dilate, open, close, gradient, top hat, black hat) for uint8 and float32 with a kernel of all ones: OpenCV's rectangle-for-iterations rewrite, its default border value and the other borders. A min or max has no rounding, so a separable pass of overlapping power-of-two windows gives cv2's bits (2772/2772 in the oracle). Other kernels raise. The phase 2 masks in geometry and skew use it; the phase 2 replay over 33 real frames gives 40/40 functions identical. 1600x1059 uint8, one thread: close 31x31 x2 2.02 -> 2.07 ms, black hat 31 0.93 -> 1.05 ms, open 9x9 0.31 -> 0.77 ms; float32 erode 15 1.25 -> 5.45 ms (not on a phase 2 path).
negpy_core.bgr_to_gray (15-bit weights, checked over all 2^24 colours), gaussian_blur_u8 (OpenCV's fixed-point kernel with error diffusion, integer rows and columns, one rounding), threshold_otsu (getThreshVal_Otsu's double arithmetic; IPP agrees with it) and canny (16-bit Sobel, L1 magnitude, TG22 suppression, hysteresis). All integer arithmetic, so all bit-exact: test_core_gray, 84 cases. cv2's parallel uint8 5x5 GaussianBlur is not deterministic: repeated calls on one image can return different bytes, so autocrop detection could differ between runs of the same frame. negpy_core matches cv2 on one thread, and the oracle pins cv2 to one thread. The phase 2 detection masks use these; replay over 33 frames gives 40/40 functions identical.
negpy_core.find_external_contours ports OpenCV 4.13's Suzuki-Abe scanner from contours_new.cpp for RETR_EXTERNAL + CHAIN_APPROX_SIMPLE: its pixel marks, the left-neighbour test that skips borders inside a traced contour, direction-change points, and the output order (each new contour prepended to the root, so the last found comes first). Contours and hierarchy match cv2 on 150 random masks and the 24-case test_core_contours. The two phase 2 contour searches use it; replay over 33 frames gives 39/39 functions identical. _ensure_color left the manifest: it is np.repeat now, with no cv2 call to port.
The first whole functions of the measure_base_log chain move to Rust: get_luminance (Numba's fastmath kernel as f64 fused multiply-adds, the Numba kernel deleted), _normalize_to_uint8 and _detection_luma (NumPy's float32 arithmetic, NaN rules and truncating cast). The functions that need them call negpy_core through unchanged Python signatures. Loops with mul_add compile twice through the multiversion crate, so a CPU with AVX2+FMA runs one instruction and the release baseline stays portable; mul_add is exactly fused on both, so the bits do not change. The manifest now also holds the pure-Python links of a chain (callers of in-scope functions under features/ and kernel/image/): 422 entries. The phase 2 corpus was re-recorded for them: 53 functions, 30 GB. Replay: 53/53 phase 2 functions identical. make bench (one thread, against the recorded Python): _detection_luma 0.56x, _normalize_to_uint8 0.79x.
negpy_core.flat_block_means computes what _base_blocks needs from the 8x8 blocks (means, max, std, band coverage) in NumPy's order: each block sums its 8 rows outer, from +0, divided by float32 64; the channel means and the flat test in float32; np.max's NaN rule. The float32 log10 and the argsort grouping stay in NumPy: on an AVX-512 CPU NumPy's log10 is SVML's, and argsort's tie order is its SIMD sort's, so neither can be copied bit for bit. The replay workload keeps stage caching on, as the preview does, so measure_base_log is recorded now. Replay: 58/58 phase 2 functions identical, _base_blocks 14/14, measure_base_log 25/25. One thread against the recorded Python (which already used the Rust primitives): _base_blocks 0.165x, measure_base_log 0.782x.
After Marcin's decision that transcendental functions come from the pure-Rust libm crate, the log10 and the grouping of _base_blocks move to Rust too; the Python function is one call. Blocks of equal brightness sort by block index, where NumPy's SIMD argsort has its own tie order. negpy_core/LEDGER.md lists the ports that differ from Python (one version bump at release covers them), and the manifest's compare field tells replay how to check them: rows in any order for _base_blocks. Replay: 58/58 phase 2 functions; _base_blocks 8 identical and 6 the same rows in another order; measure_base_log 25/25 identical.
negpy_core.find_rebate_level reads the four border strips of the film box as the Python did: the bed cut joins the float32 comparison as float32 (NEP 50), Python's round() for the ring width, NegPy's percentile helper on the strips, and np.percentile's NaN rule for the bed and the box median (np_percentiles). The first of equal levels wins, in top, bottom, left, right order. Replay: _find_rebate_level 60/60, phase 2 58/58. One thread against the recorded Python: 0.566x.
negpy_core.walk_border_depth runs the border walk per line as the NumPy version did: np.median's middle-pair float32 mean and NaN rule, the float32 margins (NEP 50), the bridge and the early stop, the masked span peak and its first near-peak index, and int64 out. The constants and their reasoning move with it; _EDGE_REBATE_STEP stays in Python too while other functions use it. Replay: _walk_border_depth 60/60, phase 2 58/58. One thread against the recorded Python: 0.414x.
The edge walk's whole subtree moves to Rust: the per-side orientation and core-column crop, the near and deep walks, _pool_border_depth (np.percentile's float64 lerp on int depths), _border_edge_scatter (Theil-Sen with float64 medians), the four deep-walk proofs, and the trim with Python's round(). The Python helpers and constants are deleted; their reasoning moves onto the Rust items. Replay skips a ported helper whose Python is gone, since its callers cover it. Replay: measure_film_edges 60/60, _trim_film_edges 60/60, phase 2 54/54.
The ring-border measurement moves to Rust: the row profile as NumPy's pairwise sum per row, the column profile summed down the columns, the 40th and 90th percentiles with np.percentile's NaN rule, the float32 threshold comparisons (NEP 50), np.argmax's first-max and first-NaN rule, and the opposite-side cap with Python's round(). stats gains pairwise_sum_f32, NumPy's contiguous float32 sum. No recorded corpus reaches these (they run on fallback paths and in batch autocrop), so they were checked against the old Python on the recorded luma planes of three neighbouring functions, with the box as given and shifted 7 px: 720/720 identical. Phase 2 replay 54/54.
_refine_film_roi_by_tiers with _estimate_tier_levels and _longest_run_above, _snap_edge_to_gradient and _smooth_signal move to Rust. np.convolve's float32 arithmetic was probed: a full-overlap output of a window up to 11 is NumPy's small_correlate (float32 sums), every other output BLAS sdot (float32 products summed in float64); windows of 32 and up take OpenBLAS's vector path and are refused, as no caller uses them. np.isclose's tolerance joins the comparison as float32. Checked against the corpus before the swap: snap 60/60, tiers 57/57, smooth_signal against np.convolve for windows 1-31. Replay: phase 2 53/53 (the deleted _estimate_tier_levels drops out).
The Sobel fallback (_refine_frame_bounds with _boundary_candidates and _dark_region_bounds) and the picture-gate orchestration move to Rust, on the already ported primitives. contourArea and boundingRect for integer points join contours.rs. NumPy rules copied: np.percentile on uint8 interpolates in float64; uint8 <= float compares in float64; np.argmax's first-max and first-NaN; max(..., key=contourArea) keeps the first of the largest. No corpus reaches the Sobel fallback, so it was checked against the old Python on the recorded images (whole, the film box, and 5% inside it): 171/171. Replay: _refine_roi_to_image 57/57, film_and_picture 25/25, phase 2 53/53.
shapes.rs ports OpenCV's Sklansky hull (sorted by x, y, then original index, with its cyclic rotation toward ascending indices), the rotating calipers of minAreaRect, RotatedRect::points, boundingRect of float points and fillConvexPoly (8-connected Bresenham outline with clipLine, then the two-edge scanline fill). Those OpenCV files build without FMA on x86-64, so plain float arithmetic matches. atan2, cos and sin are libm's; on 389 contours (recorded masks and random sets) hull, rect and box are identical to cv2, and the fill on 778 masks.
The whole outer-film detector runs in negpy_core: adaptive and structural masks, candidate scoring, consensus, surround gates and the bed snap. The trials run in parallel; candidates join in Python's order. The Python subtree is deleted; tests call negpy_core.select_consensus, film_box_covers_enough and film_surround_is_plausible directly. Replay: 60/60 detect_film_bounds_with_confidence calls bit-identical. One-thread bench against the recorded cv2 Python: 1.77x before the primitive fixes below, 0.78x after. - percentile with a sequence of q: radix select, not a full sort. - uint8 GaussianBlur: kernel-major sums, u16 rows, AVX2 dispatch. - Canny: padded i16 Sobel, no per-pixel clamps. - findContours scan: skips runs eight bytes at a time. - par_zip: rayon blocks instead of a task per element, and a move closure. A capture read through a reference made threshold_binary 4.5 ms instead of 0.1 ms on 1.8 MP.
normalize_log_image drops its Numba kernel. Numba types the span float64 there; a float32 division of the difference rounds the same, so the Rust divides in float64 and casts, as the kernel did. _block_median_grid keeps the 2x2 (sum - min - max) / 2 rule with the sum in float64, and np.median for larger blocks; rayon replaces the strip thread pool. np_median moves to stats.rs and replaces the copy in detect.rs. Replay: _block_median_grid 60/60, normalize_log_image 60/60, and every recorded caller identical. The corpus only reaches 2x2 blocks; larger blocks match the old Python on 27 upscaled and random inputs. Bench (one thread): 0.447x and 0.311x.
measure_clip_fractions, density_above_base's median, the contrast mask tail, diffusion_grid and diffusion_plane call negpy_core. Replay: same on 12 frames (contrast mask and diffusion within 1e-6, through the ledgered log10). Bench: 0.11x to 0.71x of the recorded times. sorted_channel_grid stays np.sort: NumPy's SIMD sort beats a radix port about 4x. The dead _score_contour goes, and with it geometry/logic.py's and normalization.py's cv2 imports.
normalize_linear (LogLuv) is negpy_core::loaders::normalize_linear, bit for bit with np.percentile and np.clip. The streamed LinearRaw preview takes negpy_core's resize_area, morphology_ex and a new ellipse_kernel (cv2.getStructuringElement MORPH_ELLIPSE), checked against cv2 on 3- and 4-sample tiled TIFFs. negpy_core exports OpenCV's border and morphology codes so callers need no cv2 import. PakonLoader._looks_interleaved stays: a 6000-sample probe.
detect_process_mode reads its correlations and percentiles from negpy_core::process::process_mode_stats and casts only the cropped, strided sample to float32. lightbox_level, the chart patch medians (the working decode now runs on the patch boxes only) and the mask overlap test run in Rust; lines_to_score takes negpy_core.box_filter. Replay: same on every Phase 4 recording. Side by side with the old code: detect_process_mode 3.4x and sample_patches 3.3x faster, lines_to_score equal. The percentile helper selects directly up to 131072 values instead of 32768: its histogram costs more than a select below that size. The lens fill search, chart geometry and fit stay Python: orchestration over Rust maps and 24-patch math.
…xel loops cv2.phaseCorrelate is negpy_core::phase::phase_correlate (rustfft and realfft, float32 like OpenCV, both real inputs through real FFTs). cv2 measures the shift from n / 2.0 after a roll by n / 2, half a pixel off along an odd padded DFT size; the port measures from the roll's origin (ledger row, quality gate tests/test_core_phase.py). Side by side with cv2: 1024x683 17.3 -> 5.7 ms, 257x361 1.5 -> 1.2 ms. Also in Rust: mean(axis=2) for HDR and IR alignment, RGB Scan's channel assembly and Bayer channel means (0.96 s -> 0.07 s per raw, bit for bit), the stitch proxy stretch, IR composite, overlap count, gains and clip. rgbscan.py drops its cv2 import. Replay: same on every Phase 5 recording. The machine ran at a load of 3 to 7 during the bench, so the ratios against the recorded times read high: side by side, seam_blend is 120 ms against main's 155 ms.
The normalization processors take their log density from negpy_core.log_density_open (mathf log10, ledgered: abs=2e-6 on the normalized image). The B&W stacks, the local map channels, the mask and dodge map sum, the Lab clip, the camera matrix (OpenBLAS's sgemm order, bit for bit) and the contrast mask expand (resize, place, edge pad and scale in one pass) run in Rust. The display, contact sheet and splash downsamples use resize_area; image_processor.py and preview_manager.py drop cv2. Replay: Phase 6 same except the known 1-ULP frame 0017 of PhotometricProcessor.process. contrast_mask_ev side by side on a 24 MP frame with a crop: 40.4 -> 15.8 ms (every recorded call has no plane). Placement solvers, densitometer, chart curve, carrier profiles and the processors' orchestration stay Python.
negpy_core.resize_kernel copies OpenCV's INTER_LANCZOS4 and INTER_CUBIC taps and weights, in output-row bands that filter their own source rows, with AVX2/AVX-512 passes. PrintService.apply_layout and GPUEngine._process_tiled use it (and resize_area on a shrink); print.py and gpu_engine.py drop cv2. Within 4e-7 of cv2 for Lanczos4 (ledger, quality gate). Side by side with cv2: 6 MP x1.5 42 -> 29 ms, 24 MP x1.25 138 -> 121 ms. The contact sheet's uint8 cubic tile upscale stays on cv2 for now: a float path is 2 to 2.6x slower than cv2's 16-bit fixed point.
negpy_core.resize_u8 copies OpenCV's own uint8 cubic resize (11-bit fixed-point weights, integer rows, float32 column pass over 8-value blocks): bit for bit with cv2 when IPP is off; opencv-python's IPP differs by one level on a few per cent of values (ledger). The row pass sorts each 16-byte window by channel with pshufb and multiplies with pmaddwd; the column pass is AVX2 with saturating packs; scalar fallbacks keep the same integers on other CPUs. The area shrink matches cv2 bit for bit. contact_sheet.py drops cv2. 600x900 to 1100x733 on one pinned core: 1.9 ms against IPP's 2.15 ms. On all cores 0.6 ms against 0.38 ms: the job is short enough that rayon's wake-up latency shows, where cv2's pool spins.
The last cv2 calls move to negpy_core: the overlay's zone outline (findContours RETR_LIST and OpenCV 4.13's approxPolyDP, bit for bit on 1500 fuzzed masks), the SANE row resize (OpenCV's own uint8/uint16 row-only bilinear, bit for bit without IPP), the interactive proxy, the chart fit and IR export downsamples (resize_area), and the retouch structuring elements, morphology codes and run box filter. The chart fit takes its area scale from the rounded output size, as cv2 did not (ledger). The simulated camera's preview JPEG comes from Pillow. Replay: the Phase 4 and IR corpora unchanged.
The scan preview stretches (preview_positive, _preview_u8), focus sharpness, the calibration patch percentile and clip fraction, the sensor calibration clip check and the scan writer's grey average run in negpy_core, each bit for bit with the NumPy code it replaces (uint16, float32 and float64 patches keep NumPy's percentile precision). The scope trace images, semantic ranking, shutter ladder and rebate tone average stay Python: bin grids, embeddings and per-frame values.
The camera meter (frame normalization, picture clip statistics, peak map, grey thumbnails), the raw plateau and linearity clip fractions, the half-frame gutter's column profile, the export border check, the Film Strip's D-max and the placeholder thumbnail's log stretch run in negpy_core, bit for bit with the old NumPy code except where the ledger says. srgb_to_linear, which the manifest had missed and the JPEG and 8-bit TIFF loaders run on whole images, takes a power within an ULP of the correctly rounded value; NumPy's SVML one is up to 4 ULP off (ledger). Every manifest entry is now ported or stays.
…rsions negpy imports no cv2, so opencv-python-headless moves to the dev group (the oracle for the negpy_core tests and the replay tool) and the plustek extra and group: pyopticfilm aligns multi-pass and IR scans with cv2 and skips the alignment without it. Every module imports with cv2 blocked. BOUNDS_VERSION 1 -> 2 and THUMBNAIL_RENDER_VERSION 16 -> 17, the one bump for the whole negpy_core ledger.
opencv-python-headless leaves the dev group: it stays in the plustek extra and group, where pyopticfilm's pass alignment needs it. The oracle tests get it through the plustek group, which the native CI job now syncs; make install already syncs every group.
- get_luminance reads the first three channels of any HxWxC buffer, as the Numba kernel did - find_seams falls back to the average blend on a negpy_core ValueError, as it did on cv2.error - the border-crush check returns False for a buffer too small for its ring - print_dmax takes multiversion for its mul_add loop - percentiles() had no callers left; its tests call negpy_core.percentiles - rgb_to_rgba_into docstring states the constraint only Co-Authored-By: Claude <[email protected]>
The film box was found twice per frame open (base-log meter, then auto crop), and the skew fit ran LSD twice on the same plane. A content-keyed memo (memo.rs: 128-bit key, LRU) now serves repeats of detect_film_bounds, refine_roi_to_image, the detection luma plane and coarse_angles. Other changes, all bit-exact (make replay: 102 functions same; the one PhotometricProcessor diff is also on HEAD): - detection kernels read the input directly instead of copying it - trim_film_edges copies only the lines the walk reads, sides in parallel - np_median takes the parallel histogram selection on large inputs - crop and other image-size collects allocate once - detect builds structural masks beside the threshold trials - LSD precomputes cos/sin and folds NOTDEF into `used` - mask_to_score runs in bands with a halo of the score's reach - neutral_axis maps and filters run in parallel Cache-fair bench, 8 frames, new vs mz/core HEAD: preview 1.45-1.91x, edit 1.81-2.63x, autocrop 1.73-2.58x, optical dust preview 1.23-1.47x, IR dust preview 1.21-1.24x, export 1.00-1.09x. No step slower.
Owner
Author
Performance pass (cf5df18)All changes are bit-exact. Before/after, 8 framesBest of 3, all cores. Every repeat sees new pixels, so no cache serves a repeat.
No step is slower than HEAD. Load preview runs no changed code; its 0.98x is noise. What changed
Not done
|
negpy_core gets a background pool with a quarter of the cores. A thread marked by set_background_thread(True) sends its large work there, so the live preview keeps the other cores. Results do not change. Thumbnail refresh renders two frames at once on that pool, each with its own CPU ImageProcessor. A second frame starts only when free RAM holds its preview decode above MIN_RAM_RESERVE_BYTES; composites and unknown RAM stay at one frame. Signals still leave the worker in frame order. Export encodes up to three files at once. encode_slots() sets the queue depth from free RAM before each submit, so memory pressure drops it to one. A lock keeps two encodes from taking the same output name. The detection memos get more slots, so frames in flight do not evict each other. Headless check on 4 sample RAWs: thumbnails 3.03 s serial, 1.98 s with two frames; outputs bit-equal except DSCF0966.RAF, whose LibRaw decode already differs run to run.
On a 4-core machine a quarter of the cores is one thread, which left the thumbnail refresh slower than the all-core pool it replaced.
Export spent about 420 ms per 23.5 MP frame in PIL's single-threaded JPEG encode, and a module-wide lock around PIL's MAXBLOCK kept the three export encode slots from running JPEGs side by side. negpy_core.encode_jpeg ports libjpeg-turbo's path as Pillow drives it (quality-scaled Annex K tables, fixed-point YCbCr, islow DCT, rounded quantization, optimal Huffman tables) and splits the scan with restart markers, so strips count and encode in parallel. Without restarts the bytes equal Pillow's; with them every decoded pixel does. Size grows by 0.01-0.03 %. Progressive files stay on PIL. 23.5 MP frame: PIL 325 ms, negpy_core 45 ms on all cores. CPU export of the same frame: 1.81 s before this branch's speed work, 1.28 s now.
Owner
Author
Parallel JPEG encode (1c07940) and background pool floor (8f7e427)JPEG.
Restart markers split the scan, so strips count and encode on every core.
Background pool. It now keeps at least two threads, so a 4-core machine gets 2 instead of 1 for thumbnail refresh. Checks: |
Numba left with the Rust port, so no dependency builds llvmlite from source on Intel macOS any more.
…unner checkout v7, cache v6, upload-artifact v7, download-artifact v8, github-script v9, setup-uv v10.3.2 (it no longer tags majors) and action-gh-release v3 run on Node 24; GitHub deprecates Node 20. The inputs the workflows pass exist in each new version. The release builds Linux on ubuntu-24.04: ubuntu-latest moves to Ubuntu 26 from October 19, and a newer glibc would stop the AppImage from starting on older distributions.
The thumbnail test still expected a quarter of the cores (1 on a 4-core runner), and the umask test builds an ExportWorker without __init__, so it lacked the lock that guards output names.
Python runs the interface; wgpu renders on the GPU and negpy_core does every CPU step. Running from source comes first under Getting started, with the release builds after it.
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NegPy's CPU kernels move from NumPy, Numba and OpenCV to
negpy_core, a pure-Rust PyO3 extension (abi3, rayon, no C or C++ dependency). Python keeps config and orchestration.negpy_core/manifest.tomlisported(293) orstays(119). A function that stays works on config or on a few values (profiles, per-frame lists, bin grids).negpyimports nocv2(tests/test_no_cv2.py). opencv-python stays only in theplustekextra and group, because pyopticfilm aligns multi-pass and IR scans with it. The tests use it as their oracle through that group.make replay). Ports that cannot match are innegpy_core/LEDGER.md, with their tolerance or quality gate.BOUNDS_VERSION1 -> 2 andTHUMBNAIL_RENDER_VERSION16 -> 17, one bump for the whole ledger.Ports with a quality gate (decided)
SIFT registration, the line segment detector, the DP seam finder, Navier-Stokes inpaint, phase correlation, the Lanczos4/cubic resize,
zone_anchorsand the seam/feather blends. Each has a pytest gate against cv2 or a known truth.Bugs fixed in the port, not copied
cv2.phaseCorrelatereads every shift half a pixel too far when the padded DFT size is odd. The port measures from the fftShift origin.fxwhile cv2 rounded the output size. The port takes the scale from the rounded size.Speed
Speed gate: never slower with every core in use, against the recorded Python. The machine ran under load during part of the work, so slow-looking ratios were rechecked side by side with main's code. The one accepted exception is the contact sheet's uint8 cubic tile upscale: 1.9 ms on one core against IPP's 2.15 ms, but 0.6 ms against 0.38 ms on all cores (rayon's wake-up on a short job).
Threads and repeated work
memo.rs, 128-bit key, LRU) serves these repeats. The other speed work removes input copies and runs medians, edge walks, the dust score and the neutral-axis meter on all cores. All of it is bit-exact.negpy_corehas a background pool of a quarter of the cores, at least two. A thread marked byset_background_thread(True)sends its large work there, so the live preview keeps the other cores. Results do not change.ImageProcessor. A second frame starts only when free RAM holds its preview decode; composites stay at one frame. Signals leave the worker in frame order.negpy_core(encode_jpeg), strips in parallel through restart markers. It ports libjpeg-turbo's path as Pillow drives it: without restarts the bytes equal Pillow's, with them every decoded pixel does (tests/test_core_jpeg.py). A 23.5 MP frame encodes in 45 ms instead of 325 ms, and JPEG encodes no longer queue on PIL's MAXBLOCK lock. Progressive files stay on PIL.Cache-fair bench on 8 frames (each repeat sees new pixels), against main: preview 4.9-7.9x, edit 5.4-8.3x, autocrop 5.1-8.9x, optical dust preview 4.1-5.3x, IR dust preview 3.5-4.2x, CPU export 1.9-2.6x, GPU export 1.2-1.6x. Thumbnail refresh on 4 sample RAWs: 3.03 s serial, 1.98 s with two frames.
Test plan
make rust(fmt, clippy -D warnings, cargo test),make lint,make typetests/test_core_*.pyand the suites of each touched module)make replayon the corpora for phases 2 to 6 and the IR corpus: same, except ledgered rows and the known 1-ULP frame 0017 ofPhotometricProcessor.processrelease.ymlwithpublish=falseon the four runners (run 40076207275, at afa017f)nativesame-bits job