[transfix]

Why

AddressSanitizer found two more out-of-bounds reads in the LBIE mesher while I was fixing the compute_qef crash in #762. Neither one is that crash, and the two PRs are independent: they merge cleanly together.

QUAD: assign_refine_sign_quad indexed cells outside the grid

Octree::assign_refine_sign_quad ran find_oc_id on every edge the surface crosses. That includes edges on the volume's boundary, where xyz2octcell returns -1 for the cells outside the grid. The function then:

• called get_err_grad(-1), which reads orig_vol about 18 KB before its start, too far away for ASan to flag;
• read vtx_idx_arr_refine[-1], which ASan does flag;
• could write vtx_idx_arr_refine[-1] = 1.

Any surface that leaves the volume reaches this. CI's QUAD tests never did, because they mesh a sphere inside the box.

heap-buffer-overflow READ of size 4, 4 bytes before the 18724-byte vtx_idx_arr_refine (4681 octree cells)
  LBIE::Octree::assign_refine_sign_quad  octree.cpp:6528
  LBIE::Octree::polygonize_quad          octree.cpp:5958

OPTIMIZATION: getGrad indexed 1-based

Octree::getGrad computes t1, the derivative of the Jacobian with respect to v0 for the edges v1-v0, v2-v0 and v3-v0. Its second factors were indexed 1-based:

• v1[3], v2[3] and v3[3] stood for z and read past the float[3] arrays. ASan reports this as a stack-buffer-overflow in LbieMesherTest.HexaDoMeshOptimize.
• The z line used [2] where it meant y. That read stays in bounds but computes the wrong quantity.

v3[3] reads toward mmv4, the point that optimization()'s line search fills next, and mmv4 is still uninitialized on the first call. That is a likely cause of the couple of NaN vertices arm64 Release left in that test.

What

• assign_refine_sign_quad: both of its loops skip an edge that has a cell outside the grid. is_min_edge already does the same for polygonize_quad, which never makes a quad for such an edge. An edge has either all four of its cells on the grid or two or three off it, so an off-grid cell could never have completed the 4-of-4 or 3-of-4 / opposite-pair conditions anyway. Only the undefined reads change.
• getGrad: the three second-factor indices are lowered by one. All three t1 lines now equal dJ/dv0: checked against central differences on 1000 random tetrahedra, the maximum error is 8e-10. The old z line was off by up to 4.8.

Tests

• New LbieMesherTest.QuadMeshesASurfaceThatLeavesTheVolume meshes a sphere of radius 1.2 in its [-1, 1]³ box as QUAD, at the default error and at a fine one.
• HexaDoMeshOptimize checks finite coordinates again (expect_geoframe_valid instead of the topology-only check). This PR's macOS arm64 Release lane is the real test of that.
• AddressSanitizer (Linux, GCC 13, RelWithDebInfo):
  • without the fixes, both tests are flagged in 40 of 40 runs (OMP_NUM_THREADS = 1, 2, 8 and the default);
  • with the fixes, 40 of 40 runs are clean;
  • lbie_mesher_test is 37/37 clean apart from HexaVerticesStayInsideAVolumeTheSurfaceLeaves, whose read fix(LBIE): keep getVertGrad on the grid (intermittent crash in compute_qef) #762 fixes;
  • with fix(LBIE): keep getVertGrad on the grid (intermittent crash in compute_qef) #762's commit applied as well, all 39 tests are clean;
  • algorithm_test 10/10 and geometry_test 111/111, clean.

Follow-up

• Whichever of this PR and fix(LBIE): keep getVertGrad on the grid (intermittent crash in compute_qef) #762 merges second rewords fix(LBIE): keep getVertGrad on the grid (intermittent crash in compute_qef) #762's test comment. That comment says QUAD "has its own out-of-bounds read"; it should say that SINGLE and QUAD extract no surface from that sphere.
• With both fixes, QUAD on the r = 1.5 sphere is memory-clean but gives 0 vertices, like SINGLE. That is not explained yet.