[davidtai]

Summary

Upstream Python tests draw their random input from unseeded mx.random and np.random, so each run tests other input. Tests that compare with a reference under a fixed tolerance fail now and then. Two did in our Fork Tests:

• test_ops.TestOps.test_scans (test_ops.py:2900, float16/bfloat16 cumsum against a tri-matrix sum, rtol 0.02, atol 1e-3): failed with the CPU and the GPU in run 40035016891 attempt 1 (Compute float16 and bfloat16 floor divide in float32 #40 head 842d49a); attempt 2 green on the same tree.
• test_double.TestDouble.test_gemm (test_double.py:170, float32 8x8 matmul against float64, mx.allclose defaults rtol 1e-5, atol 1e-8): failed in python-cpu (0) of main run 40090732192 (2668d98; tree = 83dbdb0, which was green).

Fix: MLXTestCase.setUp seeds mx.random and np.random with 0 before each test. Each test now draws the same input in every run. Tolerances do not change.

test_gemm is tolerance, not a stale output

• Both CPU paths clear C: float32 and float64 matmul reach matmul<float> / matmul<double> in mlx/backend/cpu/gemms/cblas.cpp (dispatch_inexact_types in matmul.cpp; bnns.cpp serves only float16/bfloat16). Both call clear_gemm_output before cblas_sgemm (line 33) and cblas_dgemm (line 77) when beta is 0 (fork section "CPU BLAS gemm clears C when beta is 0").
• Arithmetic: each element of an 8x8 product of N(0,1) inputs is a sum of 8 products, with partial sums of order 1. The float32 rounding of those partial sums is about 1e-7. Where the product element y is near zero, the tolerance is 1e-8 + 1e-5 * |y|, which is less than 1e-7 when |y| < 0.01. The seed search found 152 failing seeds of 2000 on the CPU (7.6% of runs). Every failing element was near zero, for example:
  • seed 10: element (4, 1), float64 -0.00444616983, float32 -0.00444611209, |diff| 5.77e-08, tolerance 5.45e-08
  • seed 69: element (5, 0), float64 0.00049052207, float32 0.000490445935, |diff| 7.61e-08, tolerance 1.49e-08
  • seed 291: element (1, 6), float64 0.000366440916, float32 0.000366836466, |diff| 3.96e-07, tolerance 1.37e-08

test_scans

The seed search found 4 failing seeds of 100 with the GPU (14, 55, 67, 83) and 2 of 100 with the CPU (14, 67). In each, the bfloat16 cumsum differs from the bfloat16 tri-matrix sum by more than the tolerance, for example seed 14 (GPU): element (4, 22), tri-sum 10.9375, cumsum 11.1875, |diff| 0.25, tolerance 0.22. One bfloat16 ulp at 8 to 16 is 0.0625, and a sequential sum of 32 bfloat16 values can round by a few ulps.

Commits

• 97d3d774 (diagnostic, removed by the next commit): test_gemm_seed_search and test_scans_seed_search run the unchanged upstream tests under seeds 0..1999 (gemm, CPU) and 0..99 (scans) and fail with each failing seed and its worst element. Fork Tests https://github.com/Layr-Labs/mlx/actions/runs/40118435105: FAILED as intended, python-gpu (0) job 120944935552 (scans), python-cpu (0) job 120944935584 (scans and gemm). Seed 0 passes both tests.
• 9787fed4 (fix): seeds in MLXTestCase.setUp; the search tests are removed; test_double.py and test_ops.py are equal to main. ALL GREEN: Fork Tests https://github.com/Layr-Labs/mlx/actions/runs/40119439820 (test_gemm and test_scans passed in python-cpu (0) and python-gpu (0)), Build and Test https://github.com/Layr-Labs/mlx/actions/runs/40119439885, Fork Diff Check https://github.com/Layr-Labs/mlx/actions/runs/40119439814.

What the seed changes

• Tests that seed their own generators (test_random.test_global_rng, the RNG tests in test_compile.py, the np.random.seed calls in test_ops.py and others) still do; their seed comes after setUp.
• Draws inside one test still differ from each other; only the start state of each test is fixed.
• Lost: the run-to-run variation of input. A test now checks one input set per run, the same in every run. A tolerance that is too tight for some inputs no longer shows as a flake; the seed search above is how to find one.
• The three test classes that derive from unittest.TestCase directly (test_device.TestDefaultDevice, test_extension.TestMetalExtension, test_metal_threadgroup_bounds.ThreadgroupBoundsTests) do not get the seed. None of them draws random input.

fork.yaml: section "Seeded Python tests".

Draft upstream issue (not opened)

Python tests use unseeded random input; test_gemm and test_scans fail now and then

python/tests draw from unseeded mx.random and np.random. Two tests fail for some inputs:

• test_double.test_gemm compares a float32 8x8 matmul with the float64 result under mx.allclose defaults (rtol 1e-5, atol 1e-8). When an element of the product is near zero, the tolerance (about 1e-8 + 1e-5 * |y|) is below the float32 rounding of the partial sums (about 1e-7). On the CPU it fails for 152 of seeds 0..1999, for example mx.random.seed(10): element (4, 1), float64 -0.00444616983, float32 -0.00444611209.
• test_ops.test_scans compares the bfloat16 cumsum with a bfloat16 tri-matrix sum under rtol 0.02. It fails for seeds 14, 55, 67, 83 of 0..99 on the GPU and 14, 67 on the CPU (with mx.random.seed(s); np.random.seed(s) before the test).

Possible fix: seed both generators in MLXTestCase.setUp (for example with 0), so each test sees the same input in every run. Or compare test_gemm with an atol that matches float32 rounding at the scale of the inputs.

🤖 Generated with Claude Code