EC4E — Edge Compute for Education

Handwritten-Math Extraction and Diagnostic Judge

EC4E is a notebook-first experiment evaluating whether a small, locally run vision-language model can extract handwritten mathematics and diagnose the first error in a student's work. The measured result is negative: this frozen configuration did not demonstrate reliable extraction or diagnosis.

Question

Can one local 2B vision-language model first extract handwritten work and then identify the first student error without seeing the deterministic grade or manual label?

Dataset and truth

Thirty current-pixels pages; deterministic grading yields 22 correct and 8 incorrect. Image 05 is 3*x+12 and is correct.

The 30 problem statements and handwritten pages were created by the project author specifically for this experiment. They are synthetic examples of student work, not submissions collected from students or educational records. The author created the handwriting, photographs, transcriptions, correctness labels, and failure-mode annotations. Visual review found no direct identifiers, and the JPEG files contain no EXIF, GPS, XMP, IPTC, or comment metadata. Handwriting can nevertheless be identifying in some contexts; this dataset must not be used for identity inference or re-identification.

Method

The notebook extracts all 30 pages first, freezes that cohort, and releases the model before opening evaluator-only answers. An exact mechanical grader then keeps current-pixels student correctness separate from extraction quality. The same pinned model receives one native record_diagnosis tool opportunity for each of 30 normal cases and 30 deterministically constrained incorrect-problem controls. Only after the diagnostic model is released does CPU-only evaluation open the manual labels and reconstruct the metrics below from durable evidence.

Measured results

Metric	Numerator	Denominator	Ratio
Extraction answer coverage	1	30	0.03333333333333333333333333333
Extraction answer agreement	0	30	0
Student-math agreement	0	30	0
False student-error rate	0	22	0
Transcription agreement	0	30	0
Primary-error-source agreement	0	30	0
First-problematic-step agreement	0	1	0
Failure-mode agreement	0	30	0
Incorrect-problem detection	0	30	0
Normal problem-match accuracy	0	30	0
False incorrect-problem rate	0	30	0

Invalid diagnoses: 60. Uncertain diagnoses: 0.

Only one extraction contained a deterministically gradable final answer, and it did not agree with the current-pixels student answer. Every diagnostic result was invalid, so all end-to-end diagnostic agreements and detections are misses. The zero false-positive numerators mean that no accepted explicit false claim was recorded; they do not indicate successful diagnosis.

Representative cases

See Representative cases. Stable case_id rows show a gold-correct page, an extraction/transcription issue, and an incorrect-problem control. No row is presented as a successful LM diagnosis: every diagnosis result was invalid, and extraction precedence made transcription the audited primary source for this frozen cohort. The full normal table still exposes the eight deterministically incorrect student pages.

Local cost

Phase	Input tokens	Output tokens	Elapsed seconds	Peak reserved bytes
Extraction	47,598	9,074	1336.4852782159996	2,243,952,640
Diagnosis	117,804	10,466	1392.343742844997	2,353,004,544

Measured configuration: NVIDIA RTX 500 Ada Generation Laptop GPU; driver 573.44; Torch 2.6.0+cu124; CUDA runtime 12.4; Qwen/Qwen3.5-2B revision 15852e8c16360a2fea060d615a32b45270f8a8fc; NF4 with BF16 double quantization; SDPA attention; one full_1024 view with visual-token bounds 256–2048; batch size 1.

No model weights are distributed. The pinned Qwen3.5-2B revision and all Python dependencies remain subject to their upstream licenses.

Limitations

Extraction and judging use the same model, which can introduce correlated bias. The dataset has only 30 pages. Each generation gets one attempt, so parser failures remain failures. Manual labels involve human judgment. Timing and memory are specific to the measured local hardware. Most importantly, the LM judge does not replace deterministic grading.

Reproduce

Create an isolated environment and install the pinned runtime:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Then reproduce the frozen CPU-default findings without loading the model:

run_tmp="$(mktemp -d /tmp/ec4e-jupyter.XXXXXX)"
chmod 700 "$run_tmp"
TMPDIR="$run_tmp" JUPYTER_RUNTIME_DIR="$run_tmp" \
  HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
  jupyter nbconvert --execute --to notebook \
    --output output.ipynb --output-dir "$run_tmp" \
    --ExecutePreprocessor.kernel_name=python3 \
    --ExecutePreprocessor.timeout=900 \
    notebook.ipynb

Using a Linux-native temporary directory avoids WSL permission failures when Windows TEMP or TMP variables are inherited. Measured inference flags require the pinned local model cache, an exclusive compatible 4 GB CUDA GPU, and the leakage and memory gates implemented in the notebook.

Repository layout

• notebook.ipynb: the only executable project source and the rendered report.
• problems.csv, submissions.csv, and images/: the synthetic dataset.
• evaluation/: references, manual labels, durable run evidence, and the active run pointer.
• requirements.txt: the pinned measured Python environment.

Licensing

Notebook code and prose are licensed under the Apache License 2.0. Dataset artifacts and data rendered in notebook outputs are licensed under CC BY 4.0. Cite the dataset as "EC4E — Edge Compute for Education, Will Hayden (@williamshayden), 2026."