83.7% of AI-designed parts fail real physics testing.
We ran six frontier models (GPT-5, Claude Opus 4.5, Gemini 3.1 Pro, Grok 4.5, DeepSeek V4 Pro, Qwen3-Max) through 29 mechanical design tasks, graded by finite-element analysis, not vibes. The best model fails 3 of 5 first attempts. We produce the verified human repair records that teach the fix.
EVAL-0014, a motor mounting bracket under a 415 N load. The contour on the left is the model’s answer; on the right, the human’s repair, re-solved through the same deterministic verifier until every criterion passed.
6 verified repair records in the sample. Every number re-runnable.
Most models never even build the part.
The scary failure isn’t a syntax error. It’s the part that builds, meshes, looks perfect, and then breaks under load. Almost nobody measures that bar.
Most models leak at generation, not physics. The rarest failure, a part that builds, solves, and still breaks, is the one nobody else measures.
pass@1 from 41.4% to 0%.
Same task, two designs, 23 verifier-gated iterations apart.
A human engineer worked the same task 23 times through the identical verifier, failing, adjusting, and re-solving, until it passed. Not one answer: the path to it. That trajectory is the training signal. This is not a head-to-head: the models were measured on 3 attempts each (pass@1 / pass@3 above). The human record is a different artifact, the full repair trajectory, and that trajectory is the data itself.
| Verifier iteration | Peak stress | Mass | Verdict |
|---|---|---|---|
| Attempt 1 | 945 MPa | 320 g | FAIL |
| Attempt 12 | 8,222 MPa | 166 g | FAIL |
| Attempt 23 | 115.8 MPa | 184 g | PASS |
Every attempt is kept: the geometry, the solve, the verdict. This is what a design-decision dataset looks like.
“When stress is applied, either increase the thickness so the part never reaches the breaking point, or redirect the stress into another direction.”the engineer, on his 23-iteration repair (from the record’s engineer note)
Every record, same shape.
{
// excerpt, full schema on request
"record_id": "EVAL-0014",
"engineer": "verified human repair (Fusion 360)",
"before": { "exit": 3, "peak_mpa": 289.8, "sf": 1.04,
"failed": ["peak_stress", "safety_factor", "displacement"] },
"after": { "exit": 0, "peak_mpa": 80.1, "sf": 3.74,
"mass_g": 116.3, "disp_mm": 0.29 },
"iteration_journey": "2 verifier-gated attempts",
"mesh_convergence": [
{ "elements": 1120, "peak_mpa": 72.0 },
{ "elements": 2296, "peak_mpa": 69.6 },
{ "elements": 5584, "peak_mpa": 80.1 }
],
"singularity_check": "clean",
"step_sha256": "e4899252…",
"engineer_note": "what changed, where, why, in the engineer’s words"
}
Before/after metrics, the iteration journey, a 3-size mesh-convergence study, BC integrity, the sha256 of the exact STEP, and the engineer’s note. Red+green STEP files and stress contours ship with every record.
Deterministic, and it admits its own artifacts.
Frozen criteria
Acceptance limits written per task before any model ran. No moving goalposts.
BC re-resolution
Load and mount faces are geometric selectors re-resolved on every part. Move the load face and the run is rejected (exit-4).
Artifact checks
Peaks of 11M and 16M MPa, impossible in steel, were caught as mesh artifacts and excluded. Published, not hidden.
Provenance
Every task traces to a real documented shop failure. 30 cases → 29 tasks, 1:1, machine-checked.
Building a CAD benchmark, rubric, or RLHF verifier?
The same deterministic grader behind these numbers runs as an environment: 29 frozen-criteria mechanical tasks, CalculiX + gmsh grading, boundary-condition integrity, mesh-convergence receipts on every run. Plug your model in and get pass/fail with the receipts, not a vibe score. The 23-step human journey above is exactly the loop your model would run here.
Environments, rubrics, evals: tell us what you’re grading and we’ll show you one of your model’s failing parts get repaired live.
6 verified repair records. Free.
Red+green stress contours, before/after metrics, mesh convergence, sha256’d STEPs, and the engineer’s notes. Or book 15 minutes and watch one of your model’s failing parts get repaired live.
We reply within a day. No spam. One email with the pack, one follow-up.