sema-evalsIndependent evaluations

babel-relay

Babel Relay measures what happens when the meaning of a shared term silently drifts while work passes through a relay of agents — and which mechanism, if any, surfaces the drift before it becomes a wrong outcome.

How to read the results

Each trial sends a small contract definition through a three-hop relay (specification → implementation → audit). In drift scenarios, one field of the definition is silently mutated at a relay boundary; the audit agent must then decide whether the implementation matches the definition it can see. The primary endpoints are silent divergence (drift that no agent ever surfaced) and task success, scored by a frozen deterministic parser — no LLM judge.

The five conditions decompose where protection could come from, so that a content effect is never attributed to hashing, and a detection effect is never attributed to enforcement.

baseline
No semantic material beyond the task itself. Measures the floor.
equal-prose
The full definitions travel inline as prose. Isolates the value of content alone.
opaque-resolver
The same definitions behind a content-free lookup ID. Controls for compact references without content addressing.
addressed-voluntary
Content-addressed references; agents may check them but nothing compels action. Isolates detection.
addressed-enforced
Content-addressed references plus a runtime that refuses to proceed while reference and definition disagree. Isolates enforcement.

Runs are labelled by evidence tier — deterministic harness validation, exploratory model pilot, or preregistered experiment — and nothing stronger than its label should be inferred from any table.

Findings so far

Effect sizes recomputed from each promoted pilot's public trials — differences in task-success rate, in percentage points. Exploratory, not confirmatory; read the interpretation below.

Model Trials Content
(pp)
Lookup
(pp)
Detection
alone (pp)
Enforce­ment
(pp)
Addressed
silent div.
Enforced
false halt
MiniMax-M2.5-TEE 900 +27.2 -2.2 +0.6 +38.9 0/240 (0.0%) 0/180 (0.0%)
Mistral-Nemo-Instruct-2407-TEE 150 +16.7 -10.0 -16.7 +53.3 0/40 (0.0%) 1/30 (3.3%)
Mistral-Nemo-Instruct-2407-TEE 900 +18.3 +1.7 -1.1 +45.0 0/240 (0.0%) 9/180 (5.0%)
MiniMax-M2.5-TEE · 900 trials Mistral-Nemo-Instruct-2407-TEE · 900 trials
-1001020304050 Content+27.2+18.3 Lookup-2.2+1.7 Detection alone+0.6-1.1 Enforcement+38.9+45.0

Percentage points of task success. Two dots per row — one per model, using each model's largest promoted pilot — joined by a connector; the heavier line marks zero.

Interpretation — as of 2026-07-15

These are exploratory pilots — a consistent signal, not a proven effect. Nothing below is confirmatory.

Across a 12B model and a frontier-class model, baseline drift behavior is capability-resistant: both silently execute drifted definitions in over 90% of baseline drift trials. Supplying full definitions as prose helps (+18 to +27 points) but requires no content addressing, and compact lookup is a wash. Content-addressed references made drift visible in every addressed trial across both pilots — yet visibility alone barely moved task success. Enforcement converted that detection into outcomes (+39 to +45 points over voluntary checking), reaching 93.9% and 98.9% task success respectively, with enforced-arm false halts at 5% and 0%.

The preregistered, confirmatory test of these effects has not yet run. Until it has, the strongest supportable claim is that the mechanism decomposition behaves as designed at two very different capability levels.

DateModeModel / provider Trials Silent div.
enforced
Silent div.
equal-prose
Report
2026-07-15 Model pilot MiniMaxAI/MiniMax-M2.5-TEEllm.chutes.ai 900 0.0% (0/120) 54.2% (65/120) Report
2026-07-14 Model pilot unsloth/Mistral-Nemo-Instruct-2407-TEEllm.chutes.ai 900 0.0% (0/120) 65.8% (79/120) Report
2026-07-14 Model pilot unsloth/Mistral-Nemo-Instruct-2407-TEEllm.chutes.ai 150 0.0% (0/20) 65.0% (13/20) Report

Silent divergence is the share of drift trials where injected drift went undetected and the relay proceeded — lower is better.