babel-relay
Babel Relay measures what happens when the meaning of a shared term silently drifts while work passes through a relay of agents — and which mechanism, if any, surfaces the drift before it becomes a wrong outcome.
How to read the results
Each trial sends a small contract definition through a three-hop relay (specification → implementation → audit). In drift scenarios, one field of the definition is silently mutated at a relay boundary; the audit agent must then decide whether the implementation matches the definition it can see. The primary endpoints are silent divergence (drift that no agent ever surfaced) and task success, scored by a frozen deterministic parser — no LLM judge.
The five conditions decompose where protection could come from, so that a content effect is never attributed to hashing, and a detection effect is never attributed to enforcement.
baseline- No semantic material beyond the task itself. Measures the floor.
equal-prose- The full definitions travel inline as prose. Isolates the value of content alone.
opaque-resolver- The same definitions behind a content-free lookup ID. Controls for compact references without content addressing.
addressed-voluntary- Content-addressed references; agents may check them but nothing compels action. Isolates detection.
addressed-enforced- Content-addressed references plus a runtime that refuses to proceed while reference and definition disagree. Isolates enforcement.
Runs are labelled by evidence tier — deterministic harness validation, exploratory model pilot, or preregistered experiment — and nothing stronger than its label should be inferred from any table.
Findings so far
Effect sizes recomputed from each promoted pilot's public trials — differences in task-success rate, in percentage points. Exploratory, not confirmatory; read the interpretation below.
| Model | Trials | Content (pp) |
Lookup (pp) |
Detection alone (pp) |
Enforcement (pp) |
Addressed silent div. |
Enforced false halt |
|---|---|---|---|---|---|---|---|
MiniMax-M2.5-TEE |
900 | +27.2 | -2.2 | +0.6 | +38.9 | 0/240 (0.0%) | 0/180 (0.0%) |
Mistral-Nemo-Instruct-2407-TEE |
150 | +16.7 | -10.0 | -16.7 | +53.3 | 0/40 (0.0%) | 1/30 (3.3%) |
Mistral-Nemo-Instruct-2407-TEE |
900 | +18.3 | +1.7 | -1.1 | +45.0 | 0/240 (0.0%) | 9/180 (5.0%) |
Percentage points of task success. Two dots per row — one per model, using each model's largest promoted pilot — joined by a connector; the heavier line marks zero.
Interpretation — as of 2026-07-15
These are exploratory pilots — a consistent signal, not a proven effect. Nothing below is confirmatory.
Across a 12B model and a frontier-class model, baseline drift behavior is capability-resistant: both silently execute drifted definitions in over 90% of baseline drift trials. Supplying full definitions as prose helps (+18 to +27 points) but requires no content addressing, and compact lookup is a wash. Content-addressed references made drift visible in every addressed trial across both pilots — yet visibility alone barely moved task success. Enforcement converted that detection into outcomes (+39 to +45 points over voluntary checking), reaching 93.9% and 98.9% task success respectively, with enforced-arm false halts at 5% and 0%.
The preregistered, confirmatory test of these effects has not yet run. Until it has, the strongest supportable claim is that the mechanism decomposition behaves as designed at two very different capability levels.
| Date | Mode | Model / provider | Trials | Silent div. enforced |
Silent div. equal-prose |
Report |
|---|---|---|---|---|---|---|
| 2026-07-15 | Model pilot | MiniMaxAI/MiniMax-M2.5-TEEllm.chutes.ai |
900 | 0.0% (0/120) | 54.2% (65/120) | Report |
| 2026-07-14 | Model pilot | unsloth/Mistral-Nemo-Instruct-2407-TEEllm.chutes.ai |
900 | 0.0% (0/120) | 65.8% (79/120) | Report |
| 2026-07-14 | Model pilot | unsloth/Mistral-Nemo-Instruct-2407-TEEllm.chutes.ai |
150 | 0.0% (0/20) | 65.0% (13/20) | Report |
Silent divergence is the share of drift trials where injected drift went undetected and the relay proceeded — lower is better.