sema-evalsIndependent evaluations

← babel-relay

babel-relay

20260714T170651223Z-order-20260714 · 2026-07-14 · 900 trials across 6 scenarios

Babel Relay measures what happens when the meaning of a shared term silently drifts while work passes through a relay of agents — and which mechanism, if any, surfaces the drift before it becomes a wrong outcome. About this experiment

Silent divergence — addressed-enforced 0.0% (0/120) vs equal-prose 65.8% (79/120).

Provenance

Model
unsloth/Mistral-Nemo-Instruct-2407-TEE
Provider
llm.chutes.ai
Sema version
0.3.0
Semantic backend
semahash-python-workspace-api
Canonicalization version
v2
Vocabulary root
6bb456b3062d94ec02f0a7a53ca8a0b3aefba78f24140d0129bd6da86553b070
Prompt digest
5f8976a6e93d1816dbd1341d5b906df443692e6c81b3ffe2f97e273f394aa99d
Fixture digest
a8dcdc8d29395b62cfac17b69895b0c71f76f977e3d3c3ccca4a2f9166d97e2c
Implementation commit
f9c9ba05d3d591cf80ced482ce724115c8001c33
Dependency lock digest
581175b4cb04ea42fe87ff06da4bd7e035308c88338266d4b3ea0ba9866546bd
Protocol version
0.3.0
Artifact schema version
0.3.0
Order seed
20260714
Seeds
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29

Results by condition

Condition Trials Drift
trials		 Task
success		 Silent
divergence		 Drift
detected		 Correct
halts		 False
halts
baseline 180 120 57 (31.7%) 116 (96.7%) 4 (3.3%) 4 4
equal-prose 180 120 90 (50.0%) 79 (65.8%) 41 (34.2%) 41 7
opaque-resolver 180 120 93 (51.7%) 81 (67.5%) 39 (32.5%) 39 3
addressed-voluntary 180 120 88 (48.9%) 0 (0.0%) 120 (100.0%) 37 8
addressed-enforced 180 120 169 (93.9%) 0 (0.0%) 120 (100.0%) 120 9

Counts are absolute; parenthetical percentages use each metric's natural denominator (silent divergence and detection over drift trials, task success over all trials).

Silent divergence rate by condition

0%50%100% baseline 96.7% (116/120) equal-prose 65.8% (79/120) opaque-resolver 67.5% (81/120) addressed-voluntary 0.0% (0/120) addressed-enforced 0.0% (0/120)

Task success rate by condition

0%50%100% baseline 31.7% (57/180) equal-prose 50.0% (90/180) opaque-resolver 51.7% (93/180) addressed-voluntary 48.9% (88/180) addressed-enforced 93.9% (169/180)

Per-scenario outcomes

●task success◆correct halt (drift caught)◇false halt▲silent divergence·other failure
Scenariobaselineequal-proseopaque-resolveraddressed-voluntaryaddressed-enforced
inclusive-payment-threshold▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆▲▲▲▲▲▲▲▲◆▲▲▲▲▲▲◆▲▲▲▲▲▲▲▲▲▲▲◆▲▲◆▲◆◆▲▲▲▲▲◆▲▲▲▲◆▲▲▲▲◆▲▲▲▲▲▲▲◆◆▲▲▲◆▲▲▲▲▲▲▲▲▲▲······◆··◆◆············◆·◆◆···◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆
oracle-staleness-boundary▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆▲▲▲▲▲◆▲▲◆▲▲▲▲▲▲▲▲◆◆▲▲▲▲▲▲▲◆▲▲▲▲◆▲▲▲▲▲▲◆▲▲▲▲◆▲▲◆▲▲▲▲▲▲·····◆··◆◆··◆········◆··◆··◆··◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆
refund-recipient-control●●●◇·●●●●●●●●●●◇●●●●●●●●●●●◇●●◇●◇●●●●●●●●●●●◇◇●·●◇●·●●●●●●●●●◇●●●●●●●●●●●●●●●●●●●●●●●●●●·●◇·●●●●●●●●●●◇◇◇◇●●●●●●◇●●●●◇●●●●●●●◇●●◇●●●●·●◇●●●●◇●●●●●◇●●◇
retry-cardinality▲▲▲▲▲▲▲▲▲▲▲▲◆▲▲▲◆▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆▲▲▲▲▲◆▲▲▲▲▲▲▲▲▲▲◆▲◆◆▲▲▲▲▲◆◆▲▲◆▲▲▲▲▲▲▲▲▲◆▲▲▲▲▲◆▲▲▲▲·········◆◆···········◆·◆··◆··◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆
signer-threshold-control●●◇●●●●●●●●●●·●●·●●●●●●●●●●●●●●●●●●●●◇●●●◇●●●●●●●●●·●●●●●●·●●·●●●●●·●●●●◇●●●●●●●●●●●●●●●◇●●●●●●●●●●●●●●●●●●●●●●◇●●●●●●●●●●●●●●●·●◇●●●●●●●●●●◇●●●●●●●●◇
token-decimal-domain▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆◆◆◆◆◆◆◆▲◆◆◆◆◆◆◆▲◆▲◆◆◆◆◆▲◆◆◆◆▲▲◆◆◆◆◆◆▲▲◆▲◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆▲◆▲·◆···◆◆◆◆◆◆◆·◆◆··◆◆·◆◆·◆◆◆◆··◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆

One glyph per trial, ordered by seed, so per-fixture patterns across seeds are visible. Hover a glyph for its seed and outcome.

Raw public derivative

manifest.json summary.json trials.public.jsonl

The public derivative strips raw provider payloads and caps transcript text. Full raw bundles are retained locally only.