sema-evalsIndependent evaluations

← babel-relay

babel-relay

20260715T070249170Z-order-20260714 · 2026-07-15 · 900 trials across 6 scenarios

Babel Relay measures what happens when the meaning of a shared term silently drifts while work passes through a relay of agents — and which mechanism, if any, surfaces the drift before it becomes a wrong outcome. About this experiment

Silent divergence — addressed-enforced 0.0% (0/120) vs equal-prose 54.2% (65/120).

Provenance

Model
MiniMaxAI/MiniMax-M2.5-TEE
Provider
llm.chutes.ai
Sema version
0.3.0
Semantic backend
semahash-python-workspace-api
Canonicalization version
v2
Vocabulary root
6bb456b3062d94ec02f0a7a53ca8a0b3aefba78f24140d0129bd6da86553b070
Prompt digest
5f8976a6e93d1816dbd1341d5b906df443692e6c81b3ffe2f97e273f394aa99d
Fixture digest
a8dcdc8d29395b62cfac17b69895b0c71f76f977e3d3c3ccca4a2f9166d97e2c
Implementation commit
9f57dc2fb0962b0f2938c52ae6d12c2451b3e5b8+dirty
Dependency lock digest
8492976045dc82b4c2cd05728a67b59c5e2a7c1ab03b451531bf34a010aadd27
Protocol version
0.3.0
Artifact schema version
0.3.0
Order seed
20260714
Seeds
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29

Results by condition

Condition Trials Drift
trials		 Task
success		 Silent
divergence		 Drift
detected		 Correct
halts		 False
halts
baseline 180 119 58 (32.2%) 109 (91.6%) 10 (8.4%) 10 12
equal-prose 180 120 107 (59.4%) 65 (54.2%) 55 (45.8%) 55 7
opaque-resolver 180 119 103 (57.2%) 72 (60.5%) 47 (39.5%) 47 3
addressed-voluntary 180 120 108 (60.0%) 0 (0.0%) 120 (100.0%) 52 4
addressed-enforced 180 120 178 (98.9%) 0 (0.0%) 120 (100.0%) 120 0

Counts are absolute; parenthetical percentages use each metric's natural denominator (silent divergence and detection over drift trials, task success over all trials).

Silent divergence rate by condition

0%50%100% baseline 91.6% (109/119) equal-prose 54.2% (65/120) opaque-resolver 60.5% (72/119) addressed-voluntary 0.0% (0/120) addressed-enforced 0.0% (0/120)

Task success rate by condition

0%50%100% baseline 32.2% (58/180) equal-prose 59.4% (107/180) opaque-resolver 57.2% (103/180) addressed-voluntary 60.0% (108/180) addressed-enforced 98.9% (178/180)

Per-scenario outcomes

●task success◆correct halt (drift caught)◇false halt▲silent divergence·other failure
Scenariobaselineequal-proseopaque-resolveraddressed-voluntaryaddressed-enforced
inclusive-payment-threshold▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆◆▲◆▲▲▲◆▲◆▲▲▲◆▲▲◆▲◆◆▲◆▲▲◆▲▲◆◆▲▲◆◆▲▲▲▲◆◆▲▲▲▲▲▲▲◆◆◆▲◆◆◆··········◆◆·◆◆◆◆◆·◆····◆·◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆
oracle-staleness-boundary▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆▲◆◆▲▲◆▲▲◆▲▲◆▲▲▲◆▲◆▲▲▲◆▲▲▲◆▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲◆▲▲···◆◆··◆······◆·····◆·········◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆
refund-recipient-control●●●◇●●●●●●◇●●◇◇●◇●◇◇●●◇●●●●●◇◇●◇●◇●●●●●●●◇●·●●●●●●●●◇●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●◇●●●●●●◇●●●●·●●●●●●●●●●●●●●●●●●●●●●●●●
retry-cardinality▲▲▲▲▲▲▲▲▲▲▲◆▲▲▲▲▲◆◆▲▲◆▲▲▲▲◆▲◆▲▲▲▲◆▲▲▲▲◆▲◆▲▲▲◆◆▲▲▲▲▲◆▲▲◆▲▲▲▲▲▲▲▲▲▲◆▲▲◆▲▲▲◆◆▲▲▲▲▲▲▲▲◆▲▲▲▲▲▲◆·◆····◆·◆◆··········◆····◆····◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆
signer-threshold-control●●●●●●●●●◇●◇●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●◇●●●●●●◇●●●●●●◇●●●●◇●●●●●●●●●◇●●●·●●●●●●●●●●●●◇●●●●●●●●●●◇●●●●●●●●●●●●●●◇●●●●●●●●●●●●●●·●●●●●●●●●●●●●●●●●●●
token-decimal-domain▲▲▲▲▲▲▲▲▲▲▲◆▲▲·▲▲▲▲◆▲◆◆▲▲▲▲▲▲▲◆◆◆◆◆◆▲◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆·◆◆◆◆◆◆◆◆◆◆·◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆·◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆◆

One glyph per trial, ordered by seed, so per-fixture patterns across seeds are visible. Hover a glyph for its seed and outcome.

Raw public derivative

manifest.json summary.json trials.public.jsonl

The public derivative strips raw provider payloads and caps transcript text. Full raw bundles are retained locally only.