kotodama · referent binding

Binding Atlas

Base language models acquire the referent of you in five ordered stages. Every base measured reaches the first three with enough training tokens. The fifth stage, tracking you after objects change hands, is inverted in every public model up to 8B at every token count measured, up to 36T. From 14B to 1T parameters it is weak and nearly flat in 10 of the 11 bases with a public size: a zero-shot swap shift of +.01 to +.07, against +.6 to +.9 for static you in the same models. OLMo-3 32B still inverts (−.062). In these models parameter count and depth grow together, so the size at which the stage appears is not separated from depth.

Figure 1

The staircase

The stages appear in the same order in every trajectory measured: OLMo-2 1B, 7B and 13B; SmolLM3 3B; Pythia 410M to 6.9B; the kotodama 1.4B ladder. Each stage is read with a speaker swap. Only the speaker's name on the last line changes, and a bound pronoun moves its answer with the speaker. The hatched stage is where most bases stop.

Figure 2

A size gate, then a plateau

Dynamic you at each model's final checkpoint. A positive swap shift means you follows the addressee; a negative one means you resolves to the speaker. Zero-shot (filled) is negative for every model at or below 8B, for token counts up to 36T, and turns positive between 8B and 14B on both Qwen recipes. All of these are standard-shape transformers whose depth grows with their size; depth at a fixed parameter count is a separate, unmeasured axis here. Above 14B it changes little: +.022 for Qwen3 14B, +.054 for Llama-3.1 405B, +.045 for Kimi-K2 at about 1T. In-context (3-shot, hollow) turns positive at smaller sizes and reaches +.24 for Qwen3 14B.

Figure 3

Stage cost by model size

Training tokens until each stage appears, for one data recipe (OLMo-2) at three sizes. Stage 1 costs about the same at 1B, 7B and 13B. Stage 3 takes about ten times fewer tokens at 13B than at 1B. Stage 5 is absent at the last checkpoint of all three (4T to 5T tokens). The kotodama 1.4B ladder reaches stage 1 at about 3B tokens and has not reached stage 3 by 12B.

Table 1

Levers in the kotodama stack

4B-token arms on one data order; one seed per arm except the fitted boundaries (three seeds). Set-binding and attribution are capacity probes from the full atlas. Depth, looping and fitted boundaries raise these probes; dialogue-heavy early data does not. No arm ran long enough to reach stage 5, so the levers' effect on it is unmeasured.

Full atlas

Every reachable model against every referent-binding probe, chance-normalized (0 = guessing, 1 = perfect). Each cell opens the exact text the model saw.

View
Condition
Items
−11 no data small sample (n < 30, read loosely)

Appendix

Reading the grid
  • Score = (accuracy − chance) / (1 − chance). 0 is guessing among the candidates, 1 is perfect, below 0 is worse than guessing (a systematic wrong answer — usually the shortcut).
  • Deixis columns come in pairs: the I/you/my version and a 3rd-person twin narrating the same events with names. The twin is the control — same memory load, no perspective. The Perspective gap view is twin minus deictic: how much a model loses purely by having to take a seat.
  • Shortcut-free keeps only items where the answer is neither the last-mentioned person nor the last speaker. Most deixis "success" lives in those two shortcuts; with them removed, nearly everything falls to chance on query-side you — up to 405B.
  • P3 / P2 = three or two people in the scene. P2 you is the structural case: no name tells you who you is; only the turn structure does.
  • Open models are scored by log-probability over the candidates. API models (Claude, OpenRouter rows) can't expose that, so they answer in words: reflex = "reply with only the next word", deliberate = "think it through, then answer". Compare within a scoring type first.
Findings by probe
    Caveats
      Methods and instruments