Measuring the cost of structural drift
Most agent benchmarks test models on isolated, self-contained problems. Real engineering happens inside long-lived repositories where the hardest part of a change is figuring out which four files matter and which forty files are historical accidents.
We measure legibility directly from production work in Fractal OS by linking every merged commit back to the sub-agent session that produced it11. Every session records first-edit-turn, total read-calls, edit-calls, o200k_base tokens consumed, and whether a human modified the commit before merge..
| Metric | Clean subsystem (<15k tokens) |
Drifted subsystem (48k tokens) |
Post-srclint cleanup |
|---|---|---|---|
| Median file reads before 1st edit | 4 reads | 29 reads | 6 reads |
| Median wall-clock time to 1st edit | 38s (Turn 6) | 279s (Turn 21) | 54s (Turn 8) |
| Inference cost per merged PR | $1.40 | $6.75 | $1.85 |
| PRs requiring human rework before merge | 4% | 38% | 5% |
Continuous invariants in Postgres
We track these metrics continuously in Postgres using Invariant, flagging subsystems as soon as median reads-before-first-edit or token weight per module crosses our threshold:
Reproducing on your own repository
You can inspect the token surface area and structural boundaries of any local git repository today using tc, tokdump, cq, and srclint.