Unix tools were built around bytes and newline characters because bytes and lines were what disks and terminals paid for. Language models do not pay by the line; they pay by the o200k_base token11. A 400-line minified SVG or JSON fixture can easily outweigh 2,000 lines of clean Python because punctuation and high-entropy hex strings shatter into single-byte tokens..
When an agent reads a directory, a single committed snapshot or generated asset can silently consume forty percent of its context window on every turn. Nothing in ls -lh or wc -l warns you that a file is three times heavier to a tokenizer than its byte count suggests.
tc: wc(1) for tokens
We wrote tc to drop directly into existing shell pipelines. Give it a list of files or pipe standard input to it, and it prints exact o200k_base token counts in milliseconds:
# Find the heaviest files in a package before handing it to an agent
tc src/**/*.py | sort -n | tail -n 10
tokdump: hexdump(1) for byte-level BPE
When a file has a surprisingly high token-to-byte ratio, tokdump shows you why. It formats standard input into three columns—hex byte offset, integer token IDs, and visible token boundaries—so you can see exactly where the BPE state machine splits UTF-8 sequences, identifiers, and punctuation:
Both tools are small, dependency-light, and open source. Run them locally in your shell or add a five-line check in CI so no pull request accidentally doubles the token weight of your core modules.