← Products

CLI · Rust / Python

Source ↗

tokdump

Like hexdump, but for tokens. Prints byte offsets, token IDs, and token boundaries in a three-column hexdump(1) layout.

Install

brew install omarish/tap/tokdump
tokdumptokdump showing where a string splits into tokens.tokdump$printf 'Hello, world!' | tokdump0000000 13225 11 2375 0 Hello|,|·world|!0000004$printf '💯🔥' | tokdump -n0000000 31446 107 \xf0\x9f\x92|\xaf0000002 96606 🔥0000003$Three columns: offset, token IDs, and thetext they cover. The layout hexdump has usedsince 1978.Two emoji. The first costs two tokens andsplits mid-character; the second is a singletoken.Narrow: two token IDs per row instead offour.