How the Hexum token measurements were run
Last updated 29 August 2026
What was measured
- Dataset: 12 MBPP + 12 HumanEval tasks (24 total).
- Model used on the published card: GPT-5.6 Sol, $4 / 1M input.
- Without Hexum context tokens 220,393 / $0.88 vs with 27,352 / $0.11 for this run.
- Relative cuts ~85% MBPP (87,586 → 12,973), ~89% HumanEval (132,807 → 14,379).
- Same unit tests passing.
- Measured tokens sent to the model, not a cloud invoice.
- Output-token savings from the skill are not a published study.
- Tiny prompts may save little.
Example turn (not the 24-task suite): 18.2k → 3.1k tokens, 41 files → 6. Download the table: benchmarks/results.csv. Ledger on Hexum benchmarks.
| Suite | Tasks | Without | With Hexum | Cut |
|---|---|---|---|---|
| MBPP | 12 / 12 pass | 87,586 | 12,973 | ~85% |
| HumanEval | 12 / 12 pass | 132,807 | 14,379 | ~89% |
| Combined | 24 / 24 pass | 220,393 | 27,352 | — |
The cut is from hexum_shrink. How hexum_shrink reduces tokens.
Study II — LedgerKit
Independent 43-file Python service (32,768 tokens). Same estimator, same
engine. Five scenarios: wide inspect, surgical splice, cross-file field,
greenfield authoring, failing-test debug. Default inspect 32,768 → 3,220
(90.2%). With max_token_budget=3000, 2,331 (92.9%). Splice vs
full file 1,276 → 115 (91.0%). Greenfield authoring 800 → 800 (0%).
Debug+stderr vs dump 33,162 → 3,652 (89.0%). A focused four-file read can
beat whole-repo shrink. Ledger: ledgerkit.csv.
Figures: benchmarks.
FAQ
How does Hexum reduce tokens?
The agent calls hexum_shrink before it reads widely. The model is sent this turn, not the repository. On MBPP, measured context dropped about 85% with the same tests passing. Example turn: 18.2k → 3.1k tokens, 41 files → 6. See /benchmarks and /methodology.
Did the tests still pass?
Yes. 12 / 12 MBPP and 12 / 12 HumanEval on the published card.
Is that a cloud invoice?
No. It is tokens sent to the model. Output-token savings from the skill are not a published study.
What is tokens saved?
Tokens in minus tokens out, when that is positive. Stripe is billed on tokens saved. Calls that do not save tokens still increment calls and the in/out counters.