How the Hexum token measurements were run
Last updated 12 September 2026
What was measured
- Dataset: 12 MBPP + 12 HumanEval tasks (24 total).
- Model used on the published card: GPT-5.6 Sol, $4 / 1M input.
- Without Hexum context tokens 220,393 / $0.88 vs with 27,352 / $0.11 for this run.
- Relative cuts ~85% MBPP (87,586 → 12,973), ~89% HumanEval (132,807 → 14,379).
- Same unit tests passing.
- Measured tokens sent to the model, not a cloud invoice.
- Output-token savings from the skill are not a published study.
- Tiny prompts may save little.
Example turn (not the 24-task suite): 18.2k → 3.1k tokens, 41 files → 6. Download the table: benchmarks/results.csv. Ledger on Hexum benchmarks.
| Suite | Tasks | Without | With Hexum | Cut |
|---|---|---|---|---|
| MBPP | 12 / 12 pass | 87,586 | 12,973 | ~85% |
| HumanEval | 12 / 12 pass | 132,807 | 14,379 | ~89% |
| Combined | 24 / 24 pass | 220,393 | 27,352 | — |
The cut is from hexum_shrink. How hexum_shrink reduces tokens.
Study II — LedgerKit
Independent 43-file Python service (32,768 tokens). Same estimator, same
engine. Five scenarios: wide inspect, surgical splice, cross-file field,
greenfield authoring, failing-test debug. Default inspect 32,768 → 3,220
(90.2%). With max_token_budget=3000, 2,331 (92.9%). Splice vs
full file 1,276 → 115 (91.0%). Greenfield authoring 800 → 800 (0%).
Debug+stderr vs dump 33,162 → 3,652 (89.0%). A focused four-file read can
beat whole-repo shrink. Ledger: ledgerkit.csv.
Figures: benchmarks.
FAQ
How does Hexum cut what I pay the model?
Your agent stops dumping the whole repo into the prompt. Hexum sends this task, not every file. On the published card, MBPP context dropped 85.2% and the combined 24-task cut is 87.6%, with 24/24 tests still passing. Tokens are ceil(utf8_bytes/4), not a provider invoice. See /benchmarks.
Did the tests still pass?
Yes. 12 / 12 MBPP and 12 / 12 HumanEval on the published card.
Is that a cloud invoice?
No. It is tokens sent to the model. Output-token savings from the skill are not a published study.
What does the dashboard mean by tokens saved?
How much smaller the reply was than what the agent sent, when that is a cut. Your invoice is still based on what the agent sent Hexum, not on the savings.