VocabX Cuts Qwen3-4B Inference Cost 77% With a Smaller Domain Vocabulary
VocabX cut Qwen3-4B grader inference costs 77% by shrinking vocabulary and outputs, while its quality score rose from 0.703 to 0.810 in RTX 5090 tests.
Summary
On October 21, 2026, Dharma-AI previewed VocabX for its production Portuguese essay grader built on Qwen3-4B. Seven tokenizer versions kept the four-billion-parameter base model, data, recipe and seed unchanged; each handled the same 500 requests on an RTX 5090 with vLLM, with cost calculated from runtime. VocabX cut average responses from 350 to 101 tokens, inference cost 77.0%, and a grading and screening bench score from 0.703 to 0.810. The original 151,669-entry vocabulary gave the LM head almost 10% of parameters; output rows fell from about 152,000 to 65,000.
VocabX values an output token at 7.4 times an input token, the measured decode to prefill cost ratio, then combines domain-specific subword merges with fixed-budget macro tokens for repeated words, phrases, lines, sentences and punctuated structures including JSON. Five miners found these spans; punctuation structures contributed most across eight corpora and 17.5 points on code, while omitting miners cut grader token reduction by 55.6 points. Statistical estimates rescue candidates missed by a 400-document sample: only 16.6% appeared in a contracts corpus, and an expression occurring 37 times in 50,000 documents appeared about 26% of the time. This lifted contracts reduction from 44.7% to 52.8%.
Fusing each grade with surrounding JSON scored 0.689 and cut cost 73.3%; keeping the grade separate added two response tokens but reached 0.810 and 77.0%, showing decision-bearing tokens resist compression. Across contracts, medical papers, scientific papers, news and Python code using Qwen3, Llama-3.1 and Gemma-3, VocabX beat FVT, MWT and Liquid AI's in-place BPE in all 15 combinations by 1.2 to 9.3 points. Its pruned version led estimated cost by 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, it reached 62.6% reduction with 113,426 entries versus 61.8% with 207,803 for the best in-place BPE. Under-review results may change before the forthcoming paper.
Positives
- 77.0% lower measured inference cost came without changing Qwen3-4B, the training data or the recipe.
- 350 to 101 average response tokens sharply reduced the production grader's decoding workload.
- 0.810 bench performance surpassed the production grader's 0.703 score after decision-bearing grade tokens were preserved.
- About 65,000 output rows replaced roughly 152,000, reducing a layer that represented almost 10% of model parameters.
- All 15 domain and model combinations favored VocabX over FVT, MWT and Liquid AI's in-place BPE.
- 62.6% contracts token reduction used 113,426 entries, beating in-place BPE's 61.8% with 207,803 entries.
Risks & concerns
- 0.689 bench performance resulted when grades were fused into JSON macros, showing aggressive compression can damage core decisions.
- 7.4 times higher output-token cost varies with hardware, batch size, context length and workload.
- Only 16.6% of contracts candidates appeared in a 400-document sample, creating substantial selection blind spots without statistical estimation.
- 55.6 points of grader token reduction disappeared when all five specialized span miners were not used.
- Under-review measurements may change before the VocabX paper is published.