NeoMME Unifies Text and Images, Cuts Retrieval Storage 255 Times
NeoMME's 260M and 800M encoders unite text and images, doubling page throughput and shrinking retrieval storage 255 times while retaining over 95% quality.
Summary
On September 3, 2026, Tony Wu and Aurélien Lac introduced NeoMME, 260M and 800M multilingual multimodal encoders trained with H Company compute. One bidirectional Transformer processes factorized text tokens and raw 32×32 image patches without pretrained vision or language components. Its 16,384-token context fits up to two 3840×2160 images, preserves dynamic image resolution, and combines sliding-window attention with global attention every sixth layer and at the end. A 131,000-token BPE vocabulary covers multilingual text, code, mathematics, and image transcripts. Masked discrete-diffusion pretraining used about 524 billion packed tokens, including 290 billion text-only tokens, versus ModernBERT’s 2 trillion, with NorMuon selected for data efficiency.
NeoMME-Retriever applies ColPali-style page-image retrieval, avoiding OCR while preserving layouts, charts, tables, and typography. One pass produces a mean-pooled dense vector and 128-dimensional late-interaction vectors. On ViDoRe v3, the 260M model scored 0.523 nDCG@10, best below 800M parameters and 0.002 behind 3.75B-parameter ColQwen2.5; the 800M model scored 0.556, versus 0.565 for 850M-parameter Vultron Flash. Both reached the model-size Pareto frontier and beat ColModernVBERT, ColSmol-500M, and ColPali v1.3 on cited ViDoRe comparisons. At 2048×2048 on one NVIDIA L40S, NeoMME-260M encoded 51 pages per second versus ColModernVBERT’s 26. Pooling and quantization cut average late-interaction storage from 1.5 MB to 39 kB while retaining over 99% quality, or 6 kB while retaining over 95%. Apache 2.0 checkpoints, Hugging Face Transformers support, Sentence Transformers v6 checkpoints, and a visual RAG demo are available.
Positives
- NeoMME-260M scored 0.523 nDCG@10 on ViDoRe v3, the highest result among evaluated models below 800M parameters.
- One forward pass returns compact dense embeddings and more granular 128-dimensional late-interaction embeddings.
- 51 pages per second at 2048×2048 nearly doubles ColModernVBERT’s 26-page throughput on one NVIDIA L40S.
- 6 kB per page delivers 255 times smaller late-interaction storage while preserving more than 95% of baseline retrieval quality.
- Apache 2.0 checkpoints and day-zero Hugging Face Transformers support make both model sizes openly reusable.
Risks & concerns
- 524 billion pretraining tokens are far below ModernBERT’s 2 trillion-token budget, potentially limiting text capability despite the NorMuon optimizer.
- A 2048×2048 page initially produces 4,200 vectors requiring about 2.1 MB in float32 before compression.
- NeoMME-800M’s 0.556 ViDoRe v3 score trails Vultron Flash’s 0.565, while the 260M model remains 0.002 behind ColQwen2.5.
- Aggressive 6 kB compression retains over 95% of baseline quality, meaning some retrieval accuracy is sacrificed.
- Sentence Transformers v6 supports only one retrieval head per checkpoint, requiring NeoMMEForRetrieval and a custom Trainer to train both jointly.