Transformers MICs Catch UMT5 Regression With Tiny CPU Models
Transformers MICs use tiny CPU fixtures to catch model regressions early after a UMT5 attention bug remained undetected in main for almost two months.
Summary
Published November 3, 2026, transformers-mic proposes Model Integration Contracts after Transformers enabled PyTorch scaled dot product attention for T5 on July 30, letting UMT5 decoders see future tokens during training without crashing. The defect remained in main until September 28. Replaying five months of commits with a 66 MB CPU UMT5 fixture, versus 1,227 MB for the checkpoint, caught every affected commit: label log probabilities shifted 1.22 and loss about 0.20. Reverting the fix in CI reproduced both changes; the T5 check took 13.4 seconds, excluding setup. Transformers spans more than 750 model types, 88,000 plus main CPU tests and 200,000 plus daily GPU tests, making full checkpoint testing on every pull request impractical.
A MIC pins a shrunken fixture, runnable Markdown usage path and human reviewed baselines. Compatible checkpoints can share an architecture fixture while retaining expected results; huggingface-mic publishes contracts, and a frozen Hub fleet records revisions and package versions so CI can select tests from changed files and reproduce runs. SmolLM checks deterministic token IDs, training loss, finite gradients and optimizer driven weight changes. Fixtures preserve layer kinds, preprocessing and original weight formats. Llama 4 Scout shrank from 108.6 billion to 44.7 million parameters, with vision layers cut from 34 to 2 and language layers from 48 to 4.
The tmic CLI uses an LLM through Inference Providers to draft contracts, but humans approve baselines. Random Whisper fixtures caught 18 of 24 injected faults; overfit fixtures caught all 24, yet both missed an assisted decoding bug exposed by the real checkpoint. As of November 1, MIC failures report without blocking pull requests. Coverage remains partial, with unresolved catch rates, false alarms, review costs and real weight gaps; existing tests remain.
Positives
- The 66 MB UMT5 fixture caught every affected commit from July 30 through the September 28 fix while running cheaply on CPU.
- The reverted T5 regression produced the expected failure in CI within 13.4 seconds, excluding setup.
- Llama 4 Scout shrank from 108.6 billion to 44.7 million parameters while retaining checked layer types and MoE routing.
- Overfit Whisper fixtures caught all 24 injected faults, compared with 18 for randomly initialized fixtures.
- Frozen fleet revisions preserve package versions and baselines, allowing later reproduction of each CI run.
Risks & concerns
- The UMT5 decoder could inspect future tokens for almost two months without crashes exposing the training defect.
- Full checkpoint testing cannot practically cover every pull request across more than 750 model types and 200,000 plus daily GPU tests.
- Both random and overfit Whisper fixtures missed an assisted decoding bug detected by the real checkpoint.
- As of November 1, MIC failures remain nonblocking while false alarm rates, review costs and real pull request detection are unmeasured.
- Partial model family and API coverage means regressions can still escape fixtures and require existing integration tests.