Researchers Tony Wu and Aurélien Lac introduced NeoMME, a family of 260-million and 800-million parameter multilingual multimodal encoders, according to a technical report published on Hugging Face. The system runs both text tokens and raw image patches through a single bidirectional Transformer rather than relying on a separate pretrained vision tower or an autoregressive language model decoder.
Hugging Face published checkpoints for the models under an Apache 2.0 license, making them available in the Hugging Face Transformers library. Both variants were trained from scratch using a masked discrete-diffusion objective on approximately 524 billion packed input tokens, which included 290 billion tokens from text-only examples. The developers used the NorMuon optimizer to handle data efficiency across multilingual text, code, mathematics, natural images, and document screenshots.
Architecture and Training
NeoMME splits images into non-overlapping 32×32 patches projected by a small multi-layer perceptron, while text inputs use factorized token embeddings. Both enter the same Transformer backbone. The architecture supports dynamic image resolutions to preserve aspect ratios and provides a bidirectional context window of 16,384 tokens, which accommodates up to two standard 3840×2160 4K UHD images. Its layers incorporate grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. Global attention runs on every sixth layer and the final layer, while intermediate layers use symmetric sliding-window attention. Text tokenization relies on a 131,000-token byte-pair encoding vocabulary trained from scratch.
During masked discrete-diffusion training, text-only inputs receive a corruption rate sampled uniformly between 0 and 1. Multimodal inputs apply masking rates between 0.3 and 1 to eligible text tokens while leaving image patches visible. High masking forces the model to recover descriptions directly from visual features.
Document Retrieval Benchmarks
Fine-tuned as NeoMME-Retriever using ColPali's page-image approach, the model encodes document page screenshots directly to preserve layouts, tables, and typography without optical character recognition preprocessing. The retriever produces both dense embeddings through mean pooling and 128-dimensional late-interaction token vectors in a single forward pass.
On the ViDoRe v3 benchmark, NeoMME-Retriever-260M scored 0.523 in nDCG@10, trailing ColQwen2.5 by 0.002 points while using roughly 14 times fewer parameters. The 800M model reached 0.556 nDCG@10, within 0.009 of the 850-million parameter Vultron Retriever Flash. On an NVIDIA L40S GPU processing matched 2048×2048 image inputs, the 260M model encoded about 51 pages per second, compared with 26 pages per second for ColModernVBERT.
Testing on ViDoRe v3 demonstrated that hierarchical token pooling combined with asymmetric quantization lowered late-interaction storage from an average of roughly 1.5 megabytes per page to 6 kilobytes per page with binary document embeddings and a pooling factor of 8. That configuration retained over 95 percent of baseline nDCG@10 retrieval performance. Fine-tuning checkpoints are also compatible with Sentence Transformers v6 for independent head training, while dual-head training uses the native retrieval class.
