Hugging Face published technical details on July 6, 2026, outlining the data strategy used to pre-train its 7-billion-parameter PRX vision model, detailing a pipeline built on public and internal datasets, vision-language re-captioning, and dual data storage formats.
Hugging Face uses Lance as a columnar format for dataset curation and Mosaic Data Shards (MDS) to stream data during distributed training. Hugging Face noted that while MDS provides deterministic shuffling and elastic checkpoint resumes, its rigid structure makes dataset editing difficult. Lance handles feature engineering, allowing predicate pushdown and vector search across hundreds of millions of rows.
Architecture and Storage Choices
Hugging Face switched its text encoder to Qwen3-VL for the PRX run, computing text latents on the fly inside the training loop instead of pre-computing them. Hugging Face reported that this added a 3% to 4% throughput cost, equivalent to roughly one extra day on a 30-day training run. In exchange, smaller MDS shard sizes allowed the entire pre-training dataset to fit directly on the SSD-backed shared filesystem of its SLURM cluster.
The team also converted training images to JPEG at quality 92 rather than lossless PNG formats. Testing across resolution tiers showed that JPEG re-encoding introduced minimal perceptual loss, while a comparison between two identical 1024-pixel PRX models trained on PNG and JPEG inputs yielded practically indistinguishable outputs.
Captioning, Bucketing, and Filtering
Hugging Face re-captioned all pre-training images to create standardized, detailed descriptions. Using a Ray Data pipeline, Hugging Face selected Qwen3-VL-8B to generate single paragraphs of 100 to 200 words. Benchmarks against candidate captioners showed Qwen3-VL-8B processed 20 images per second per Nvidia H200 GPU while delivering strong image generation scores on downstream metrics.
Image processing relies on resolution tiers starting at 512 pixels and reaching 4,096 pixels, divided into 13 aspect-ratio buckets per tier. Images requiring upscaling beyond 33% are discarded. To perform content filtering and deduplication, Hugging Face ran Qwen3-8B in a text-only pass over captions at 200 captions per second per GPU to classify samples as visual, text, or non-safe-for-work. Instead of rewriting dataset shards, the loader uses per-shard skip lists to bypass flagged items and perceptual-hash duplicates at load time.
