Hugging Face and SkyPilot have launched a joint storage integration allowing compute jobs on any cloud to read models and datasets directly from Hugging Face Storage without egress fees. Hugging Face announced the backend integration on July 7, 2026, introducing store: hf as a first-class storage option within SkyPilot task configurations.
SkyPilot schedules computing tasks across more than 20 cloud providers, Kubernetes clusters, Slurm environments, and on-premises hardware. Under the new system, jobs access Hugging Face Buckets or repositories using an hf:// URL scheme alongside an existing account token passed via environment variables.
Mounting relies on Hugging Face's hf-mount Filesystem in Userspace (FUSE) backend, displaying remote storage as a local path alongside existing SkyPilot mounts like gcsfuse and blobfuse2. Hugging Face and SkyPilot configured the system to perform lazy reads, downloading only requested byte ranges during execution to allow processing units to begin work immediately. Tasks can also select a copy mode that downloads full files ahead of execution using the huggingface_hub client.
Cost and Performance Benchmarks
Hugging Face charges no egress or content delivery network fees to read stored data, setting storage costs at $12 to $18 per terabyte per month compared to standard AWS S3 rates of roughly $23 per terabyte plus egress fees. Writes back to Hugging Face Buckets remain subject to standard outbound network egress rates charged by the compute host provider.
In joint benchmark tests using a fine-tuning run of Qwen/Qwen3.5-4B on the Multilingual-Thinking dataset, model loading completed in about 30 seconds at speeds up to 500 megabytes per second across AWS, Google Cloud, and Lambda. Checkpoints measuring 8.43 gigabytes streamed to Hugging Face Buckets at write speeds of approximately 168 megabytes per second on AWS, 123 megabytes per second on GCP, and 112 megabytes per second on Lambda.
Deduplication and Open Source History
Storage buckets run on the Xet engine, which divides files into approximately 64-kilobyte chunks and deduplicates overlapping data. Re-uploading an unchanged 8.43-gigabyte checkpoint file during testing took eight seconds to transfer chunk hashes, compared to 24 seconds for the initial upload.
Developer Nikhil Jha contributed the initial codebase for the integration. Engineers at Hugging Face subsequently upstreamed FUSE fixes to enable mounts inside unprivileged containers, while SkyPilot integrated the component into its core storage backend.
