The Allen Institute for AI deployed a budget-based cluster scheduler to distribute computing time across thousands of Nvidia H100, B200, and B300 GPUs, Hugging Face reported. The change replaces a priority queue setup that led to resource squatting and priority inflation across clusters ranging from 88 to 1,024 chips.
Ai2 provides capacity to roughly 150 internal researchers whose workload demand outstrips available supply by two to three times. Under the institute's previous system, researchers marked workloads with high priority and opted out of preemption. That setup led to researchers parking idle placeholder jobs to guarantee access for debugging, while on-call engineers spent shift hours manually negotiating server shutdowns on malfunctioning hardware.
Budgets and contracts
The institute reorganized scheduling around GPU time budgets set by managers, pairing the system with a hierarchical fair-share scheduler tracked across a seven-day lookback window. Research leads and program managers distribute proportional time shares rather than assigning dedicated chips. Jobs drawn from an allocated budget receive preemption protection for a declared minimum runtime, capped at eight hours. Jobs submitted without a budget claim run for free under unallocated occupancy but can be preempted at any moment.
Automatic draining on unhealthy hosts cut hardware repairs requiring a human-in-the-loop by 74%. In a 30-day testing window following an incremental rollout that began in late July, teams received 98% of their owed GPU hours, while cluster occupancy remained steady at 98%. Unallocated workloads accounted for 18% of delivered GPU time, maintaining machine occupancy when funded projects were not ready to run.
Queue latency shifts
Wait times for small debug jobs dropped sharply under the new policy. Workloads requiring a small number of chips and 15 minutes or less of runtime saw 90th-percentile queue times fall from two hours to 30 seconds. On the institute's largest H100 cluster, median queue latency fell from five minutes to 24 seconds, while 90th-percentile wait times dropped from 2.8 hours to 1.8 hours.
The policy created new friction for interactive development sessions, which previously stayed active for up to a week. Because protection now caps at eight hours, interactive sessions become preemptible after that window, wiping out unpreserved analysis state. To address the issue, Ai2 is building a dedicated CPU-only cluster next to on-premises storage and developing restorable session tooling, while monitoring whether runtime guarantees cause capacity fragmentation for its largest distributed training runs.
