ServiceNow CoreAI released AutoSynthData, an automated pipeline that generates synthetic training tasks for enterprise AI agents by analyzing where models fail. Hugging Face published the technical report detailing the system on Friday.
The system benchmarks a target model against a stronger teacher model inside an agent environment to locate capability gaps. Rather than reusing evaluation prompts directly, AutoSynthData distills observed failures into capability specification cards that define tools, workflows, and success criteria. A generator then creates new tasks with altered entities, states, and prompt phrasing.
AutoSynthData builds datasets across two stages. An initial target phase generates and vets core tasks, while a multiply phase produces variants with fresh user requests and configurations. To prevent drift, a multiplied sample cannot seed another multiplied sample. Each candidate task must pass positive verification, confirming the teacher's reference trajectory succeeds, and negative verification, which alters expected outcomes to ensure flawed solutions fail. Candidates that fail these checks pass to an automated critic for bounded repair attempts.
Testing on EnterpriseOps Gym
Researchers tested the framework on the EnterpriseOps Gym benchmark across two enterprise domains. In the Hybrid domain, the team used Gemma-4-26B-A4B-it as the target model and Qwen3.8-27B as the teacher, producing 2,000 synthetic training tasks in about 18 hours. Fine-tuning Gemma on this data raised its mean Pass@1 rate by 7.2 percentage points, a 35% relative gain, at epoch 5. Verifier success reached 68.55%, up from 63.01%, closing 59% of the original gap between Gemma and the reference model.
A second trial in the ITSM domain paired Gemma-4-26B-A4B-it with DeepSeek-V4.1-Flash as the teacher model. AutoSynthData generated 1,994 synthetic training samples across 66 hours. Generation took longer because the test used a larger teacher model and preceded pipeline throughput optimizations. Fine-tuning on these samples increased Gemma's mean Pass@1 rate from 18.77% to 27.18%.
