ALTK-Evolve added consistency guidelines and a diagnostic tool named the Consistency Analyzer to target repeat failure rates in autonomous software agents, Hugging Face reported on September 15, 2026. Author Ashwath Vaithinathan Aravindan outlined the release, which addresses systems that complete a task during initial testing but fail when executing the same workflow again.
Standard evaluations typically track Mean@k, an average pass rate across k trials. In AppWorld test_normal benchmarks across 168 tasks, a ReAct agent using GPT-4.1 at temperature 0.0 recorded a 77.4 percent average success rate across five repetitions. That same configuration succeeded across all five runs on 53.0 percent of tasks. The resulting difference between the average score and the all-run success rate, known as the consistency gap, measured 24.4 percentage points, expanding to 30 points on hard tasks.
Measurement of all-run reliability relies on Pass^k, which demands success on every trial. This metric differs from Pass@k, an optimistic measure common in code generation that checks whether at least one attempt out of k succeeds. Hugging Face noted that Pass^k is always less than or equal to Mean@k, which in turn is less than or equal to Pass@k.
Distribution shifts
Flat token probability distributions cause decision instability across runs. While sharp distributions place dominant probability mass on a single token, flat distributions divide probability across competing tokens. Platform-level factors like GPU floating-point non-associativity and request batching can reorder these near-tied probabilities. Because multi-step agents chain dozens of actions, minor variations compound into divergent paths even under greedy decoding with fixed seeds.
Diagnostic pipeline
The Consistency Analyzer replays a single recorded trajectory to locate vulnerable decision steps. Operating as a black-box tool without ground-truth answers or model logits, it issues one model call per step requesting five completions against existing context. Each step receives a score in a diagnostic scorecard. From those flagged steps, the system generates targeted guidelines, such as applying line-anchored regex patterns rather than substring counts when reading checkbox markers, or verifying search queries before executing note tasks.
Testing the generated guidelines on AppWorld test_normal reduced the consistency gap from 24.4 percentage points to 12.0 percentage points. Five-run aggregate consistency climbed from 53.0 percent to 69.0 percent, while average accuracy rose from 77.4 percent to 81.0 percent. Medium tasks gained 22.9 percentage points, hard tasks rose 14.3 percentage points, and easy tasks improved by 12.2 percentage points.
The guidelines also transferred to distinct task variants within the same AppWorld scenario, lifting five-run consistency by 13.0 percentage points. On the weaker gpt-oss-120b model, identical-task five-run consistency increased from 10.1 percent to 16.1 percent, while similar-task consistency rose by 8.7 percentage points. Hugging Face released the analyzer and guideline generation code through the ALTK-Evolve open-source repository on GitHub alongside a technical report on arXiv.
