Hugging Face published an evaluation of ALTK-Evolve showing that agentic memory performance depends on calibrating the volume of past guidance to a model's underlying capability rather than maximizing injected context.
The evaluation tested eight language models on AppWorld, a benchmark comprising 585 multi-step tasks across nine simulated applications. Performance was scored using Task Goal Completion (TGC) for individual task success and Scenario Goal Completion (SGC) for passing every variation of a scenario.
Model tiers and memory dosage
Stronger models with headroom benefited most from complete guideline sets. DeepSeek-V3.2, a 671-billion parameter mixture-of-experts model, gained 9.5 percentage points in TGC and 16.1 percentage points in SGC when supplied with its full self-mined guideline set.
Smaller architectures struggled under large context loads. The 117-billion parameter gpt-oss-120b model improved by 16.1 percentage points on both TGC and SGC using curated retrieval—a fixed core set combined with task-relevant guidelines—whereas injecting all guidelines delivered lower accuracy gains and inflated token counts.
Frontier systems also recorded reliability gains on scenario-level tests. Claude Opus 4.6 improved SGC by 7.1 percentage points to reach 94.6 percent, while GPT-5.5 gained 7.2 percentage points on SGC to reach 89.3 percent.
Saturated models showed no measurable improvement. GLM-5, a 745-billion parameter model, recorded zero gain over its baseline scores of 87.5 percent TGC and 80.4 percent SGC.
Inference costs and retrieval
Token measurements showed that curated retrieval minimized overhead for smaller models. Running gpt-oss-120b with selective memory increased average task tokens by 5 percent, from 110,000 to 116,000, compared to a 51 percent rise to 166,000 tokens under full injection.
DeepSeek-V3.2 consumed 263,000 tokens per task with full memory against a 148,000-token baseline, representing a 78 percent increase. The model maintained an average of roughly 18 to 19 ReAct steps across both runs, indicating that token growth stemmed from prompt input size rather than longer step counts.
Future research directions include replacing cosine-similarity guideline retrieval with an outcome-trained selector and exploring teacher-distilled memory for models lacking sufficient self-distillation signal. Researchers also plan controlled tests to separate context-window capacity from model capability and evaluate ALTK-Evolve beyond AppWorld.
