Researchers from Ai2Comms introduced TutorMoments on August 7, 2026, to evaluate whether language models know when to help students and when to hold back, according to reporting published on Hugging Face. The benchmark tests language models on decision points drawn from real math tutoring sessions with U.S. students in grades 2 through 7.
The preview release includes TutorMoments-Preview, a dataset of 462 de-identified, text-only transcripts collected from a high-dosage math tutoring program where most students attend Title I schools. Parents and guardians agreed to share the data under a research clause. Anonymization occurred in two stages, first by the provider and subsequently through a math-aware data pipeline.
Twenty-seven U.S. math teachers provided more than 1,500 key-moment annotations and several thousand free-text comments across the transcripts. Each key moment represents a decision point where a tutor must choose between scaffolding to make a problem accessible or pushing for intellectual rigor. When teacher annotators disagreed on a decision point, the system assigned a majority label to establish the ground truth.
Testing language model tutors
To execute an evaluation, TutorMoments pauses a transcript at an annotated key moment and hands control to a test language model for five turns. A second language model acts as a simulated student during these replay sessions. An automated classifier, validated against teacher annotations, then rates whether the model's actions lined up with what the moment called for.
Testing across seven language models showed that default prompts cause models to over-help by providing excessive support and rarely encouraging deeper thinking. Supplying an evaluation-aware prompt that explicitly defined scaffolding, over-scaffolding, and pushing for rigor improved scores across all tested models. Despite these prompt improvements, language models still exhibited wide variations in decision-making and relied on a narrower set of teaching strategies than human educators.
Human benchmarks and project limits
Human tutors in the transcript dataset scored 0.458 on appropriate scaffolding, 0.182 on appropriate rigor, and 0.496 on avoiding over-scaffolding. Researchers emphasized that human scores reflect a naturalistic baseline rather than an optimal ceiling, as teacher annotators deliberately targeted transcript moments where tutoring could have been improved.
The scoring pipeline identified rigor pushes less reliably than scaffolding actions, reflecting a distribution of 260 rigor moments compared to 738 scaffolding moments in the annotation set. The current framework evaluates model behavior during short replay turns rather than tracking long-term student learning outcomes in actual classrooms.
Ai2Comms released the dataset of de-identified transcripts, the replay pipeline source code, and model replay files for public reproducibility. The development of TutorMoments received financial support in part from the Gates Foundation and Learning Commons.
