OpenDiscoveryTrace is a new dataset designed to address the limitations of current benchmarks for autonomous AI scientists. Existing evaluations focus solely on final outputs, neglecting the underlying reasoning process. This makes it difficult to understand how models arrive at conclusions and identify potential failure modes. The dataset contains 558 complete trajectories from seven models: three frontier models (GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro) and four open-weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B). Each trajectory records a structured trace with nine fields per step, including thoughts, tool calls, and confidence levels.
Pilot analysis of 363 trajectories judged by LLMs revealed substantial differences in performance between the models. Claude Opus 4.6 produced significantly more errors (2.5 per trajectory) compared to GPT-5.4 (0.08 per trajectory), with qualitatively different error profiles. The dataset includes 60 live-retrieval variant trajectories. The dataset supports research on process-level evaluation, scientific agent auditing, and AI governance.
The dataset, trace schema, agent harness, and benchmark definitions are available under a CC BY 4.0 license. This allows for standardized evaluation and comparison of AI scientist workflows. The data provides a means to diagnose issues and understand the strengths and weaknesses of different models.
Source: https://arxiv.org/abs/2609.09203