Introducing WikiBench
LangChain has developed WikiBench, a new benchmark designed to measure the effectiveness of OpenWiki, an open-source agent for generating and maintaining codebase documentation. The core goal of WikiBench is to determine if a wiki genuinely assists coding agents and to track improvements made to OpenWiki itself. The benchmark utilizes a structured approach to evaluation, focusing on the quality of the generated documentation.
The Evaluation Process
The WikiBench framework employs a ‘verifier’ agent, which utilizes a ‘reader agent’ to answer questions about the codebase. These questions are automatically generated based on thematic areas within the repository, categorized as ‘coverage’ and ‘retrieval’ questions. Coverage questions assess the agent’s ability to understand the necessary context for changes, while retrieval questions probe the agent’s understanding of specific behaviors and interactions within the code. The entire process leverages a Harbor framework for long-running tasks.
Scoring and Model Performance
To evaluate the generated answers, a series of LLM judges are used to assess the presence and grounding of facts. The overall score for a question is determined by the percentage of facts correctly answered, with a score of 0.6 achieved if only three out of five expected facts are present. Experiments compared OpenWiki 0.3.0 with earlier versions and three harness: Bare DeepAgents, OpenWiki 0.2.5, and OpenWiki 0.3.0. Luna was used as the base model across all runs.
Key Findings and Cost Analysis
OpenWiki 0.3.0 demonstrated substantial improvements, particularly in ‘coverage’ questions. When evaluated with models like DeepSeek Flash ($0.44) and GLM 5.2 ($9.18), DeepSeek Flash showed a significant cost advantage – roughly one-sixth the cost of GLM 5.2. Analysis revealed that models like DeepSeek Flash and GLM 5.2 read roughly three times as many files as Luna, but produced only 1.2–1.5 times as many pages, indicating a focus on understanding the codebase rather than generating extensive output. The benchmark highlighted the efficiency gains achieved when combining the wiki with the source code, resulting in higher accuracy and lower costs compared to using the source alone.
