HybridDeepResearch is a new benchmark designed to evaluate autonomous agents capable of integrating information from both the open web and relational databases. The benchmark comprises 380 tool-dependent tasks, sourced from LiveSQLBench-Base-Lite databases and public web corpora. These tasks are categorized into three reasoning patterns: SQL2S, S2SQL, and Parallel. Evaluations were conducted across proprietary and open-weight models, including GLM-5.2, Claude-Sonnet-4.6, and GPT-5, utilizing various agentic scaffolds. Results indicate that these models achieve approximately 50-54% Pass@8 on the benchmark’s hard subset.
The research highlights a significant challenge: preserving constraints when transitioning between structured and unstructured data sources. Directional reasoning – specifically, deriving conclusions based on a single data source – proved substantially more difficult than parallel intersection tasks. This suggests that current agentic systems struggle to maintain accuracy when integrating information without a clear, consistent framework. The benchmark’s design emphasizes the 'handoff' between systems, a critical aspect of real-world problem-solving.
Code and datasets for HybridDeepResearch are publicly available on GitHub (https://github.com/Snowflake-AI-Research/HybridDeepResearch) and Hugging Face (https://huggingface.co/datasets/Snowflake/HybridDeepResearch). This open access facilitates further research and development in agentic systems capable of handling complex analytical tasks across diverse data modalities.
Source: https://arxiv.org/abs/2609.09410