A research paper presents a systematic black-box framework for evaluating agentic systems. The framework focuses on identifying vulnerabilities in systems that read untrusted inputs and call tools with real permissions. The approach utilizes a seven-domain taxonomy mapping observable behaviors to risk categories. Automated red teaming generates 120 adversarial scenarios per domain, and human-validated evaluation is conducted using LLM judges.
Empirical validation was performed across two agent architectures, CrewAI and AutoGen, using four base models. The results indicated an average governance risk of 56.25%, 65% privacy risk in multi-agent configurations, and agent behavior vulnerabilities reaching 85%. These findings demonstrate the potential for significant risk in agentic AI deployments.
This black-box approach allows for the identification of critical architectural vulnerabilities without requiring privileged access to the system. The framework offers a scalable path toward safer agent deployments, particularly important given the increasing use of agentic systems in production environments.
This research addresses the growing concern of security risks associated with agentic AI systems. The framework provides a practical tool for engineers to proactively assess and mitigate these risks.
Source: https://arxiv.org/abs/2609.09647