Anthropic’s new plugin evals workflow for Claude Code provides a mechanism to assess plugin behavior. The claude plugin eval command runs plugins against realistic prompts, comparing the output with a run without the plugin loaded. This allows developers to determine if the plugin triggers correctly, survives edits, and outperforms a baseline model. The workflow assesses three key questions: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model.
The evaluation process utilizes a suite of six grader types. Four graders are free and compute results from transcripts and disk files: regex, tool_used, tool_order, and file_exists. Two graders require calling a judge model, incurring billing costs: llm and baseline. The llm grader assesses the reply against prose criteria, while the baseline grader compares the response against a reference answer. Each case runs twice – with and without the plugin – and the difference (Δ) indicates the plugin’s contribution.
During a typical run, a case might involve six runs, costing an estimated $0.41 and taking 74 seconds. The results are stored in an evals/results/<timestamp>/report.html file, detailing per-grader verdicts and judge votes. The workflow is initiated with claude plugin eval init, which proposes cases and graders, and then attempts to run them. The CI invocation uses --trust-plugin to trust the plugin and --bare <name> to create a blank template.
Key findings include a Δ near zero with the tool_used: Skill grader failing, which indicates the skill doesn’t trigger on natural phrasing. This highlights a limitation of the claude plugin validate tool, which focuses on manifest syntax rather than behavioral analysis. The workflow offers controls via --threshold, --max-cost-usd, and --trust-plugin for CI gate usage, mitigating potential usage-limit errors.
