testing-vlsi-verify-agentic

Imported from ProgMastermind/vlsi-verify-agentic (.agents/skills/testing-vlsi-verify-agentic/SKILL.md). Install upstream with npx skills add ProgMastermind/vlsi-verify-agentic --skill testing-vlsi-verify-agentic. Copyright stays with the author.

Devin Secrets Needed

• None for deterministic local runtime testing.
• OPENAI_API_KEY is required for OpenAI-backed demo configs/scripts such as examples/devin_pipeline_demo/run_demo_llm.sh and examples/uart_lite/orchestrator_config.llm.json, which make real OpenAI-compatible calls and log token usage.
• GITHUB_PERSONAL_ACCESS_TOKEN is only needed if posting PR comments or creating/pushing GitHub branches.

Setup

From the repo root, install dependencies with:

python -m pip install -e ".[dev]"

For real HDL evidence, ensure these tools are available:

iverilog -V
vvp -V
verilator --version

If Verilator or Icarus is unavailable, Agent 6 may exercise a simulator-unavailable or partial fallback path. For tests that require real HDL execution, install the simulator first and make that explicit in the report.

Primary Deterministic End-to-End Test

Run the APB example with a fresh run id:

rm -rf workspace/e2e-cli-test
python run.py --spec examples/apb_slave/spec.md --dut examples/apb_slave/rtl/apb_slave.sv --run-id e2e-cli-test

Expected stdout:

Run e2e-cli-test ended in TERMINATE; workspace=workspace/e2e-cli-test

Verify these artifact assertions:

• workspace/e2e-cli-test/run_state.json has phase == "TERMINATE", iteration_count == 1, expected coverage history for that config, human_review_queue == [], and feature_attempts.F5 == 2.
• workspace/e2e-cli-test/reports/closure_decisions.json has iteration == 1, diagnoses == [], halt_recommended == true, and halt_reason == "closure".
• workspace/e2e-cli-test/tb/apb_slave_seq_lib.sv contains class apb_slave_F5_directed_seq, addr == 12'hfff, and evidence cg_apb.cp_addr.unmapped.
• workspace/e2e-cli-test/reports/signoff_package.md contains closed vPlan rows, final coverage, Iterations: 1, and Human review queue items: 0.

Live Demo with Functional Coverage Extraction

This path uses Agent 6's Icarus-based functional coverage harness that emits FCOV_BIN lines and parses them into structured JSON coverage reports.

rm -rf workspace/fcov-test
./examples/devin_pipeline_demo/run_demo.sh fcov-test

Expected stdout:

Run fcov-test ended in TERMINATE; workspace=workspace/fcov-test

Verify these artifact assertions:

• run_state.json has phase == "TERMINATE", iteration_count == 1, coverage_history == [83.33, 91.67], human_review_queue == [], feature_attempts.F5 == 2.
• sim/iteration_0_functional_coverage.log contains FCOV_BIN|cg_apb.cp_addr.unmapped|0|1|F5 (hole present).
• sim/iteration_1_functional_coverage.log contains FCOV_BIN|cg_apb.cp_addr.unmapped|1|1|F5 (hole closed).
• sim/iteration_0_functional_coverage.json has coverage_source == "simulator_functional_coverage_stdout", simulator == "icarus", hit_bins == 10, total_bins == 12, coverage_overall_pct == 83.33, and coverage_holes[0].name == "cg_apb.cp_addr.unmapped".
• sim/iteration_1_functional_coverage.json has coverage_overall_pct == 91.67 and coverage_holes == [].
• reports/artifact_audit.json has all_required_present == true and coverage_source == "simulator_functional_coverage_stdout".
• reports/signoff_package.md contains Coverage: 91.67%.
• sim/iteration_0_rtl_smoke.log and sim/iteration_1_rtl_smoke.log contain COMPILE_PASS and RTL_SMOKE_PASS module=devin_pipeline_controller.
• sim/iteration_0_verilator_lint.log contains VERILATOR_LINT_PASS.
• Agent sequence in reports/agent_trace.jsonl: 1,2,3,4b,4e,4d,4c,4a,5,6,7,6,7.

UART Lite Official Demo Test

Use this when validating the official VLSI-named uart_lite workflow or Agent 6's multi-DUT functional-coverage dispatch. This path is shell-only; do not record an idle desktop.

rm -rf workspace/e2e-uart-lite-test
python run.py \
  --config examples/uart_lite/orchestrator_config.json \
  --spec examples/uart_lite/spec.md \
  --dut examples/uart_lite/rtl/uart_lite.v \
  --run-id e2e-uart-lite-test

Expected stdout:

Run e2e-uart-lite-test ended in TERMINATE; workspace=workspace/e2e-uart-lite-test

Verify these artifact assertions:

• run_state.json has phase == "TERMINATE", iteration_count == 0, coverage_history == [92.31], and human_review_queue == [].
• sim/iteration_0_rtl_smoke.log contains COMPILE_PASS, RTL_SMOKE_PASS module=uart_lite, and no RTL_SMOKE_FAIL.
• sim/iteration_0_functional_coverage.log contains COMPILE_PASS, FCOV_DONE, no FCOV_FAIL, and these FCOV_BIN lines:
  • FCOV_BIN|cg_uart.cp_reset.defaults|1|1|F23
  • FCOV_BIN|cg_uart.cp_ctrl.rw|1|1|F1
  • FCOV_BIN|cg_uart.cp_cfg.legal_write|1|1|F6
  • FCOV_BIN|cg_uart.cp_cfg.reserved_parity|1|1|F4
  • FCOV_BIN|cg_uart.cp_cfg.reserved_stop|1|1|F5
  • FCOV_BIN|cg_uart.cp_cfg.baud_clamp|1|1|F6
  • FCOV_BIN|cg_uart.cp_tx.push|1|1|F2
  • FCOV_BIN|cg_uart.cp_tx.full_pslverr|1|1|F21
  • FCOV_BIN|cg_uart.cp_rx.empty_pslverr|1|1|F22
  • FCOV_BIN|cg_uart.cp_irq.en_rw|1|1|F16
  • FCOV_BIN|cg_uart.cp_irq.w1c|1|1|F17
  • FCOV_BIN|cg_uart.cp_ctrl.loopback|1|1|F15
  • FCOV_BIN|cg_uart.cp_addr.unmapped_pslverr|0|1|F20
• sim/iteration_0_functional_coverage.json has coverage_source == "simulator_functional_coverage_stdout", simulator == "icarus", total_bins == 13, hit_bins == 12, coverage_overall_pct == 92.31, and exactly one required hole: cg_uart.cp_addr.unmapped_pslverr mapped to F20.
• spec_ir.json has ip_name == "uart_lite".
• tb/ contains the expected uart_lite_* artifacts: interface, transaction, driver, monitor, sequencer, agent, coverage, assertions, scoreboard, check tasks, RAL, sequence library, and package.
• tb/ contains no devin_pipeline_controller* files.
• reports/agent_trace.jsonl first 11 agent_end events are exactly 1,2,3,4b,4e,4d,4c,4a,5,6,7.
• sim/iteration_0_verilator_lint.log contains VERILATOR_LINT_PASS or an explicit simulator-unavailable skip. If Verilator was expected to be installed, treat a skip as an environment issue and report it.

OpenAI-Backed Live Demo Test

Use this when validating the live demo path that should prove real OpenAI calls and artifact audit logging:

rm -rf workspace/llm-audit-test
./examples/devin_pipeline_demo/run_demo_llm.sh llm-audit-test

Expected stdout includes:

Run llm-audit-test ended in TERMINATE; workspace=workspace/llm-audit-test
LLM calls:           workspace/llm-audit-test/reports/llm_calls.jsonl
Artifact audit:      workspace/llm-audit-test/reports/artifact_audit.json

Verify these artifact assertions:

• reports/llm_calls.jsonl has 13 OpenAI-compatible calls in order: agent1, agent2, agent3, agent4b, agent4e, agent4d, agent4c, agent4a, agent5, agent6, agent7, agent6, agent7.
• Every LLM record has positive latency_s, response_chars, and total_tokens; summed total_tokens should be greater than 1000 for a real workload.
• reports/agent_llm/ has 13 JSON reports with agent_id, owned_artifact, inputs_checked, actions_taken, edge_cases_considered, and artifact_quality_gate.
• run_state.json has phase == "TERMINATE", iteration_count == 1, coverage_history == [83.33, 91.67], human_review_queue == [], and feature_attempts.F5 == 2.
• reports/agent_trace.jsonl has 13 agent_end events in order 1,2,3,4b,4e,4d,4c,4a,5,6,7,6,7 and 13 agent_llm_audit events.
• sim/iteration_0_verilator_lint.log and sim/iteration_1_verilator_lint.log contain VERILATOR_LINT_PASS; if Verilator is unexpectedly absent, report that instead of accepting a skip.
• sim/iteration_0_rtl_smoke.log and sim/iteration_1_rtl_smoke.log contain COMPILE_PASS and RTL_SMOKE_PASS module=devin_pipeline_controller, and no skip/fail markers.
• Functional coverage files use coverage_source == "simulator_functional_coverage_stdout"; iteration 0 has hit_bins == 10, coverage_overall_pct == 83.33, hole cg_apb.cp_addr.unmapped; iteration 1 has hit_bins == 11, coverage_overall_pct == 91.67, no holes.
• reports/artifact_audit.json has all_required_present == true, coverage_source == "simulator_functional_coverage_stdout", and includes required spec, vPlan, TB, simulator, trace, signoff, LLM, and functional coverage artifacts.
• reports/signoff_package.md contains Coverage: 91.67%, LLM calls logged: 13, Iterations: 1, and Human review queue items: 0.

Regression Checks

python -m pytest
python -m ruff check .

Expected results: pytest reports all tests passing and ruff prints All checks passed!.

Key Distinguishing Assertions

To prove the new functional coverage path is real (not the old deterministic model):

• For examples/devin_pipeline_demo/run_demo.sh, coverage values must be 83.33 and 91.67 (from 10/12 and 11/12 bins), NOT the old 89.0 and 91.5 (from 178/200 and 183/200 bins).
• For examples/uart_lite/, coverage must be 92.31 from 12/13 cg_uart.* bins, with source simulator_functional_coverage_stdout and simulator icarus.
• coverage_source must be "simulator_functional_coverage_stdout", NOT "deterministic_demo_coverage_model_gated_by_real_rtl_smoke".
• sim/iteration_*_functional_coverage.log must contain FCOV_BIN|... lines (protocol emitted by the Icarus harness), NOT just RTL smoke logs.
• Old sim/iteration_*_coverage_model.json files should NOT exist; they are replaced by sim/iteration_*_functional_coverage.json.

UVM Limitations

Verilator 4.038 and Icarus 11.0 cannot compile SystemVerilog covergroup or uvm_pkg. Agent 4c still emits UVM-style coverage source as a generated artifact, but the executable coverage harness uses plain Verilog $display("FCOV_BIN|...") protocol parsed by Agent 6. Full UVM functional coverage extraction requires a UVM-capable simulator (e.g., Synopsys VCS, Cadence Xcelium, or Siemens Questa). Document this limitation if it comes up during testing.

Reporting Notes

This is a CLI/library repo, so do not record an idle browser. Provide command output and artifact assertions as text evidence instead. If testing a real simulator path, clearly state which simulator binary was installed and used. Always distinguish real simulator evidence from deterministic demo coverage provenance.