_shared

Imported from posit-dev/vip (examples/_shared/AGENTS.md). Install upstream with npx skills add posit-dev/vip --skill _shared. Copyright stays with the author.

Writing a VIP test extension

You are being asked to write a VIP test extension. Here is the contract.

VIP (Verified Installation of Posit) is a pytest-bdd test suite that validates a deployment of Posit Connect, Workbench, and/or Package Manager. An extension is a directory of .feature/.py pairs that VIP loads alongside its own built-in tests. This document is a reference, not a tutorial -- read the scenario file(s) already in this directory for a worked example, and come back here when you need to know what a fixture returns or whether a marker exists.

How an extension is loaded

vip verify --config vip.toml --extensions /path/to/this/directory

or, in vip.toml:

[general]
extension_dirs = ["/path/to/this/directory"]

Either way, every .py file in the directory is collected by pytest and gets full access to VIP's fixtures and plugin behavior (auto-skip, min_version gating, JSON/JUnit/SARIF reporting).

Four-layer architecture

VIP (and every extension) is structured in four layers, each one only talking to the layer directly below it (see docs/test-architecture.md for the full guide):

1. Test -- .feature files (Gherkin scenarios). Business language only; no URLs, status codes, or selectors.
2. DSL -- step definitions (given/when/then functions) + fixtures, via pytest-bdd. Thin: push logic down to layer 3.
3. Driver Port -- client interfaces (src/vip/clients/) or Playwright page objects. This is what a step definition calls.
4. Driver Adapter -- httpx (API calls) or Playwright (browser automation). The actual wire/DOM traffic.

The auto-skip contract

Every scenario must carry a product marker: @pytest.mark.connect, @pytest.mark.workbench, or @pytest.mark.package_manager, applied directly on the @scenario function in the .py file. VIP's plugin (src/vip/plugin.py) reads these pytest markers during collection and deselects (not skips -- removes from the run entirely) any test whose product is not configured in vip.toml.

A .feature file can carry the matching Gherkin tag (@connect above a Scenario: block) too, and pytest-bdd does turn that into the same pytest marker -- but only for that one scenario. Tagging the whole Feature: block applies the tag to every scenario in the file, which causes wrong deselection when a file mixes products. Omitting the marker entirely is the single most common mistake: the scenario then runs unconditionally, against whatever client fixture happens to be available (or None), instead of being cleanly excluded when its product isn't configured.

Version gating

Use @pytest.mark.min_version(product="connect", version="2024.09.0") on a @scenario function to skip a scenario when the deployed product version is below the given one. product is one of connect, workbench, package_manager. When the deployed version can't be determined, the test is skipped and flagged N/A-by-version rather than run optimistically.

Public fixtures (from VIP core)

All fixtures below are defined in src/vip/fixtures.py, which VIP registers as a pytest plugin, so they are available to your extension wherever it lives on disk -- do not redefine them in your own conftest.py, or you will silently shadow the config-driven values.

One gap to know about: the Connect content-cleanup fixtures stay scoped to VIP's own test package, so an extension that creates Connect content is responsible for removing it (connect_client.cleanup_content(guids)).

connect_client, workbench_client, pm_client, and kubernetes_client can all be None. Guard against that (skip or assert) before using one, the same way VIP's own tests do.

workbench/test_session_capacity_k8s.py does not consume the kubernetes_client fixture. Because that fixture folds "Kubernetes is not configured" and "client construction failed" into the same None, and the test needs to report those two outcomes differently (not_applicable vs. unproven), it builds its own KubernetesClient instead. Follow that pattern if your own test needs the same distinction.

Shared Gherkin steps

VIP registers these Given steps globally, so a .feature file in your extension can use them without you writing a step definition. Each one skips the scenario when that product is absent from vip.toml, which is a useful complement to the product marker: the marker decides whether the scenario is collected at all, the step decides whether it runs.

Every other step in your .feature files is yours to define.

Registered markers

Registered in src/vip/plugin.py (pytest_configure):

Client entry points (src/vip/clients/)

These are the classes behind the *_client fixtures above -- reach for them through the fixtures rather than constructing your own.

Rules

• Use pytest-bdd: @scenario, @given, @when, @then.
• Mirror scenario names exactly between the .feature file and the @scenario(...) call in the .py file -- pytest-bdd matches on the literal string.
• Apply the product marker (@pytest.mark.connect, etc.) directly on every @scenario function, not only as a Gherkin tag.
• Prefer VIP's client fixtures (connect_client, workbench_client, pm_client) over calling httpx yourself. A bare httpx.get() ignores the TLS and proxy settings VIP runs with ([tls] insecure / ca_bundle and the [proxy] block), so it fails against deployments behind a private CA or on a proxy-only network -- the environments VIP exists to validate. The clients carry all of it. If you must reach a non-Posit endpoint, read those settings off vip_config rather than assuming a direct route out.
• Do not hardcode URLs or credentials. Read them from vip_config or the narrower fixtures above so the extension checks whatever deployment the user configured, not a fixed address.
• Use target_fixture= to pass state from a when step to a then step, not a module-level global.

Worked example (minimal shape)

# my_check.feature
Feature: My check
  @connect
  Scenario: My endpoint responds successfully
    Given I have my endpoint to verify
    When I request my endpoint
    Then it responds successfully

# my_check.py
import pytest
from pytest_bdd import given, scenario, then, when


@pytest.mark.connect
@scenario("my_check.feature", "My endpoint responds successfully")
def test_my_check():
    pass


@given("I have my endpoint to verify")
def have_endpoint():
    pass


@when("I request my endpoint", target_fixture="status")
def request_endpoint(connect_client):
    # connect_client already carries your proxy, CA bundle and TLS settings.
    return connect_client.health()


@then("it responds successfully")
def responds_ok(status):
    assert status < 400