projspec

Imported from fsspec/projspec (AGENTS.md). Install upstream with npx skills add fsspec/projspec. Copyright stays with the author.

AGENTS.md — projspec

This document is a guide for AI coding agents working in this repository. It covers the architecture of the projspec package (located in src/projspec/), the three central class families, the contract every parse() method must honour, and the conventions used throughout.

Extensions and the Qt application (vsextension/, pycharm_plugin/, src/projspec/qtapp/) are out of scope. See vsextension/ACTIONS.md for the VSCode extension specification.

Repository layout

src/projspec/
  __init__.py          # public re-exports: Project, ProjectSpec, get_cls
  proj/
    base.py            # Project, ProjectSpec, ProjectExtra, ParseFailed
    *.py               # one file per concrete spec type
  content/
    base.py            # BaseContent + content registry
    *.py               # one file per concrete content type
  artifact/
    base.py            # BaseArtifact, FileArtifact + artifact registry
    *.py               # one file per concrete artifact type
  utils.py             # AttrDict, camel_to_snake, run_subprocess, …
  config.py            # get_conf / set_conf
vsextension/           # a UI for vscode, calling projspec as a subprocess
qtapp/                 # standalone UI on pyqt5, calling projspec in-process
tests/
  conftest.py          # shared fixtures (proj = Project("/data"))
  test_basic.py        # smoke tests
  test_roundtrips.py   # serialise / deserialise round-trips
  …

The three class families

1. Project (proj/base.py:43)

The top-level container for a parsed directory. It is not subclassed.

Key attributes set during __init__ → resolve():

Project.resolve() iterates every registered ProjectSpec subclass and calls cls(proj) (which runs match()) then inst.parse(). A ValueError / ParseFailed means the directory did not match that type and is silently skipped. Any other exception is logged but does not abort parsing.

ProjectExtra subclasses are handled differently: their contents and artifacts are merged directly into proj.contents / proj.artifacts rather than being stored in proj.specs.

2. ProjectSpec (proj/base.py:435)

Base class for every concrete project type. Subclasses are auto-registered on import via __init_subclass__ using their snake-case name as the key (proj/base.py:511).

Lifecycle inside Project.resolve():

cls(proj)          ← __init__ calls self.match(); raises ParseFailed if False
inst.parse()       ← populate self._contents and self._artifacts

Important class-level attribute:

Instance attributes after parse():

Public properties .contents and .artifacts delegate to _contents / _artifacts and call parse() lazily if they are None (proj/base.py:466).

ProjectExtra (proj/base.py:542)

A special subclass of ProjectSpec for cross-cutting concerns (CI/CD, Docker, pre-commit, requirements files, …). These specs are not standalone projects. After parsing, Project.resolve() merges their contents / artifacts into the root project rather than storing them in proj.specs.

3. BaseContent (content/base.py:11)

A dataclass holding descriptive information extracted from a project. Content objects are read-only descriptions; they have no executable behaviour.

Every subclass is a @dataclass that must include proj: Project as its first field (inherited from BaseContent).

Subclasses are auto-registered on import via __init_subclass__ (keyed by snake-case name).

Concrete content classes:

Helper enums used by Environment:

• Stack (PIP, CONDA, NPM) — packaging technology
• Precision (SPEC, LOCK) — how precisely the environment is pinned

4. BaseArtifact (artifact/base.py:14)

An executable action or producible output attached to a project.

Constructor signature: __init__(self, proj: Project, cmd: list[str] | None, **kwargs)

All extra keyword arguments are stored via self.__dict__.update(kwargs).

Key interface:

Subclasses are auto-registered on import via __init_subclass__.

FileArtifact (artifact/base.py:108) specialises BaseArtifact for outputs that are one or more files. Constructor adds fn: str (glob pattern for output path). _is_done() / _is_clean() check for the file's existence via proj.fs.glob(self.fn).

Concrete artifact classes:

Writing a parse() method

parse() is the core obligation of every ProjectSpec subclass. The base implementation simply raises ParseFailed, so not calling super().parse() is normal.

Contract

1. Populate self._contents and self._artifacts — both must be AttrDict instances (or remain empty AttrDict()). They must not be None after parse() returns.

2. Grouping convention — keys inside _contents / _artifacts are snake-case type names ("environment", "wheel", "process", …). If there are multiple instances of the same type, the value is itself an AttrDict keyed by an identifying name (e.g. "default", "test", "main").

   # single item
   self._contents["python_package"] = PythonPackage(proj=self.proj, package_name="foo")
   
   # multiple items of the same type
   self._artifacts["process"] = AttrDict(
       main=Process(proj=self.proj, cmd=["python", "__main__.py"]),
   )

3. Every content/artifact must receive proj=self.proj — this back- reference is required by BaseContent / BaseArtifact.

4. Raise ParseFailed (or any ValueError) on unrecoverable bad state — for example if a required file is malformed and you cannot produce meaningful output. Do not raise for optional fields that simply aren't present.

5. Read files via self.proj.get_file(name) or self.proj.fs — never use plain open(). This keeps parsing compatible with remote filesystems (S3, GCS, HTTP, …).

6. Use self.proj.basenames for existence checks — it is a {basename: full_path} dict of the top-level directory, already loaded, so it is cheap to query.

7. Keep it cheap — match() runs for every registered type on every directory; parse() runs immediately afterwards if match() returns True. Read only the files you actually need and avoid recursive directory traversal.

8. Use self.proj.pyproject for any data in pyproject.toml — it is a @cached_property that is shared with other specs in the same resolve pass.

Minimal example

from projspec.proj.base import ProjectSpec, ParseFailed
from projspec.content.environment import Environment, Stack, Precision
from projspec.artifact.python_env import LockFile
from projspec.utils import AttrDict


class MyTool(ProjectSpec):
    """Projects managed by mytool (mytool.toml present)."""

    spec_doc = "https://mytool.example.com/spec"

    def match(self) -> bool:
        return "mytool.toml" in self.proj.basenames

    def parse(self) -> None:
        import toml
        from projspec.utils import PickleableTomlDecoder

        try:
            with self.proj.get_file("mytool.toml") as f:
                meta = toml.load(f, decoder=PickleableTomlDecoder())
        except (OSError, ValueError):
            raise ParseFailed("Could not read mytool.toml")

        packages = meta.get("dependencies", [])
        self._contents = AttrDict(
            environment=Environment(
                proj=self.proj,
                stack=Stack.PIP,
                precision=Precision.SPEC,
                packages=packages,
                channels=[],
            )
        )
        self._artifacts = AttrDict(
            lock_file=LockFile(
                proj=self.proj,
                cmd=["mytool", "lock"],
                fn=f"{self.proj.url}/mytool.lock",
            )
        )

ProjectExtra.parse() — additional note

ProjectExtra.parse() should write to self._contents / self._artifacts (accessed as self.contents / self.artifacts via the property) exactly as above. Project.resolve() merges these into the root project automatically.

Registry and discovery

All three class families self-register via __init_subclass__:

projspec.proj.base.registry        # ProjectSpec subclasses
projspec.content.base.registry     # BaseContent subclasses
projspec.artifact.base.registry    # BaseArtifact subclasses

Keys are snake-case class names produced by camel_to_snake(cls.__name__). A new spec is discovered automatically the moment its module is imported. All concrete specs are imported in src/projspec/proj/__init__.py, so adding a new file there is all that is needed to register a new type.

get_cls(name, registry="proj") (utils.py:358) looks up any class by name across all registries.

AttrDict (utils.py:50)

A dict subclass that also supports attribute-style read access.

d = AttrDict(foo=42)
d.foo  # → 42
d["foo"]  # → 42

parse() always assigns AttrDict instances to self._contents and self._artifacts, never plain dict.

Serialisation

Every content, artifact, and spec class implements to_dict(compact=True).

• compact=True — human-readable condensed form (strings, nested dicts).
• compact=False — full form including a "klass" key that encodes the category and snake-case class name, enabling round-trip deserialisation via utils.from_dict().

The test suite exercises round-trips in tests/test_roundtrips.py.

Testing

pytest tests/

The proj fixture in tests/conftest.py returns projspec.Project("/data") — the repository root itself — which is a real PythonLibrary + GitRepo + Pixi project, so most tests work against live on-disk data.

New specs should add at minimum:

1. A match() unit test verifying the positive and negative cases.
2. A parse() unit test asserting expected keys in .contents and .artifacts.
3. A round-trip test (to_dict → from_dict) if the spec introduces new content or artifact classes.