pyyaml-include

Imported from sigil-enterprises/pyyaml-include (AGENTS.md). Install upstream with npx skills add sigil-enterprises/pyyaml-include. Copyright stays with the author.

pyyaml-include - Project Memory

Project Overview

A PyYAML extension constructor that enables including other YAML files into current YAML documents. Supports local and remote files (HTTP, S3, SFTP) via fsspec.

• Package: pyyaml-include
• Main module: yaml_include (imported as yaml_include)
• Python: >=3.9
• License: GPL-3.0-or-later
• Status: Production/Stable

Breaking Change: v2.0 is NOT backward compatible with v1.0

Codebase Architecture

Directory Structure

src/yaml_include/
├── __init__.py        # Public API exports
├── constructor.py     # Core Constructor class (main YAML tag handler)
├── data.py            # Data dataclass for include statement representation
├── representer.py     # Representer for YAML serialization
├── funcs.py           # Helper functions (load, lazy_load)
└── _version.py        # Version info (auto-generated by setuptools-scm)

tests/
├── test_basic.py      # Basic include tests
├── test_custom_loader.py  # Custom loader (JSON/TOML) tests
├── test_dataclass.py  # Data class tests
├── test_deep.py       # Deep/nested include tests
├── test_dump.py       # Serialization/dump tests
├── test_loadfunc.py   # load() function tests
├── test_multi.py      # Multiple file include tests
└── _internal.py       # Internal test utilities

Key Classes

Constructor (src/yaml_include/constructor.py:67)

The core class that handles YAML include tags. It's a dataclass with configurable attributes:

• fs: fsspec AbstractFileSystem (defaults to local filesystem)
• base_dir: Base directory for relative paths (can be string, PathLike, or callable)
• autoload: If True, auto-loads included files; if False, returns Data objects
• custom_loader: Optional custom loader function for non-YAML formats (JSON, TOML)

Key methods:

• __call__(loader, node) - Invoked by PyYAML when encountering include tag
• load(loader_type, data) - Loads and parses included file(s)
• managed_autoload(autoload) - Context manager for temporary autoload state

Data (src/yaml_include/data.py:8)

Immutable dataclass (frozen) representing a YAML include statement:

• urlpath: URL/path of the file (supports wildcards: **, ?, [..])
• flatten: If True, flattens sequences from multiple matched files
• sequence_params: Positional parameters for include statement
• mapping_params: Named parameters for include statement

Representer (src/yaml_include/representer.py:13)

Handles serialization of Data objects back to YAML tags.

Important: When creating, do NOT include ! prefix in tag name:

rpr = yaml_include.Representer("inc")  # Correct: no "!" prefix

Helper Functions (src/yaml_include/funcs.py)

• load(obj, loader_type, constructor, inplace, nested) - Recursively loads all Data instances
• lazy_load(obj, loader_type, constructor, nested) - Generator version for lazy loading

Development Setup

Dependencies (from pyproject.toml)

Runtime:

• PyYAML~=6.0
• fsspec>=2021.04.0
• typing-extensions (Python <3.11)

Dev Groups:

• test: coverage, toml, fsspec[http]
• typed: mypy, types-PyYAML
• docs: Sphinx, myst-parser, sphinx-book-theme, etc.
• ipy: ipykernel

Build System

• Backend: setuptools.build_meta
• Versioning: setuptools-scm (writes to src/yaml_include/_version.py)

Design Decisions & Patterns

1. fsspec for Unified Filesystem Interface

Uses fsspec to support multiple filesystems (local, HTTP, S3, SFTP) through a single API. The fs attribute of Constructor holds the filesystem instance.

2. Wildcard Pattern Matching

Shell-style wildcards (**, ?, [..]) are supported via WILDCARDS_PATTERN regex in constructor.py:44.

Important: Using ** in large directories or remote filesystems can be slow/expensive.

3. autoload Mode for Serialization

When autoload=False, the constructor returns Data objects instead of loaded content. This enables:

• YAML serialization of include statements (with Representer)
• Lazy loading via yaml_include.load() or yaml_include.lazy_load()

4. Custom Loader Support

The custom_loader parameter allows parsing non-YAML formats:

def my_loader(urlpath, file, Loader):
    if urlpath.endswith(".json"):
        return json.load(file)
    if urlpath.endswith(".toml"):
        return toml.load(file)
    return yaml.load(file, Loader)

5. Parameter Passing to fsspec

Parameters in YAML include statements are passed to fsspec methods:

• With scheme + wildcard → fsspec.open_files()
• With scheme + no wildcard → fsspec.open()
• No scheme + wildcard → fs.glob() then fs.open()
• No scheme + no wildcard → fs.open()

6. Type Safety

Uses type hints throughout. Conditional imports for Python version compatibility:

• TypeGuard (3.10+)
• Self (3.11+)

7. PyYAML Integration

Works with PyYAML's loader system via yaml.add_constructor():

yaml.add_constructor("!inc", yaml_include.Constructor(), yaml.Loader)

Common Usage Patterns

Basic Local File Include

import yaml
import yaml_include

yaml.add_constructor("!inc", yaml_include.Constructor(base_dir='/path/to/config'))

# In YAML:
# file: !inc include.d/1.yml

Remote File Include (HTTP)

import fsspec
import yaml
import yaml_include

http_fs = fsspec.filesystem("http", client_kwargs={"base_url": f"http://{HOST}:{PORT}"})
ctor = yaml_include.Constructor(fs=http_fs, base_dir="/")
yaml.add_constructor("!inc", ctor, yaml.Loader)

Serialization Support

import yaml
import yaml_include

ctor = yaml_include.Constructor(autoload=False)
yaml.add_constructor("!inc", ctor)

rpr = yaml_include.Representer("inc")
yaml.add_representer(yaml_include.Data, rpr)

Important Notes

• Version incompatibility: v2.0 breaks v1.0 API
• maxdepth auto-conversion: The maxdepth parameter is force-converted to int (PyYAML sometimes treats numbers as strings in constructors)
• flatten requirement: All matched files must have top-level Sequence objects when using flatten=True
• Recursive loading: load() and lazy_load() are recursive and can cause stack overflow on deeply nested structures
• Tag naming: Representer takes tag WITHOUT ! prefix (unlike add_constructor)
• Default loader: Uses yaml.CSafeLoader if libyaml is available, else yaml.safe_load
• Wildcard patterns: Does NOT support ^ for pattern negation

Testing

Tests are organized by feature:

• test_basic.py - Single file includes
• test_multi.py - Multiple file includes with wildcards
• test_deep.py - Nested includes
• test_custom_loader.py - JSON/TOML loading
• test_dump.py - Serialization with Representer
• test_loadfunc.py - load() and lazy_load() functions
• test_dataclass.py - Data class behavior

Run tests with: pytest tests/