libby

Imported from Deeplearn-PeD/libby (AGENTS.md). Install upstream with npx skills add Deeplearn-PeD/libby. Copyright stays with the author.

AGENTS.md - Coding Agent Guidelines for Libby D. Bot

Libby D. Bot is an AI-powered librarian for RAG. It processes PDF documents, generates embeddings, and enables question answering over document collections.

Build/Lint/Test Commands

Package Manager (uv)

uv sync                    # Install dependencies
uv add <package-name>      # Add a dependency
uv add --dev <package-name> # Add a dev dependency

Running Tests

uv run pytest                           # Run all tests
uv run pytest tests/test_cli.py         # Run a single test file
uv run pytest tests/test_cli.py::TestLibbyInterface::test_initialization_default  # Single test
uv run pytest -v                        # Verbose output
uv run pytest tests/brain/              # Run tests in directory
uv run pytest -k "embed"                # Run tests matching pattern

Running the TUI

uv run libby              # Launch the interactive Textual TUI

Running the Legacy CLI (for scripting)

uv run libby-cli embed --corpus_path /path/to/docs --collection_name my_collection
uv run libby-cli answer "What is the main topic?" --collection_name my_collection
uv run libby-cli generate "Write a summary..." --output_file output.txt

Code Style Guidelines

Imports

Order: standard library → third-party → local. Separate groups with blank lines.

# Standard library
import os
from glob import glob

# Third-party
import fitz
import loguru
from pydantic import BaseModel, Field
from sqlmodel import SQLModel, Field, create_engine, Session

# Local
from libbydbot.brain.ingest import PDFPipeline
from libbydbot.settings import Settings

Type Annotations

Use Python 3.10+ union syntax (int | None instead of Optional[int]). Use list[str] and dict[str, Any] instead of List[str] and Dict[str, Any].

Naming Conventions

• Classes: PascalCase (DocEmbedder, PDFPipeline, LibbyInterface)
• Functions/Methods: snake_case (embed_text, retrieve_docs)
• Private methods: Prefix with underscore (_generate_embedding)
• Constants: UPPER_SNAKE_CASE (PROVIDERS)
• Module singletons: lowercase (logger, settings)

Docstrings

def embed_text(self, doctext: str, docname: str, page_number: int):
    """
    Embed a page of a document.
    :param doctext: page of a document
    :param docname: name of the document
    :param page_number: page number
    """

Error Handling

Use try/except with loguru logging. Log levels: logger.info() for normal operations, logger.warning() for recoverable issues, logger.error() for errors.

try:
    session.add(doc_vector)
    session.commit()
except IntegrityError as e:
    session.rollback()
    logger.warning(f"Document {docname} already exists: {e}")
except ValueError as e:
    logger.error(f"Error: {e} generated when processing: {doctext}")
    session.rollback()

Class Structure

Order: __init__ → @property → public methods → private methods (_ prefix) → dunder methods.

Database Patterns

Supports SQLite, DuckDB, and PostgreSQL:

if self.dburl.startswith("sqlite"):
    # SQLite-specific code
elif self.dburl.startswith("duckdb"):
    # DuckDB-specific code
else:
    # PostgreSQL code

Configuration

Use pydantic-settings with SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="allow").

Testing

Both unittest.TestCase and pytest styles are acceptable. Use pytest fixtures for mocking:

@pytest.fixture(autouse=True)
def mock_embeddings():
    with patch('libbydbot.brain.embed.DocEmbedder._generate_embedding') as mocked:
        mocked.return_value = np.zeros(1024).tolist()
        yield mocked

Async Patterns

Use nest_asyncio.apply() and asyncio.run() for async operations.

Project Structure

libby/
├── libbydbot/           # Main package
│   ├── __init__.py      # Persona class
│   ├── cli.py           # TUI entry point (Textual)
│   ├── cli_legacy.py    # Legacy Fire-based CLI
│   ├── settings.py      # Pydantic settings
│   ├── tui/             # Textual TUI
│   │   ├── app.py       # Main App
│   │   ├── libby.tcss   # TUI styles
│   │   ├── screens/     # TUI screens
│   │   │   ├── dashboard.py
│   │   │   ├── chat.py
│   │   │   ├── embed.py
│   │   │   ├── wiki_browser.py
│   │   │   ├── wiki_ingest.py
│   │   │   └── settings.py
│   │   └── widgets/     # Reusable TUI widgets
│   │       └── status_bar.py
│   └── brain/           # Core functionality
│       ├── __init__.py  # LibbyDBot class
│       ├── embed.py     # Document embedding
│       ├── ingest.py    # PDF ingestion
│       ├── memory.py    # Chat history
│       ├── analyze.py   # Article summarization
│       ├── wiki.py      # LLM Wiki manager
│       ├── wiki_models.py # Structured wiki schemas
│       └── graph.py     # Knowledge graph over wiki pages + chunks
├── tests/
│   ├── conftest.py      # Shared fixtures
│   ├── test_tui.py      # TUI tests
│   └── brain/
└── pyproject.toml

Environment Variables

• PGURL - PostgreSQL connection URL
• OLLAMA_HOST - Ollama server URL (default: http://localhost:11434)
• GEMINI_API_KEY - Google Gemini API key
• ZHIPUAI_API_KEY - Zhipu AI API key (for GLM models; required by the default wiki model glm-5-turbo)
• WIKI_BASE_PATH - Base directory for LLM wikis (default: ~/.libby/wikis)
• WIKI_AUTO_INGEST - Automatically ingest documents into the wiki after embedding (default: True). Reads straight from the embedding table, so no PDF re-parsing is needed.
• WIKI_MODEL - Dedicated LLM model for wiki ingest/query/summary (default: glm-5-turbo). Must be a non-thinking model that supports structured tool-calling — thinking models (kimi-k2.6, deepseek-v4-flash/pro) reject pydantic-ai tool_choice and yield empty summaries. Working options: glm-5-turbo/glm-5.2 (Zhipu), gpt-4o, gemini-2.5-flash.
• WIKI_GRAPH_ENABLED - Maintain a knowledge graph over wiki pages and embedded chunks (default: True)

Supported Models

Llama3.2 (default), Gemma3, GPT-4o, Qwen3, Gemini

Database URLs

• SQLite: sqlite:///path/to/db.sqlite
• DuckDB: duckdb:///path/to/db.duckdb
• PostgreSQL: Full postgresql:// URL

Release Process

Use the release script to commit changes, bump version, and create git tags:

./scripts/release.sh

The script will:

1. Show current git status and pending changes
2. Ask for version bump type (major/minor/patch)
3. Update version in pyproject.toml
4. Commit all changes with a message
5. Create an annotated git tag (e.g., v0.9.0)

After running the script, push changes with:

git push && git push --tags

Version Bump Guidelines

• patch: Bug fixes, documentation updates, small tweaks (0.8.0 → 0.8.1)
• minor: New features, backward-compatible changes (0.8.0 → 0.9.0)
• major: Breaking changes, major refactors (0.8.0 → 1.0.0)

LLM Wiki Conventions

Libby maintains an LLM Wiki — a persistent, compounding markdown knowledge base that sits between raw embedded documents and the user. The wiki is stored as Obsidian-compatible markdown files.

Wiki Directory Structure

Each collection gets its own wiki under WIKI_BASE_PATH (default: ~/.libby/wikis/<collection_name>/):

<collection_name>/
├── index.md          # Content-oriented catalog of all pages
├── log.md            # Chronological append-only record of operations
├── sources/          # One page per ingested source document
├── entities/         # Pages for people, organizations, objects
├── concepts/         # Pages for topics, theories, ideas
└── synthesis/        # Overviews, answers, comparisons, analyses

Page Format

All pages use YAML frontmatter and Obsidian-style [[wikilinks]]:

---
title: Page Title
date_created: 2026-04-21T10:00:00
---

# Page Title

Content here with [[Other Page]] links.

Ingest Workflow

1. Read source content (from raw PDF or embedded chunks via WikiManager.ingest_from_embeddings, which reads straight from the embedding table — no PDF re-parsing, works when source files are no longer on disk).
2. Generate SourceSummary (entities, concepts, contradictions).
3. Generate WikiUpdatePlan (pages to create/update).
4. Write/update:
   • sources/<doc_name>.md
   • entities/<entity>.md (create or append mention)
   • concepts/<concept>.md (create or append mention)
   • synthesis/overview.md (if synthesis notes exist)
5. Update index.md and append to log.md.

Auto-ingest: when WIKI_AUTO_INGEST is True (the default), the embed routes automatically run ingest_from_embeddings for each newly embedded document, so the wiki stays in sync with the embedding store without manual steps. This can be disabled via the WIKI_AUTO_INGEST env var.

Per-part merging: large documents that were split into parts at embedding time (stored under several doc_name values that share a part suffix, e.g. report_part1, report_part2, report (1)) are concatenated back into a single source by ingest_from_embeddings (default merge_parts=True), so the wiki gets one sources/<doc_name>.md page per original document. To fix a wiki that already has one page per part, call WikiManager.consolidate_part_pages() (or POST /api/wiki/consolidate / libby-cli wiki_consolidate), which merges the part pages and rewrites inbound [[wikilinks]].

Query Workflow

1. Read index.md to identify candidate pages.
2. Read the most relevant pages (up to 15, keyword-ranked).
3. LLM synthesizes a cited answer using [[Page Name]] citations.
4. Optionally file the answer back into synthesis/ and update index.md.

Lint Workflow

Periodic health-checks scan for:

• Orphan pages — pages with zero inbound wikilinks
• Broken links — wikilinks pointing to non-existent pages
• Contradictions — conflicting claims between pages
• Stale claims — claims superseded by newer sources
• Missing pages — important terms mentioned but lacking dedicated pages

Auto-fix creates stub pages for broken links with status: stub frontmatter.

Knowledge Graph

Libby maintains a NetworkX knowledge graph per collection, persisted as graph.json inside the wiki directory. WikiKnowledgeGraph (libbydbot/brain/graph.py) builds and queries it.

• Page nodes — one per source/entity/concept/synthesis page (id = relative path).
• Chunk nodes — one per embedded chunk (id = chunk:<doc_hash>).
• Edge types — mentions, mentioned_in, related_to, links_to, chunk_of (chunk → source), references (chunk → entity/concept matched by name).
• Broken wikilinks become stub nodes; stubs are upgraded when real pages appear.
• The graph is updated incrementally on ingest and can be fully rebuilt from disk (chunk nodes and reference edges survive rebuilds).

Graph capabilities:

• Graph-aware query — WikiManager.query() ranks pages via seed matching + 1-hop neighborhood expansion + degree, replacing the naive keyword filter.
• Path — shortest path between two concepts.
• Explain — a node's attributes plus inbound/outbound connections.
• Subgraph query — ranked nodes/edges for a question.
• Hubs — most-connected nodes.
• Visualization — export an interactive graph.html via pyvis.

CLI Commands (Legacy)

uv run libby-cli wiki_ingest --corpus_path /path/to/docs --collection_name my_collection
uv run libby-cli wiki_query "What is the main topic?" --collection_name my_collection --file_answer
uv run libby-cli wiki_lint --collection_name my_collection --auto_fix
uv run libby-cli wiki_status --collection_name my_collection
uv run libby-cli wiki_consolidate --collection_name my_collection
uv run libby-cli wiki_graph --collection_name my_collection
uv run libby-cli wiki_path "Alice" "Bob" --collection_name my_collection
uv run libby-cli wiki_explain "Alice" --collection_name my_collection
uv run libby-cli wiki_graph_export --collection_name my_collection

In the TUI, use the Wiki Browser screen (Ctrl+W) and the Ingest and Graph buttons.

REST API Endpoints

• POST /api/wiki/ingest — ingest a source into the wiki
• POST /api/wiki/ingest-from-embeddings — build/update the wiki straight from the embedding table (no PDF re-parsing)
• POST /api/wiki/consolidate — merge per-part source pages into one collective page per document
• POST /api/wiki/query — query the wiki
• POST /api/wiki/lint — lint the wiki
• GET /api/wiki/status/{collection_name} — wiki statistics (includes graph stats)
• GET /api/wiki/pages/{collection_name} — list all wiki pages grouped by category
• GET /api/wiki/page/{collection_name} — read a single wiki page (?category=...&page=...)
• POST /api/wiki/graph/rebuild — rebuild the knowledge graph
• GET /api/wiki/graph/{collection_name} — knowledge graph statistics
• GET /api/wiki/graph/{collection_name}/viz — serve the interactive graph.html visualization (rebuilt fresh on each request)
• POST /api/wiki/graph/path — shortest path between two nodes
• POST /api/wiki/graph/explain — explain a node
• POST /api/wiki/graph/query — ranked subgraph for a question