Imported from jwvolschenk/llama-cpp-monitor (AGENTS.md). Install upstream with npx skills add jwvolschenk/llama-cpp-monitor. Copyright stays with the author.

llama-cpp-monitor — Agent Instructions

Lightweight monitoring dashboard for a local llama.cpp inference server. Python stdlib only (no pip dependencies). Chart.js loaded from CDN in dashboard.html.

Read README.md for environment variables, API endpoints, and systemd setup.

Project layout

Architecture (high level)

monitor.py (MONITOR_PORT, default 8100)
  ├── Background thread (every POLL_INTERVAL s, default 1s)
  │     ├── GET {LLAMA_URL}/health
  │     ├── GET {LLAMA_URL}/metrics  (Prometheus text → parse_prometheus_metrics)
  │     ├── GET {LLAMA_URL}/props    (once, cached)
  │     └── nvidia-smi --id={GPU_INDEX}
  ├── compute_deltas(prev, curr)     → instant throughput (idle → 0)
  ├── compute_hourly_averages()      → 1h rolling prefill/decode rates
  └── MonitorHandler (GET /, /api/status, /api/history, /api/props)

Key idea: counters from llama.cpp are cumulative; the dashboard shows delta-based instant throughput so charts spike when busy and flatline when idle.

Agent operating guidance

1. Keep changes minimal — this is a small, self-contained tool. Prefer editing monitor.py and dashboard.html only unless the task touches deployment docs or the service file.
2. No external Python dependencies — use only the stdlib (http.server, urllib, threading, subprocess, etc.). Do not add requirements.txt or pip packages unless explicitly requested.
3. Preserve delta semantics — new metrics should follow the existing pattern: raw cumulative values in collect_once(), derived instant values in compute_deltas(), and chart-friendly fields on each history point.
4. Thread safety — shared state (current_status, history*, props_cache) is guarded by lock. Any new shared state must use the same lock.
5. Config via env vars — new tunables belong as os.environ.get(...) at the top of monitor.py and should be documented in README.md.
6. Dashboard contract — dashboard.html expects specific JSON shapes from /api/status and /api/history. If you change field names or types, update both sides.

High-value symbols (start here)

Prometheus metric names used (llama.cpp): llamacpp:prompt_tokens_total, llamacpp:tokens_predicted_total, llamacpp:requests_processing, llamacpp:requests_deferred, etc.

Running

python3 monitor.py
# Dashboard at http://localhost:8100 (or MONITOR_PORT)

Useful env overrides:

LLAMA_HOST=localhost LLAMA_PORT=8080 MONITOR_PORT=8100 GPU_INDEX=1 python3 monitor.py

Conventions

• Python 3.10+ type hints (dict | None, etc.) — match existing style in monitor.py
• No request logging in MonitorHandler.log_message (intentionally silenced)
• LLAMA_API_KEY sent as Authorization: Bearer … when set
• Dark-theme dashboard; keep UI changes consistent with existing CSS variables in dashboard.html

Code intelligence: using CodeDB

CodeDB is an indexed, symbol-aware navigation layer (MCP tools). Use it before blind grep or reading whole files — this repo is small, but CodeDB still gives outlines, dependency context, and targeted reads.

First-time setup for this project

This directory is not a git repository. CodeDB refuses non-git roots by default. Use one of:

1. Project .codedbrc (included here) — require_git_repo = false. CodeDB loads config from the MCP server’s startup CWD, so ensure that setting is active (e.g. start MCP from this directory, or mirror the line in your global ~/.codedb/.codedbrc / binary-dir config).
2. Environment override — CODEDB_REQUIRE_GIT_REPO=0 when starting the MCP server.
3. git init — a fresh repo with no commits still satisfies the work-tree check if you prefer to keep the default.

Index once per session (lazy MCP mode):

codedb_index path=/mnt/storage/repos/llama-cpp-monitor

Confirm health:

codedb_status
codedb_tree

Re-index after large structural changes or if results look stale:

codedb_index path=/mnt/storage/repos/llama-cpp-monitor force=true

Recommended tool flow

1. codedb_status — confirm snapshot and scan state.
2. codedb_tree or codedb_ls — orient (only a handful of source files).
3. codedb_symbol / codedb_word — locate a function or identifier.
4. codedb_outline — see structure before reading (especially dashboard.html).
5. codedb_read — read only the line range you need (line_start / line_end).
6. codedb_callers / codedb_deps / codedb_relations — impact analysis before edits.

Chain steps in one call with codedb_query when narrowing then reading:

[
  {"op": "symbol", "name": "compute_deltas"},
  {"op": "read", "context_lines": 5}
]

Project-specific examples

Tool quick reference

Investigation patterns

Pattern A — Add a new metric to the dashboard

1. codedb_symbol("collect_once") + codedb_symbol("compute_deltas") — see where raw and derived metrics live.
2. codedb_search(query="deltas") in dashboard.html — find chart bindings.
3. Extend collector → deltas → refresh() in HTML; keep field names consistent.

Pattern B — Change an API response shape

1. codedb_symbol("MonitorHandler") + codedb_read on do_GET.
2. codedb_word("api/status") or search fetch('/api/status') in dashboard.html.
3. Update handler and frontend together.

Pattern C — Debug GPU stats

1. codedb_symbol("get_gpu_stats") — nvidia-smi invocation and field mapping.
2. codedb_search(query="GPU_INDEX") — env wiring.

Guardrails

• Do not commit .env or real API keys; use .env.example as the template.
• Do not add heavyweight frameworks (Flask, FastAPI, etc.) without explicit request — stdlib HTTP server is intentional.
• Do not change poll/history defaults without updating README.md and any chart time-window logic in dashboard.html.
• GPU stats require nvidia-smi on the host; handle missing GPU gracefully (existing {"error": ...} pattern).