proxmox-k8s-public

Imported from wrightpt/proxmox-k8s-public (AGENTS.md). Install upstream with npx skills add wrightpt/proxmox-k8s-public. Copyright stays with the author.

AGENTS.md — Proxmox K8s Platform

This file is the authoritative reference for AI coding agents working on this repository. It describes the actual project structure, technology stack, build processes, and operational conventions. Read this first before making any changes.

1. Project Overview

This repository contains the complete infrastructure and application stack for a prediction-market platform running on a self-hosted Kubernetes cluster built with Talos Linux. The stack is managed entirely via GitOps (ArgoCD) — no manual kubectl apply or helm upgrade is permitted on managed resources.

The repository is a monorepo with four main domains:

1. Infrastructure & GitOps (kubernetes/, talos-configs/, talos-pxe/) — Cluster bootstrap, platform services, and workload manifests.
2. Frontend Applications (nextjs-market/) — A pnpm-workspace monorepo containing the main prediction-web app and an operations-dashboard.
3. Backend Microservices (services/) — Standalone services for betting bots, NBA predictions, AI error analysis, metrics export, and real-time log streaming.
4. Operational Tooling (scripts/, docs/) — Shell scripts for cluster operations, backups, monitoring, and extensive runbooks.

Trading Strategy North Star

When working on the trading bot, research harness, operations dashboard, or client-facing strategy docs, translate the user's "highest return possible" goal into this objective:

Maximize durable, after-cost expected return per unit of drawdown, liquidity risk, venue risk, and operational risk.

Daily operation means the system scans markets, refreshes signals, marks PnL, updates risk state, and explains whether it traded, managed an open position, or stood down. It does not mean forcing a new trade every calendar day, and it does not mean promising guaranteed daily profit.

Future agents should optimize for:

• Positive expectancy after fees, slippage, funding, borrow/transfer costs, and realistic fill assumptions.
• Capital-efficient carry and basis opportunities that can stay open while the risk gate remains favorable.
• Clear no-trade reasons when the market is crowded, stale, illiquid, regime-shifted, or not worth the fee churn.
• Paper-first validation with forward evidence before live execution.
• Explicit kill switches, exposure limits, reconciliation, and audit trails before any real capital is connected.

Do not make changes whose main effect is increasing trade count, dashboard excitement, or client promises. A bot that refuses a bad trade is behaving correctly.

Current direction (2026-05-10): build a multi-asset perpetual-futures strategy engine. XMR funding-carry remains a research/paper sleeve, not the whole objective. Perps can be longed or shorted directly, so do not assume spot shorting. Funding is signed by position and settlement schedule; it must be modeled alongside maker/taker fees, spread, slippage, margin/liquidation risk, venue risk, and operational risk.

Hyperliquid execution note (2026-05-13): signed Hyperliquid order placement exists only through the gated testnet/mainnet execution path in services/xmr-market-data/src/jobs/hyperliquid_testnet_paper.py. Testnet is CronJob-suspended plus env-gated; mainnet is separately opt-in and forward-gate blocked. The job reconciles venue positions before fresh submissions and uses a pure risk-exit policy to emit reduce-only closes when stop/risk thresholds trip.

Hyperliquid/neutral-carry update (2026-05-13): signed Hyperliquid execution work is testnet-first. The lead delta-neutral sleeve should be modeled as external spot/index/reference exposure plus a Hyperliquid perp leg unless a specific Hyperliquid spot pair has been validated for symbol mapping, liquidity, fees, and execution quality. Use actual account fee tiers from Hyperliquid userFees when available, hourly funding semantics for Hyperliquid perps, and public/read-only exchange snapshots before signed testnet orders. Do not connect mainnet capital from strategy work without explicit live approval, compliance review, margin/symbol/fee validation, reconciliation, and kill switches.

Hyperliquid local capability note (2026-05-13): the Hyperliquid Python SDK is installed in the services/xmr-market-data/.venv environment. A Hyperliquid signing key may be available through the approved env/Vault path; never read, print, or commit key material. Treat any existing tiny perp test position as live state that must be reconciled before further signed testnet or mainnet actions.

Implementation: the trading bot and research harness live at services/xmr-market-data/. Its canonical guide is services/xmr-market-data/AGENTS.md. For the XMR variant taxonomy and lineage, see services/xmr-market-data/docs/STRATEGY_REGISTRY.md. For the multi-asset perp direction, see services/xmr-market-data/docs/PERP_STRATEGY_ENGINE_PLAN.md.

2. Technology Stack

Cluster Infrastructure

Data & Messaging

Frontend / Application Tier

Backend Services

3. Project Structure & Module Organization

proxmox-k8s/
├── kubernetes/                 # GitOps manifests — THE SOURCE OF TRUTH for cluster state
│   ├── bootstrap/              # ArgoCD bootstrap chain
│   ├── apps/
│   │   ├── platform/           # cert-manager, Vault, Traefik, Cloudflare, ArgoCD
│   │   ├── data-services/      # CNPG, Redis, ClickHouse, MinIO
│   │   ├── workloads/          # prediction-web, monero, betbot, nba-predictions
│   │   ├── security/           # External Secrets Operator
│   │   ├── networking/         # Cilium, nodelocal-dns
│   │   ├── monitoring/         # Prometheus rules, Alertmanager, Perses
│   │   ├── ops/                # CronJobs
│   │   └── prediction-web-dev/ # Dev hot-swap environment
│   ├── base/                   # Foundational namespaces (13 base ns, sync-wave -10)
│   ├── helm-values/            # Centralized Helm values (24 charts)
│   ├── monitoring/             # Prometheus alerting rules
│   ├── namespaces/             # Additional env namespaces
│   └── vault-policies/         # Vault ACL policies
│
├── nextjs-market/              # pnpm workspace monorepo (NOT Turborepo)
│   ├── apps/
│   │   ├── prediction-web/     # Main prediction market app (port 3000)
│   │   └── operations-dashboard/ # Observability & admin UI (port 3003)
│   └── pnpm-workspace.yaml     # Workspaces: apps/*, packages/*
│
├── services/                   # Standalone microservices
│   ├── betbot/
│   ├── nba-predictions/
│   ├── ai-error-analyzer/
│   ├── prediction-metrics/
│   ├── realtime-logs/
│   ├── btc-funding-rates/
│   └── xmr-market-data/        # Multi-asset perp research + XMR paper trader (see services/xmr-market-data/AGENTS.md)
│
├── talos-configs/              # Talos machine configs (ACTIVE in pxe-configs/)
│   ├── pxe-configs/            # MAC-addressed node configs, talosconfig, kubeconfig
│   ├── 3-node-cluster/         # Legacy install artifacts
│   └── configs/, fixed/        # Other variants
│
├── talos-pxe/                  # PXE boot assets
│   ├── boot.ipxe               # Primary maintenance-mode boot
│   ├── boot-debug.ipxe         # Verbose debug boot
│   ├── talos/                  # vmlinuz + initramfs.xz
│   └── configs/                # Runtime working copies
│
├── scripts/                    # ~40 operational shell scripts
│   ├── backup-*.sh             # Cluster backups
│   ├── bmc.sh                  # Supermicro BMC wrapper
│   ├── db-query.sh             # PG query via kubectl
│   ├── populate-vault-secrets.sh
│   ├── sync-app.sh             # ArgoCD sync trigger
│   └── add-opnsense-firewall-rules-*.sh
│
├── docs/                       # Runbooks and architecture docs
│   ├── runbooks/               # Talos rebuild, PG backup, etcd restore
│   ├── incidents/              # Post-incident reports
│   ├── monero/                 # Monero guides
│   └── security/               # Image build policy
│
├── infra/containers/           # Custom builds (Monero wallet RPC)
├── db/                         # schema.sql (minimal)
├── .github/workflows/          # GitHub Actions
└── backups/                    # Backup target (gitignored)

Key Conventions

• GitOps ONLY: Every cluster change must go through Git commit and ArgoCD sync. Manual kubectl apply or helm upgrade on managed resources is forbidden.
• ArgoCD App-of-Apps: bootstrap/bootstrap-manager-app.yaml (wave -100) → bootstrap/argocd-config-app.yaml (wave 0) → apps/platform/gitops/root/root-app-manager.yaml (wave 5) → all individual *-app.yaml applications.
• Sync Waves: Resources declare argocd.argoproj.io/sync-wave annotations to control deployment order (namespaces at -10, cert-manager at 1, Vault at 2, workloads at 5+).
• Node Affinity: Most workloads pin to talos-cp-1 and talos-cp-2, explicitly excluding talos-cp-3 (consumer hardware). Control-plane nodes run workloads via tolerations.

4. Build, Test, and Development Commands

Root-level (nextjs-market monorepo)

# From nextjs-market/
pnpm dev              # Start prediction-web dev server (port 3000)
pnpm dev:web          # Same, shorthand
pnpm build            # Build prediction-web
pnpm build:all        # Build all workspace packages
pnpm lint             # Lint prediction-web
pnpm ci:web           # Lint + unit tests for @prediction-market/web

prediction-web (nextjs-market/apps/prediction-web)

pnpm dev              # Local dev server (auto-detects K8s vs local)
pnpm build            # Production build (32GB heap, 12 workers)
pnpm build:prod       # Explicit production env build
pnpm typecheck        # TypeScript check (tsconfig.app.json)
pnpm lint             # ESLint on src/**/*.ts, src/**/*.tsx
pnpm test:unit        # Jest unit tests (jest.config.unit.js)
pnpm test:e2e         # Jest E2E tests (jest.config.e2e.mjs)
pnpm test:integration # Jest integration tests (requires test DB)
pnpm test:playwright  # Playwright E2E tests (playwright.config.ts)
pnpm test:playwright:ui   # Playwright with UI mode
pnpm db:migrate       # Run dbmate migrations (db/migrations/)
pnpm db:new <name>    # Create new migration
pnpm worker           # Run wallet-queue worker
pnpm pool-worker      # Run pool worker via tsx

operations-dashboard (nextjs-market/apps/operations-dashboard)

pnpm dev              # Local dev (port 3003)
pnpm dev:local        # Direct Next.js dev
pnpm build            # Production build
pnpm start            # Start production server
pnpm lint             # ESLint

Services

Each service in services/ has its own build process:

# betbot (TypeScript)
cd services/betbot
pnpm install
pnpm build            # tsc → dist/
pnpm start            # node dist/index.js

# nba-predictions (Python)
cd services/nba-predictions
pip install -r requirements.txt
python src/jobs/daily_update.py
pytest tests/         # pytest suite

# prediction-metrics (TypeScript)
cd services/prediction-metrics
pnpm install
pnpm start            # node src/index.js

Cluster Operations

# Talos health
talosctl -n 10.10.0.11 health --server

# ArgoCD sync status
kubectl get applications -n argocd

# GitOps dry-run validation
kubectl apply --dry-run=client -f <manifest.yaml>

# Trigger ArgoCD sync for an app
./scripts/sync-app.sh <app-name>

5. Code Style Guidelines

TypeScript / JavaScript

• Indentation: 2 spaces
• Formatter: Prettier (run via pnpm lint -- --fix)
• Linter: ESLint (Next.js config for apps)
• File naming:
  • Components: PascalCase.tsx
  • Utilities/hooks: camelCase.ts
  • Manifests: kebab-case.yaml
• Imports: Prefer absolute imports using path aliases (@/components, @/lib, etc.)

Python

• No explicit linter config detected. Follow PEP 8 conventions.
• requirements.txt per service. No central lockfile.

Kubernetes / YAML

• Indentation: 2 spaces
• Line length max: 150 (per .yamllint)
• Unix newlines required
• Preserve ArgoCD annotations: Never remove argocd.argoproj.io/sync-wave or argocd.argoproj.io/sync-options without explicit reason.
• Dry-run before commit: kubectl apply --dry-run=client -f <file>

Pre-commit Hooks

Configured in .pre-commit-config.yaml:

• Shellcheck (shell scripts)
• yamllint (YAML files, ignores .github/ and credentials/)
• hadolint (Dockerfiles)
• Trailing whitespace / EOF fixer
• detect-private-key
• Gitleaks is disabled (commented out) in pre-commit but .gitleaks.toml exists for CI scanning.

6. Testing Instructions

Frontend (prediction-web)

• Unit tests: jsdom environment, babel-jest transform, coverage threshold 60% global.
• Playwright: Tests in tests/e2e-playwright/. Projects: Chromium, Firefox, WebKit, Mobile, Smoke, Unauthenticated. Global setup handles auth.
• Integration tests: Require a running test database (managed via scripts/test-db.sh).
• No formal coverage threshold beyond the 60% global in unit tests, but add regression tests for bug fixes.

Python (nba-predictions)

cd services/nba-predictions
pytest tests/

• Tests cover feature engineering (test_features.py) and prediction logic (test_predict.py).
• Validates no data leakage (e.g., shift(1) usage in rolling features).

Other Services

• betbot: Has vitest in devDependencies but no test files exist yet.
• prediction-metrics, realtime-logs, ai-error-analyzer, btc-funding-rates: No tests detected.

GitOps Validation

Before committing manifest changes:

kubectl apply --dry-run=client -f <file>
yamllint <file>

7. GitOps & Deployment

CI/CD Workflows (.github/workflows/)

Deployment Patterns

• Canary: prediction-web-canary runs in parallel with production. CI auto-deploys to canary on main merge. Production promotion is manual via production branch push.
• Image Updates: Workflows use sed to update image tags in Kubernetes manifests, then commit with 5-attempt rebase/push retry logic.
• Commit Format: ci(<app>): deploy main-<short-sha>
• Concurrency: Deploy workflows use concurrency: group: deploy-main to serialize deployments.
• Registry: GHCR (ghcr.io/wrightpt/<image>). Login via secrets.GH_PAT.

Docker Build Strategy

• prediction-web: Dockerfile.simple is the canonical production Dockerfile. Multi-stage: base → builder → deployer → runner. Uses pnpm deploy and Next.js output: 'standalone'. Based on node:22-alpine.
• operations-dashboard: Dockerfile. Multi-stage, node:20-alpine, standalone output, exposes 3003.
• betbot: Dockerfile. Multi-stage, node:22-alpine, runs as betbot user (uid 1001).

8. Runtime Environments & K8s Hot-Swapping

Production Namespaces

Dev Hot-Swap Environments

The project uses an in-cluster dev hot-swap pattern instead of local .env files:

prediction-web-dev (kubernetes/apps/workloads/prediction-web-dev/):

• Runs a node:20-alpine pod with sleep infinity.
• Local code is synced via dev-sync.sh + dev-shell.sh (or dev-auto-sync-shell.sh).
• Accessible via dev.3xmr.com.
• Secrets injected from Vault via External Secrets Operator (no .env files in the pod).

ops-dashboard-dev (kubernetes/apps/platform/observability/ops-dashboard-dev/):

• Same two-terminal pattern (dev-sync.sh + dev-shell.sh).
• Accessible via its dev IngressRoute hostname.

Log Pipeline

• Apps log to stdout/stderr.
• Vector DaemonSet collects logs and ships to Elasticsearch (logs-<env>-YYYY.MM.DD).
• Operations Dashboard queries historical logs from Elasticsearch via /api/internal/search.

9. Security Considerations

Secret Management

• Vault is the source of truth for all secrets. Do not commit secrets to Git.
• .env files are used for local development but are gitignored.
• External Secrets Operator syncs Vault KV v2 secrets into Kubernetes Secrets.
• Vault must be unsealed after bootstrap (vault-init-keys-backup.json).

Known Sensitive Files (DO NOT READ/WRITE UNLESS NECESSARY)

• .env — Contains live secrets.
• talos-configs/pxe-configs/talosconfig — Talos admin credentials.
• talos-configs/3-node-cluster/kubeconfig — Kubernetes admin credentials.
• credentials/ — Wallet credentials and scripts.
• backups/ — Database backups.
• .mcp.json — Contains API keys.

Gitleaks

• .gitleaks.toml defines custom rules for Monero RPC creds, Supabase keys, Vault tokens, and high-entropy strings.
• .gitleaksignore lists known false positives.
• Pre-commit Gitleaks hook is currently disabled; scanning is CI-only.

Network Security

• Traefik uses RE2 regex for routing (incident 2025-10-21 demonstrated the danger of non-RE2 regex).
• OPNsense firewall rules automate WAN→Traefik access (ports 80/443).
• Monero stack uses defense-in-depth: NetworkPolicies → JWT → Method whitelist → Rate limit → Digest auth → Wallet encryption.

10. Payment Rail Guidance

The platform is in a mixed Monero/Tron migration state.

• Monero is LEGACY. Do not add new Monero-specific behavior unless explicitly requested. The Monero stack (kubernetes/apps/workloads/monero/) includes a StatefulSet daemon, hot/warm wallets, and a wallet-rpc-proxy. Safe shutdown procedures are critical to prevent LMDB corruption.
• Tron is the preferred path. Extend the emerging Tron layer (src/server/tron, src/workers/tron-worker-main.ts, ops dashboard src/features/tron-ops) into the canonical payment adapter.
• The internal ledger and market engine are chain-agnostic. Payment rails should be thin adapters around deposit address assignment, deposit detection, withdrawals, sweeps, and reconciliation.
• When modifying payment routes, move toward one canonical payment service layer. Keep old /api/wallet/* or chain-specific endpoints as compatibility wrappers only.

Before changing any payment code, read:

• nextjs-market/apps/prediction-web/docs/PAYMENT_RAIL_ARCHITECTURE.md
• docs/monero/SAFE-SHUTDOWN-PROCEDURE.md

11. Talos & Cluster Infrastructure

Cluster Topology

Talos Configuration

• Active configs: talos-configs/pxe-configs/ (Git-backed source of truth).
• Talosconfig: talos-configs/pxe-configs/talosconfig
• Kubeconfig: talos-configs/3-node-cluster/kubeconfig
• Node configs are named by MAC address (mac-<MAC>.yaml).
• Talos versions: installer:v1.11.2, kubelet:v1.34.1.

PXE Boot

• PXE server runs at 10.10.0.74:8080.
• talos-pxe/boot.ipxe boots Talos in maintenance mode.
• talos-pxe/boot-debug.ipxe chains MAC-specific configs dynamically.

12. Database & Migrations

Prediction Web

• Migration tool: Dbmate (Go binary).
• Migration directory: nextjs-market/apps/prediction-web/db/migrations/
• Commands: pnpm db:migrate, pnpm db:new <name>, pnpm db:status
• Schema snapshots via pg_dump into database/app/schema.snapshot.sql.
• Drift detection scripts: pnpm db:check (runs app + engine drift checks).

Betbot

• Has its own PostgreSQL pool and migration logic in services/betbot/src/db/migrate.ts.

General Access

• Use scripts/db-query.sh to run SQL against the prediction_market database via a temporary kubectl-run postgres pod.
• scripts/psql-prediction is another helper for direct psql access.

13. Commit & Pull Request Guidelines

• Commits: Short, imperative summaries. Group related changes.
  • Examples: bootstrap: patch cert-manager sources, feat(payments): add Tron withdrawal queue, fix(lmsr): clamp b parameter
• PRs: Detail scope, testing evidence, and impacted components. Link issues/epics when available.
• Evidence: Include Kubernetes command output or screenshots when work affects cluster state or UI.
• Manifest changes: Always include kubectl apply --dry-run=client output or yamllint clean result.

14. Critical Operational Notes

What Breaks Things

1. Manual kubectl/helm on ArgoCD-managed resources → Will be reverted by ArgoCD self-heal and creates sync loops.
2. Forgetting Traefik RE2 regex → Using unsupported regex syntax in IngressRoute causes 404s for all routes (incident 2025-10-21).
3. Unsafe Monero shutdown → Pulling a monerod pod without the 300s graceful termination + preStop hook risks LMDB corruption.
4. Unsealed Vault → If Vault seals, External Secrets Operator stops syncing secrets and apps may fail to start.
5. talos-cp-3 workload scheduling → This node is consumer hardware; workloads should not be scheduled here (use node affinity to exclude it).

Quick References

• ArgoCD UI: Accessible via Cloudflare Tunnel (check kubernetes/apps/platform/networking/cloudflare-tunnel/).
• Traefik Dashboard: Check Traefik IngressRoute for dashboard access.
• Vault Unseal: vault operator unseal (keys in vault-init-keys-backup.json).
• Cluster Health: talosctl -n 10.10.0.11 health --server