Instruction file imported from anthr76/infra (.github/instructions/.instructions.md). Copyright stays with the author.

Infrastructure Repository Instructions

This repository contains ### Kubernetes & GitOps

• Follow GitOps principles: all changes via Git commits
• Use Kustomize for configuration management, avoid raw YAML duplication
• Namespace Organization: Each directory under k8s/base/ maps 1:1 to a Kubernetes namespace (e.g., k8s/base/volsync-system/ → volsync-system namespace)
• Helm Repositories: All HelmRepository resources belong in k8s/base/flux-system/helm-chart-repositories/
• Secret Management: Use existing ClusterSecretStore (gcp-kutara-prod) via External Secrets, never commit secrets
• Implement proper resource requests/limits for all workloads
• Use NetworkPolicies for security isolation
• Apply consistent labeling: app.kubernetes.io/name, app.kubernetes.io/component
• Use HelmReleases with versioned charts, avoid latest tags
• VolSync Backup Strategy: Use VolSync ReplicationSource for PVC backups to rsync.net with proper scheduling
• Implement monitoring for all custom applications
• Dependency Management: All container images, Helm chart versions, and dependencies should be managed by Renovate
• Pin specific versions for container images and Helm charts to enable automated updates via Renovate PRsete infrastructure-as-code for a sophisticated home lab environment using GitOps principles. It includes Terraform for infrastructure provisioning, Kubernetes configurations, and Flux for continuous delivery.

Repository Overview

This is a multi-environment, multi-cloud infrastructure repository that includes:

• Infrastructure-as-Code (IaC): Terraform configurations for cloud resources, DNS, and virtualization
• GitOps: Flux CD for Kubernetes application deployment and management
• Multi-Cloud: Google Cloud Platform (GCP), Cloudflare, Backblaze B2, DigitalOcean
• Virtualization: Libvirt/QEMU for on-premises VMs running Fedora CoreOS
• Kubernetes: Multiple clusters with comprehensive workload management
• Secret Management: SOPS for encrypted secrets, External Secrets for Kubernetes

Related Repositories:

• System Configuration: anthr76/snowflake - Contains system-level configuration for nodes, routers, and infrastructure components## Directory Structure

• armature/prod/ - Production infrastructure Terraform configurations

  • cloud-dns/ - Cloudflare DNS management
  • gcp/ - Google Cloud Platform resources (bootstrap, projects, GKE)
  • scr1/ - On-premises site resources and VMs
  • nwk1/ - Network site resources
  • b2/ - Backblaze B2 storage buckets

• k8s/ - Kubernetes manifests organized by namespace and cluster

  • base/ - Base Kustomize configurations
  • clusters/ - Cluster-specific overlays

• docs/ - Documentation for the infrastructure setup

Technology Stack

Infrastructure & Cloud

• Terraform: Infrastructure provisioning and state management
• Terraform Cloud: Remote state management (organization: "kutara", "rabbito-home")
• Google Cloud Platform: Cloud infrastructure, GKE, Secret Manager
• Cloudflare: DNS management and CDN
• Backblaze B2: Object storage for backups
• Fedora CoreOS: Immutable OS for containers and Kubernetes nodes
• Libvirt/QEMU: Virtualization for on-premises infrastructure
• System Configuration: Node and router configurations managed in anthr76/snowflake

Development & Automation

• Flox: Nix-based development environment for consistent tooling across contributors
• Justfile: Task automation and command standardization (preferred over shell scripts)
• Renovate: Automated dependency management for container images, Helm charts, and Terraform modules

Kubernetes & GitOps

• Flux CD: GitOps continuous delivery operator
• Kustomize: Kubernetes configuration management
• Helm: Package management via HelmReleases
• SOPS: Secrets encryption with age/PGP keys
• External Secrets: Kubernetes secret synchronization
• MetalLB: Load balancer for bare metal Kubernetes
• Cilium: Container networking (CNI)
• Rook Ceph: Distributed storage
• VolSync: Asynchronous PVC replication for backup and disaster recovery

Applications & Services

• Home Automation: Home Assistant, ESPHome, Zigbee2MQTT, Z-Wave JS UI
• Media Services: Plex ecosystem, cross-seed, qBittorrent
• Monitoring: Prometheus, Grafana, Alert Manager
• Networking: Unifi Controller, pfSense
• Databases: PostgreSQL (CloudNative-PG), Redis, CouchDB
• Backup & Replication: VolSync for PVC backup/restore to external storage (rsync.net)

Coding Standards & Guidelines

Terraform

• Use consistent resource naming: {service}-{environment}-{resource_type}
• Always specify provider versions in required_providers blocks
• Use data sources for external references (e.g., SOPS files)
• Implement proper variable validation and descriptions
• Use terraform fmt and follow HashiCorp style guidelines
• Store state remotely in Terraform Cloud
• Use SOPS for sensitive variables (*.sops.yaml files)
• Module Versioning: Pin Terraform module versions for Renovate to manage updates
• Avoid using latest or branch references for module sources

Kubernetes & Flux

• Follow GitOps principles: all changes via Git commits
• Use Kustomize for configuration management, avoid raw YAML duplication
• Implement proper resource requests/limits for all workloads
• Use NetworkPolicies for security isolation
• Apply consistent labeling: app.kubernetes.io/name, app.kubernetes.io/component
• Use HelmReleases with versioned charts, avoid latest tags
• Implement monitoring for all custom applications
• Use External Secrets for secret management, never commit secrets

File Naming & Organization

• Terraform files: main.tf, variables.tf, providers.tf, outputs.tf
• Kubernetes Namespace Mapping: Each directory under k8s/base/ must map 1:1 to a Kubernetes namespace
  • Example: k8s/base/volsync-system/ contains resources for the volsync-system namespace
  • Example: k8s/base/home/ contains resources for the home namespace
• Helm Repositories: All HelmRepository CRs belong in k8s/base/flux-system/helm-chart-repositories/
• External Secrets: Use the existing gcp-kutara-prod ClusterSecretStore for all secret management
• Avoid Custom Scripts: Use Justfile for automation instead of shell scripts or custom tooling
• Development Environment: Use the Flox manifest (.flox/env/manifest.toml) for managing development tools and dependencies
• Use meaningful directory names that reflect their purpose
• Keep related resources grouped logically

Security Best Practices

• Never commit unencrypted secrets or sensitive data
• Use SOPS for secret encryption with appropriate key management
• Implement least-privilege IAM policies
• Use service accounts with minimal required permissions
• Enable audit logging for all critical resources
• Regularly rotate credentials and certificates

Automation & Scripting Guidelines

• Prefer Justfile: Use just commands over shell scripts for automation
• Avoid Custom Scripts: Custom shell scripts should be rare; use Justfile tasks instead
• Development Tools: Manage tool dependencies via the Flox manifest (.flox/env/manifest.toml), not system-level installation
• Task Documentation: All Justfile tasks should have clear descriptions and usage examples

Documentation

• Update relevant documentation for any infrastructure changes
• Include migration guides for breaking changes
• Document any manual intervention requirements
• Keep README files updated with current deployment procedures

Development Workflow

Development Environment Setup

This repository uses Flox (Nix-based development environments) to ensure consistent tooling across all contributors:

# Enter the development environment (installs all required tools)
flox activate

# Verify tools are available
just --version
kubectl version --client
flux version

Benefits of Flox:

• Consistent tool versions across all developers
• Automatic installation of required dependencies
• No need to manually install kubectl, flux, etc.
• Environment isolation prevents version conflicts

Development Environment File: .flox/env/manifest.toml defines all required tools and their versions. kubectl-volsync is a custom Go build under .flox/pkgs/kubectl-volsync/; run flox build kubectl-volsync once to compile it.

Dependency Management with Renovate

This repository uses Renovate for automated dependency management. All version updates should go through Renovate:

• Container Images: All image.tag values in HelmReleases and Kubernetes manifests
• Helm Chart Versions: Chart versions in HelmRelease spec.chart.spec.version
• Terraform Modules: Module version constraints in Terraform configurations
• GitHub Actions: Action versions in workflow files
• Base Images: Dockerfile FROM statements and container registry references

Renovate Guidelines:

• Never manually update versions that Renovate manages
• Review Renovate PRs for breaking changes and release notes
• Use semantic versioning constraints where appropriate
• Pin specific versions rather than using latest or floating tags
• Test Renovate updates in staging before merging to production

Making Changes

1. Environment Setup: Use flox activate to enter the development environment with all required tools
2. Create feature branch from main
3. Test changes in development/staging environment first
4. Use just commands for common operations (defined in Justfile) instead of custom scripts
5. Validate Terraform with terraform plan
6. Test Kubernetes changes with dry-run deployments
7. Update documentation as needed

Flux Operations

• Use kubectl with Flux annotations for manual reconciliation
• Monitor Flux controller logs for deployment status
• Use flux CLI for debugging GitOps issues
• Suspend/resume resources during maintenance windows

Tools & Commands

• just - Task runner with predefined infrastructure operations (preferred over custom scripts)
• flux - Flux CLI for GitOps operations
• kubectl - Kubernetes cluster interaction
• terraform - Infrastructure provisioning
• sops - Secret encryption/decryption
• kustomize - Configuration management
• Renovate - Automated dependency updates via pull requests for container images, Helm charts, and Terraform modules
• Flox - Development environment management with Nix for consistent tooling and dependencies

Environment-Specific Notes

Production Environments

• scr1 - Primary on-premises site with Kubernetes cluster
  • ⚠️ HIGH PRIORITY: scr1 should eventually be renamed to qg1 due to physical location change
  • This migration is complex due to extensive cross-references in DNS, Terraform states, Kubernetes configs, and certificates
  • Breaking change that requires careful planning and coordinated updates across the entire infrastructure
• nwk1 - Secondary network site
• All production changes require careful planning and staging

Cloud Resources

• GCP projects use random suffixes for uniqueness
• Terraform states are managed remotely
• Cross-cloud networking configured via Cloudflare

Backup Strategy

• Backblaze B2 for object storage backups
• VolSync: Persistent volume replication to rsync.net for off-site backup and disaster recovery
• Persistent volumes backed by Ceph storage
• Configuration backed up via Git repository

Repository Maintenance Notes

Dead Code Cleanup Required

This repository contains significant amounts of dead/commented code that needs cleanup:

• Terraform: Many .tf files contain large commented blocks (e.g., armature/prod/b2/buckets.tf, GKE configurations)
• Kubernetes: Unused manifest files and deprecated configurations
• Legacy Infrastructure: Old VM definitions, unused network configurations
• Commented Resources: Extensively commented-out Terraform resources that should be removed or uncommented

Guidelines for Dead Code Cleanup:

• Remove commented code blocks that are confirmed unused
• Archive old configurations to separate branches if needed for reference
• Document reasoning for any temporarily disabled resources
• Prioritize cleanup during major refactoring efforts
• Always verify dependencies before removing seemingly unused code

Critical Migration: scr1 → qg1

Status: High Priority, Breaking Change Required

The scr1 site designation needs to be migrated to qg1 due to physical infrastructure relocation. This affects:

• DNS Records: *.scr1.rabbito.tech domains and subdomains
• Terraform States: Remote state names and resource references
• Kubernetes Clusters: Cluster names and ingress configurations
• Certificates: TLS certificates tied to scr1 domains
• Monitoring: Grafana dashboards, Prometheus rules, alert configurations
• Documentation: All references in docs and README files
• Hostnames: VM hostnames and network configurations

Migration Strategy Needed:

1. Inventory all scr1 references across the repository
2. Plan DNS migration strategy with minimal downtime
3. Coordinate Terraform state migrations
4. Update Kubernetes cluster configurations
5. Regenerate certificates for new domains
6. Update monitoring and alerting rules
7. Comprehensive testing in staging environment

When making suggestions or changes, always consider the impact on the entire infrastructure ecosystem and follow the established patterns and conventions used throughout the repository. Be aware of the extensive dead code cleanup needed and the high-priority scr1→qg1 migration requirements.

Cross-Repository Dependencies

System Configuration Repository

For system-level configuration of nodes, routers, and infrastructure components, refer to:

• Repository: anthr76/snowflake
• Scope: Operating system configurations, network device settings, base system setup
• Relationship: The snowflake repository handles host-level configuration while this repository manages containerized applications and cloud infrastructure

When making infrastructure changes that affect system-level configurations (networking, storage, security policies), consider the impact on configurations managed in the snowflake repository.