Imported from Nyxonios/dotfiles (.config/opencode/skills/gcx-metrics-logs/SKILL.md). Install upstream with npx skills add Nyxonios/dotfiles --skill gcx-metrics-logs. Copyright stays with the author (MIT).

Skill: gcx-metrics-logs

Query metrics and logs from the terminal using gcx, the Grafana Cloud CLI.

Arguments: $ARGUMENTS (e.g., a LogQL / PromQL expression, service name, grep term, or a natural-language request like "show me 5xx errors for nginx in the last hour")

⚠️ CRITICAL: Always use --agent flag

This skill is consumed by an AI agent (Sisyphus / OpenCode). Every gcx command MUST include the --agent flag to ensure JSON-only output, no interactive spinners, no color codes, and no terminal UI prompts. Without --agent, output may be corrupted by ANSI escape sequences or TUI elements, making it impossible for the agent to parse.

Default behaviour: All examples below include --agent. When formulating any new command, append --agent immediately after gcx and before the subcommand, e.g.:

gcx --agent metrics query 'up' --since 5m

Overview

This skill uses the gcx CLI (installed at /usr/local/bin/gcx) to query:

• Metrics via the Metrics Platform (Thanos / Prometheus) — the default Prometheus datasource
• Logs via Loki v2 — the primary log aggregation backend

gcx is pre-configured on this machine with a default context connecting to https://monitor.evroc.dev.

Data Sources

You typically do not need to pass --datasource explicitly because gcx uses the correct default for the signal you query (metrics vs logs).

Authentication

gcx is already authenticated via a service-account token stored in ~/.config/gcx/config.yaml. No login is required.

If you ever need to verify:

gcx config check          # verify token is valid
gcx config view            # show current context
gcx config list-contexts   # show all contexts

Key Commands

1. Query Metrics (PromQL)

# Basic instant query
gcx --agent metrics query 'up'

# Range query — last 1 hour with 15s step
gcx --agent metrics query 'rate(http_requests_total[5m])' --since 1h --step 15s

# Visualise as an in-terminal graph
gcx --agent metrics query 'sum(rate(node_cpu_seconds_total[5m])) by (mode)' --since 1h -o graph

# JSON output for piping to jq
gcx --agent metrics query 'up{job="node-exporter"}' --since 30m -o json

# Query a specific datasource explicitly
gcx --agent metrics query -d prometheus 'up' --since 5m

2. Query Logs (LogQL)

# All logs for a service in the last 30 minutes
gcx --agent logs query '{app="nginx"}' --since 30m

# Logs matching a text substring (case-sensitive, ~grep-like)
gcx --agent logs query '{app="api"} |= "error"' --since 1h

# Exclude noisy health-check lines
gcx --agent logs query '{app="api"} != "/healthz"' --since 1h

# Regex match
gcx --agent logs query '{app="api"} |~ "(error|panic|fatal)"' --since 1h

# JSON log field extraction and filter
gcx --agent logs query '{app="api"} | json | level="error"' --since 1h

# Limit number of lines (default 50, max 5000)
gcx --agent logs query '{app="ingress"} |= "500"' --since 15m --limit 200

# Live tail (if supported by datasource)
gcx --agent logs query '{app="myapp"}' --since 1m --follow

3. Discover Labels & Streams

# List all metric labels
gcx --agent metrics labels

# Values of a specific label
gcx --agent metrics labels -l job

# List all log stream labels
gcx --agent logs labels

# Values of a log label
gcx --agent logs labels -l app

# Find active log streams for a selector
gcx --agent logs series --match '{app="nginx"}'

4. Explore Datasources

# List all configured datasources
gcx --agent datasources list

# Get details of a datasource
gcx --agent datasources get loki-v2

5. Use the Knowledge Graph (kg)

Grafana Knowledge Graph (kg) knows about our telemetry topology:

# List schema / scopes / known telemetry configs
gcx --agent kg describe

Common Query Patterns for Our Infrastructure

Node / Host Metrics

# CPU utilisation by mode
gcx --agent metrics query 'sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance, mode)' --since 1h

# Memory utilisation %
gcx --agent metrics query '(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100' --since 1h

# Disk space warning (< 10 % free)
gcx --agent metrics query 'node_filesystem_avail_bytes / node_filesystem_size_bytes * 100 < 10' --since 5m

# High load
gcx --agent metrics query 'node_load1 > 4' --since 10m

Kubernetes / Container Metrics

# Pod restart rate
gcx --agent metrics query 'sum(rate(kube_pod_container_status_restarts_total[10m])) by (namespace, pod)' --since 1h

# Container CPU throttling
gcx --agent metrics query 'rate(container_cpu_cfs_throttled_seconds_total[5m])' --since 30m

# High memory usage containers
gcx --agent metrics query 'container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.85' --since 10m

API / HTTP Error Patterns

# 5xx rate per handler
gcx --agent metrics query 'sum(rate(http_requests_total{status=~"5.."}[5m])) by (handler)' --since 1h

# 95th percentile latency
gcx --agent metrics query 'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, handler))' --since 1h

# Error logs for an API service
gcx --agent logs query '{app="api-gateway"} |= "error"' --since 30m

# Combined: logs for a specific trace ID
gcx --agent logs query '{app="api-gateway"} |= "trace_id=abc123"' --since 1h

Storage / Ceph Patterns

# Ceph health
gcx --agent metrics query 'ceph_health_status' --since 5m

# High bucket usage bytes (from metrics proxy whitelist)
gcx --agent metrics query 'max_bucket_usage_bytes' --since 1h

Output Formats

Flags Reference

Time Ranges

• --since 5m → last 5 minutes (quick health check)
• --since 15m → last 15 minutes (recent incidents)
• --since 1h → last hour (standard investigation window)
• --since 24h → last 24 hours (daily trends)
• --since 7d → last 7 days (weekly patterns)

You can also use absolute times:

gcx metrics query 'up' --from 2026-06-01T08:00:00Z --to 2026-06-01T09:00:00Z

Troubleshooting

LogQL Quick Reference

Before

PromQL Quick Reference

Notes

• Read-only: This skill only runs gcx query / gcx labels / gcx series / gcx describe. It never runs gcx create, gcx delete, gcx edit, or gcx resources push.
• Agent flag is mandatory: Because this skill is invoked by an AI agent, every gcx command must include the --agent flag immediately after gcx (before the subcommand). This suppresses interactive TUI elements, spinners, color codes, and ensures clean JSON/machine-readable output. Without it, parsing may fail.
• Datasource defaults: gcx automatically uses the configured default datasource for the signal type (metrics → Prometheus/Thanos, logs → Loki v2). You only need -d when querying a non-default datasource.
• Context: If multiple contexts exist, gcx uses current-context from ~/.config/gcx/config.yaml. Check with gcx --agent config view.
• Performance: Very broad queries (e.g. {app!=""} over 24h) can be expensive and slow. Always scope with specific labels and narrow --since windows first.
• Label discovery: If you don't know the exact label names, use gcx --agent logs labels or gcx --agent metrics labels first, then refine.
• Namespaces: Many Kubernetes metrics include a namespace label; use it to scope queries if the cluster has many tenants.