Tracing and observability

Every stage of a request - signal extraction, policy filtering, ranking, planning, execution, learning, health, cache, guard, shadow rollout, fair share and the autopilot - emits spans and events into a Tracer. Sinks decide what happens to them: keep the last N in memory for /trace, aggregate them into Prometheus counters, log them as JSON lines, append them to a file or forward them to OpenTelemetry. Instrumentation is always on and free when no sink is attached (a shared no-op span, no allocation), so the SDK stays zero-dependency and the cost of capturing everything is opt-in.

from opensmartroute import Router, configure_tracing
from opensmartroute.observability import MemorySink, MetricsSink

tracer = configure_tracing(MemorySink(), MetricsSink())      # process-wide; or configure_tracing() -> settings
router = Router(reg)                                          # picks up the global tracer
d = router.route("Prove that sqrt(2) is irrational")
for e in tracer.find(MemorySink).trace(d.request_id):        # spans + events of that request, in time order
    print(e["kind"], e["name"], e.get("duration_ms"), e["attributes"])
print(tracer.find(MetricsSink).prometheus())                 # osr_* exposition text

span  route            2.31ms {'text_sha256': '9f2c…', 'text_len': 35, 'target': 'large', 'confidence': 0.82, 'elapsed_ms': 2.1, ...}
event route.signals          {'duration_ms': 0.4, 'complexity': 0.71, 'domains': ['math'], 'reasoning_need': 0.9, 'contains_pii': False, ...}
event route.policy           {'pool': 2, 'admissible': 2, 'rejections': {}}
event route.rank             {'role': None, 'candidates': 2, 'strategies': {'capability': 0.05, ...}, 'top': [{'id': 'large', 'utility': 0.82, ...}]}

osr route "text" --events prints the same trace after the decision; osr serve exposes it over HTTP and the hosted platform shows it per workspace in the dashboard (activity drawer, /platform/dashboard/events, playground Trace tab - see the platform endpoints below).

What is captured#

Span	Opened by	Attributes (end of span)
request	EnterpriseRouter.route	middleware (class names), tenant, target, confidence, audited
http.request	osr serve middleware	method, path, app, status_code, target, inbound traceparent
route	Router.route	text_sha256, text_len, tenant, objective, kinds, candidates, exclude, plan, task_id, session_id, target, kind, confidence, elapsed_ms, alternatives, abstain, plan_slots
plan	Router._build_plan (inside route)	primary, kind, slots
execute	execution.execute / aexecute	target, kind, plan_slots, task_id, runner, ok, latency_ms, cost_usd, tokens, outcomes, system_prompt_len, pii_redacted, pii_restored
autopilot.cycle	Autopilot.run_once	reason, online, accepted, outcome, cycle

Event	Emitted when
route.signals	signals extracted: duration_ms, complexity, domains, actions, language, token estimate, PII, jailbreak risk, tools, task type, reasoning need
route.policy	hard constraints applied: pool, admissible, rejections (target -> reason)
route.no_route	no admissible target (before NoRouteError is raised)
route.pinned	a rule pinned a target (kept tells whether it survived the policy)
route.narrowed	the retriever shortlisted a large catalogue
route.shortlist	a strategy shortlisted candidates
route.rank	strategies scored (strategies with per-strategy ms, top 3, excluded_by_floor)
route.escalate	the LLM judge was consulted because confidence was low
route.explore	the bandit chose to explore (served tells whether the exploration was taken)
route.abstain	the calibrated confidence was below the abstention threshold
route.fallback	a fallback target replaced an unavailable one
plan.slot	a plan slot was filled (or not: filled=False, reason)
execute.step	one execution step (persona, skill, primary) finished
learn.outcome	Router.learn / learn_correction recorded an outcome (corrected, learner)
learn.task_credit	task-level credit assigned to the plan's targets
learn.calibrate	the confidence calibrator was refit (temperature)
learn.improve	a SelfImprover cycle finished (rows, champion_accuracy, challenger_accuracy, accepted, reason; inside an autopilot.cycle span when the autopilot ran it)
learn.promote	a challenger SLM replaced the serving champion (rows, trained_at, targets, saved)
health.breaker	a circuit breaker changed state (previous, state; level warning when it opens)
cache.hit / cache.miss	CacheMiddleware lookup (target, age_s on a hit)
tenant.rejected	TenantMiddleware refused a request (reason, tenant)
route.slow	TimeoutMiddleware saw a decision exceed its budget
guard.blocked / guard.flagged / guard.redacted	GuardMiddleware verdicts (reasons, gadget, entities)
shadow.compare / shadow.verdict	ShadowMiddleware compared production and candidate; the A/B test reached a verdict
fairshare.throttled / fairshare.capped	FairShareMiddleware delayed or refused a tenant
autopilot.drift	the autopilot detected quality drift and scheduled a cycle

opensmartroute.observability.SPAN_NAMES and EVENT_NAMES list the vocabulary; a test asserts every emitted name belongs to it.

Privacy. Request text never enters a span or event: the route span carries text_sha256 (16 hex characters) and text_len (text_digest()); attribute strings are truncated to OSR_OBSERVABILITY_ATTRIBUTE_MAX_LEN (500) and nested values are JSON-cleaned. Sinks receive ids, numbers, target ids, reasons and scores.

Spans, events, trace ids#

An Event is a flat record: name, kind (event or span), ts, trace_id, span_id, parent_id, request_id, level, status and duration_ms (spans only) and attributes. A span's closing record is the same shape as an event so every sink handles one type.

The trace id of a request is derived once, at the root span: an inbound W3C traceparent wins, otherwise a UUID request id is reused, otherwise the request id is hashed (so a request_id supplied by the caller yields a stable trace id). Nested spans inherit trace id, request id and the sampling decision through a contextvars context, so threads and asyncio tasks spawned by the router carry the right parent. Tracer.current() returns the innermost open span; current_tracer() resolves the tracer to report to (the open span's, else one bound with use_tracer, else the global one), which is how middleware, the health registry and the executor emit without holding a reference.

Sinks#

Sink	Purpose
MemorySink(max_events=1000)	ring buffer; events(request_id=, trace_id=, name=, kind=, level=, limit=) (name may end with *), trace(request_id); backs /events, /trace/{id} and osr route --events
MetricsSink()	counters and p50/p95/p99 per span name; snapshot() for /stats, prometheus() for /metrics (osr_events_total, osr_spans_total, osr_span_duration_ms, osr_span_decisions_total, osr_executions_total, osr_execution_cost_usd_total, osr_learn_outcomes_total, osr_breaker_transitions_total, osr_cache_events_total, osr_span_errors_total)
LoggingSink(log=None)	one JSON line per event on the opensmartroute.events logger, level follows the event
FileSink(path)	append-only JSONL, one event per line (ship with any log forwarder)
OpenTelemetrySink() (adapters.optional, otel extra)	live bridge: each span becomes an OTel span (parented through the OTel context, joined to traceparent), each event a span event and an osr.events counter, durations an osr.span_duration_ms histogram; the host configures the SDK, exporter and resource

Write your own by subclassing EventSink and implementing emit(event) (optionally span_start, span_end, snapshot). A sink that raises is counted in Tracer.sink_errors and logged (first ten occurrences); it never breaks routing.

Configuration#

configure_tracing(*sinks, sample_rate=None) installs sinks on the process-wide tracer that every Router uses by default. With no sinks it reads ObservabilitySettings (OSR_OBSERVABILITY_*):

Setting	Env	Default	Effect
metrics	OSR_OBSERVABILITY_METRICS	true	attach a MetricsSink
memory_events	OSR_OBSERVABILITY_MEMORY_EVENTS	1000	attach a MemorySink of that size (0 disables)
log_events	OSR_OBSERVABILITY_LOG_EVENTS	false	attach a LoggingSink
events_file	OSR_OBSERVABILITY_EVENTS_FILE	""	attach a FileSink at that path
otel	OSR_OBSERVABILITY_OTEL	false	attach an OpenTelemetrySink (needs the otel extra)
sample_rate	OSR_OBSERVABILITY_SAMPLE_RATE	1.0	share of root spans that are recorded
attribute_max_len	OSR_OBSERVABILITY_ATTRIBUTE_MAX_LEN	500	longest attribute string kept

Router(reg, tracer=Tracer([...])) gives one router a private tracer; RouterBuilder(...).with_tracing(*sinks_or_tracer, sample_rate=) does the same for an EnterpriseRouter (no arguments: build from settings). osr, osr serve, the Docker image and the hosted platform call configure_tracing() / Tracer.from_settings() at start-up, so the environment variables above are all that is needed in a container.

HTTP endpoints (osr serve)#

Endpoint	Returns
GET /metrics	MetricsTelemetry exposition followed by the tracer's MetricsSink counters (names are disjoint)
GET /events?request_id=&trace_id=&name=route.*&kind=&level=&limit=200	recent events from the memory buffer (404 when OSR_OBSERVABILITY_MEMORY_EVENTS=0)
GET /trace/{request_id}	every span and event of one request, in time order, with its trace_id
GET /stats	adds an observability block: sinks, sample rate, sink errors and each sink's snapshot

Every traced response carries X-OSR-Trace-Id; a client that sends traceparent gets its own trace id back, so the routing spans line up with the caller's trace in Jaeger, Tempo, Azure Monitor or any OTel backend. Probes (/healthz, /readyz), /metrics, /events, /trace/*, /whoami and the OpenAPI pages are never traced. The endpoints follow the server's access control: with OSR_SERVER_AUTH_TOKENS set, /events, /trace/* and /stats need a token (/metrics stays open for scrapers).

HTTP endpoints (hosted platform)#

The platform API (platform/api) exposes the same tracer per workspace, so a tenant only ever sees its own requests - and it is the platform's own observability: it persists every span and event to its database (osr_platform.telemetry, OSR_PLATFORM_TELEMETRY_STORE, on by default, off the request path through a queued writer; OSR_PLATFORM_TELEMETRY_RETENTION_DAYS), keeps per-minute HTTP counters for a day, computes time series from its usage records and evaluates alert rules in-process (osr_platform.alerts). Nothing has to be exported to a metrics, tracing or alerting system.

Endpoint	Returns
GET /api/v1/trace/{request_id}	the activity row of the request plus its spans and events (from the buffer, or from the store once evicted - source says which), the feedback reported for it (outcomes) and its audit records (audit); 404 for another workspace's request
GET /api/v1/events?name=&kind=&level=&request_id=&since=&until=&limit=	events joined to the workspace through its request ids (deployment-wide events on plans with stats); since / until select a time range from the store (default the last 6 hours)
GET /api/v1/telemetry/series?window=	the workspace's requests, failures, p50 / p95 latency, cost and tokens per bucket (1h .. 30d) with per-target and per-endpoint totals; http (plans with stats) adds the deployment's requests, 4xx / 5xx and latency percentiles per bucket
GET /api/v1/alerts	active conditions - budgets, failing requests, and on plans with stats readiness, 5xx rate, latency SLO, breakers, drift, autopilot errors, a stale SLM, tracing off - each with the dashboard page that shows or fixes it
GET /api/v1/status	public readiness with the tracing state, for status pages (503 while a check fails)
GET /api/v1/activity	request log with traced: true on rows whose spans are kept and outcome summarising the feedback per request

POST /api/v1/route returns trace_id and X-OSR-Trace-Id. The dashboard renders the history at /platform/dashboard/events (1 hour to 7 days), a span waterfall from every traced activity row and a Trace tab in the playground; the overview shows the 24-hour series and the alerts, /platform/dashboard/health the deployment series, the Systems card (/api/v1/status) and the same alerts.

OpenTelemetry end to end#

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))

from opensmartroute import configure_tracing
from opensmartroute.adapters.optional import OpenTelemetrySink

configure_tracing(OpenTelemetrySink())      # or OSR_OBSERVABILITY_OTEL=true

OTel spans are named osr.route, osr.plan, osr.execute, ...; attributes carry the osr. prefix (osr.target, osr.confidence, osr.request_id, osr.trace_id). OpenTelemetryTelemetry (the older Telemetry port) still records the osr.route_latency_ms histogram per decision; the sink is the richer, per-stage view and the two can run side by side.

Relationship to telemetry and audit#

Telemetry sinks (LoggingTelemetry, MetricsTelemetry, OpenTelemetryTelemetry) observe decisions and outcomes on an EnterpriseRouter; AuditSink keeps a hash-chained, tamper-evident record of who got what. The tracer observes how the decision was made - every stage, with timings, on plain and enterprise routers alike. Use audit for compliance, telemetry for dashboards you already have and the tracer for debugging, SLOs per stage and distributed tracing. RouteTrace.explain() remains the human-readable explanation attached to a RouteDecision.

MetricsTelemetry is the dashboard store: snapshot() returns decisions_total, decisions_by_target, policy_rejections_total, errors_by_type, route_latency_ms (p50, p95, p99, mean, count), route_latency_histogram_ms, mean_confidence, outcomes_by_target (count, success rate, cost, quality, latency) and uptime_s; prometheus(namespace="osr", extra={...}) renders the same as Prometheus text (osr_route_decisions_total{target="..."}, osr_route_latency_ms_bucket{le="..."}, osr_outcome_cost_usd_total, osr_decision_cache_hits_total, extras as gauges). osr serve and the hosted platform expose it at GET /metrics; RouterBuilder.with_cache(ttl_s, max_size) adds a decision cache whose CacheMiddleware.stats() feed the osr_decision_cache_* series, and with_audit(sink, outcomes=True) chains outcomes into the audit trail alongside decisions.