Qyra

Configure OpenTelemetry tracing for self-hosted Qyra

Export distributed traces from your Qyra instance to any OpenTelemetry-compatible backend

🛠 This page is for engineering teams self-hosting their own Qyra instance. For instance health metrics, see Prometheus metrics.

Qyra can export distributed traces using the OpenTelemetry SDK, so you can follow a request or scheduled job across the API server, scheduler, and warehouse queries in any OpenTelemetry-compatible backend (for example Grafana Tempo, Jaeger, Honeycomb, or Datadog).

Tracing runs in one of two exclusive modes:

  • Sentry mode (default): spans are created and exported through Sentry, controlled by the Sentry environment variables.
  • OpenTelemetry mode: spans are created by the OpenTelemetry SDK and exported according to the standard OTEL_* environment variables. Sentry still captures errors, but receives no spans.

Enabling OpenTelemetry tracing

By default, Qyra traces through Sentry. To switch to OpenTelemetry mode, set the following environment variable on every Qyra container (API server and scheduler):

QYRA_OTEL_TRACES_ENABLED=true

Then point the OTLP exporter at your collector:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

Configuration options

VariableDescription
QYRA_OTEL_TRACES_ENABLEDSwitches tracing to OpenTelemetry mode: spans are created by the OpenTelemetry SDK and Sentry receives errors only. When false, tracing runs through Sentry. (default=false)
QYRA_OTEL_TRACES_SAMPLE_RATETrace sampling ratio from 0.0 to 1.0. Falls back to SENTRY_TRACES_SAMPLE_RATE (deprecated) when unset. (default=1)
QYRA_OTEL_ALWAYS_SAMPLE_AI_TRACESSet to false to stop always-sampling AI agent traces, so they follow the global sampling ratio instead. (default=true)
QYRA_OTEL_DB_TRACES_ENABLEDAdds spans for application database (Postgres) queries to traces, including the SQL statement. Requires OpenTelemetry mode. (default=false)
QYRA_OTEL_DB_TRACES_MAX_QUERY_LENGTHMaximum length of the SQL statement recorded on database spans; longer statements are truncated. Must be a non-negative integer. (default=1022)
OTEL_SDK_DISABLEDStandard OpenTelemetry kill switch. When true, OpenTelemetry mode is off even if QYRA_OTEL_TRACES_ENABLED=true.
OTEL_TRACES_EXPORTERComma-separated exporters: otlp, console, zipkin, or none. Unsupported values are ignored with a startup warning; none overrides any other value. (default=otlp)
OTEL_EXPORTER_OTLP_PROTOCOLOTLP protocol: grpc, http/json, or http/protobuf. OTEL_EXPORTER_OTLP_TRACES_PROTOCOL takes precedence. Unsupported values fall back to the default. (default=http/protobuf)
OTEL_EXPORTER_OTLP_ENDPOINTBase URL of your OTLP collector. Handled by the OpenTelemetry Node SDK, along with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
OTEL_SERVICE_NAMEService name attached to exported spans. (default=qyra)
OTEL_LOG_LEVELEnables OpenTelemetry SDK and exporter diagnostics at the given level (e.g. DEBUG). Qyra redacts credentials, tokens, and span payloads from diagnostic output.

OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG are overridden by Qyra. Control sampling with QYRA_OTEL_TRACES_SAMPLE_RATE instead.

Choosing an exporter

Qyra defers exporter and protocol selection to the OpenTelemetry Node SDK's standard environment variable handling:

  • OTEL_TRACES_EXPORTER supports otlp (default), console, zipkin, and none. You can combine exporters with a comma-separated list. If the list contains none, no traces are exported regardless of the other values.
  • With the otlp exporter, OTEL_EXPORTER_OTLP_PROTOCOL (or the traces-specific OTEL_EXPORTER_OTLP_TRACES_PROTOCOL) selects grpc, http/json, or http/protobuf (default).
  • Unsupported exporter or protocol values are ignored with a warning in the Qyra logs at startup.

For example, to export traces over gRPC with an authentication header:

QYRA_OTEL_TRACES_ENABLED=true
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com:4317
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <token>"

Sampling

QYRA_OTEL_TRACES_SAMPLE_RATE sets the head-sampling ratio for trace roots, from 0.0 (nothing) to 1.0 (everything, the default). Child spans follow their root's decision, so a sampled request captures the whole waterfall.

Two behaviours to be aware of:

  • AI agent traces are always sampled regardless of the ratio, so a broken agent run always has a trace to debug. Set QYRA_OTEL_ALWAYS_SAMPLE_AI_TRACES=false to make them follow the global ratio instead.
  • OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG are overridden by Qyra's own sampler and have no effect.

Requests to health checks (/api/v1/health, livez), status polling endpoints, favicon.ico, and robots.txt are never traced.

Database query tracing

Set QYRA_OTEL_DB_TRACES_ENABLED=true to add a span for each application database (Postgres) query, so you can see where a request spends time inside Qyra's own database:

QYRA_OTEL_TRACES_ENABLED=true
QYRA_OTEL_DB_TRACES_ENABLED=true
  • Database spans only appear inside an existing trace, so they follow the sampling decision of their parent request or job.
  • Each span records the SQL statement, truncated to QYRA_OTEL_DB_TRACES_MAX_QUERY_LENGTH characters (default 1022). Invalid values fall back to the default with a warning.
  • This traces queries to Qyra's application database only, not queries sent to your data warehouse.

Troubleshooting

At startup, Qyra logs a line confirming the tracing configuration, including the active exporters, OTLP protocol, and sampling ratio. This confirms configuration only; exporter connectivity is not validated at startup.

If traces aren't arriving in your backend, set OTEL_LOG_LEVEL=DEBUG to enable OpenTelemetry SDK and exporter diagnostics in the Qyra logs. Diagnostic output is redacted before logging: URL credentials and query strings are stripped, values containing tokens or API keys are omitted, and span payloads are not printed.

Metrics

This page covers traces only. For instance health metrics (CPU, memory, event loop, query durations), see Prometheus metrics, which can also feed an OpenTelemetry backend through the collector's Prometheus receiver.