Skip to main content

OpenTelemetry and Custom Metrics

Send application metrics and traces to SailOps with OTLP.

SailOps can inject OpenTelemetry configuration into compatible workloads or provide the environment variables you need for manual setup. Telemetry is scoped to the organization, project, and workload so it appears in the right Grafana dashboards and usage reports.

Supported workloads​

The Monitoring Center currently discovers OpenTelemetry-ready connections for:

  • containers
  • jobs
  • cronjobs

Other resources still appear in built-in infrastructure metrics and logs, but automatic OpenTelemetry connection guidance is limited to these workload types.

Enable project observability​

Project observability injects SailOps-managed OTLP environment variables into compatible workloads in a project.

  1. Open Monitoring.
  2. Go to Project observability.
  3. Select a project.
  4. Click Enable.

After activation, the page shows each project's sync status and workload coverage:

StatusMeaning
EnabledSailOps will manage OpenTelemetry variables for the project
SyncedCompatible workloads have the expected variables
PartialSome workloads were configured and others need attention
ErrorSailOps could not sync one or more workloads
Missing tokenThe workload's ingestion secret is missing or incomplete — resync or rotate the project token to restore it
Legacy tokenThe workload still uses an older token and should be rotated or resynced

Injected environment​

SailOps injects these variables:

VariableSecretDescription
OTEL_EXPORTER_OTLP_ENDPOINTNoSailOps OTLP HTTP endpoint
OTEL_EXPORTER_OTLP_HEADERSYesAuthorization header with the ingestion bearer token
OTEL_RESOURCE_ATTRIBUTESNoSailOps labels for organization, project, and workload
OTEL_SERVICE_NAMENoWorkload name used as the OpenTelemetry service name
caution

Treat OTEL_EXPORTER_OTLP_HEADERS as a secret. The UI masks it after creation.

Manual setup​

Use manual setup when a workload cannot use project-level injection or when you want to configure an external OpenTelemetry collector.

  1. Open Monitoring.
  2. Go to Project observability.
  3. Open Project env or a workload's environment modal.
  4. Copy the non-secret variables and configure the secret token through your secret manager.
  5. Configure your OpenTelemetry SDK or collector to export OTLP over HTTP.

The authorization header must use a bearer token:

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <INGESTION_TOKEN>"

Token rotation and revocation​

Maintainers can rotate or disable project observability from the Project observability page.

ActionEffect
ResyncReapplies the expected OpenTelemetry environment to compatible workloads. If any workload's ingestion secret is missing or holds a revoked token, the resync automatically rotates the project token.
RotateCreates a replacement ingestion token and resyncs compatible workloads
DisableRemoves SailOps-managed observability injection on resync
tip

If a workload shows the Missing token status, use Resync to restore it. The resync detects missing or revoked credentials and automatically provisions a new ingestion token when needed.

What appears in Grafana​

Custom metrics and traces appear in managed Grafana after your application emits OTLP data. Traces with recent activity are also listed under Monitoring → Observability, ordered by start time. Metric names, labels, and resource attributes come from your OpenTelemetry SDK or collector configuration, plus the SailOps resource attributes injected by the platform.

If dashboards are empty:

  • confirm Grafana is Ready
  • confirm the project or workload is Configured
  • resync the project if any workload shows Missing token
  • verify your application exports OTLP metrics or traces
  • check that the workload uses the SailOps endpoint and bearer header

Usage and quotas​

SailOps tracks observability usage for metric samples, trace spans, active series, and retained telemetry storage. You can view usage under Monitoring → Usage → Telemetry.

The Usage page shows:

  • metric samples ingested
  • trace spans ingested
  • retained storage in GiB-month
  • active series average, p95, and maximum
  • included quota for your plan
  • current and projected overage
  • per-project usage breakdown