OpenTelemetry and Custom Metrics
Send application metrics and traces to SailOps with OTLP.
SailOps can inject OpenTelemetry configuration into compatible workloads or provide the environment variables you need for manual setup. Telemetry is scoped to the organization, project, and workload so it appears in the right Grafana dashboards and usage reports.
Supported workloads
The Monitoring Center currently discovers OpenTelemetry-ready connections for:
- containers
- jobs
- cronjobs
Other resources still appear in built-in infrastructure metrics and logs, but automatic OpenTelemetry connection guidance is limited to these workload types.
Enable project observability
Project observability injects SailOps-managed OTLP environment variables into compatible workloads in a project.
- Open Monitoring.
- Go to Project observability.
- Select a project.
- Click Enable.
After activation, the page shows each project's sync status and workload coverage:
| Status | Meaning |
|---|---|
| Enabled | SailOps will manage OpenTelemetry variables for the project |
| Synced | Compatible workloads have the expected variables |
| Partial | Some workloads were configured and others need attention |
| Error | SailOps could not sync one or more workloads |
| Missing token | The workload's ingestion secret is missing or incomplete — resync or rotate the project token to restore it |
| Legacy token | The workload still uses an older token and should be rotated or resynced |
Injected environment
SailOps injects these variables:
| Variable | Secret | Description |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | No | SailOps OTLP HTTP endpoint |
OTEL_EXPORTER_OTLP_HEADERS | Yes | Authorization header with the ingestion bearer token |
OTEL_RESOURCE_ATTRIBUTES | No | SailOps labels for organization, project, and workload |
OTEL_SERVICE_NAME | No | Workload name used as the OpenTelemetry service name |
Treat OTEL_EXPORTER_OTLP_HEADERS as a secret. The UI masks it after creation.
Manual setup
Use manual setup when a workload cannot use project-level injection or when you want to configure an external OpenTelemetry collector.
- Open Monitoring.
- Go to Project observability.
- Open Project env or a workload's environment modal.
- Copy the non-secret variables and configure the secret token through your secret manager.
- Configure your OpenTelemetry SDK or collector to export OTLP over HTTP.
The authorization header must use a bearer token:
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <INGESTION_TOKEN>"
Token rotation and revocation
Maintainers can rotate or disable project observability from the Project observability page.
| Action | Effect |
|---|---|
| Resync | Reapplies the expected OpenTelemetry environment to compatible workloads. If any workload's ingestion secret is missing or holds a revoked token, the resync automatically rotates the project token. |
| Rotate | Creates a replacement ingestion token and resyncs compatible workloads |
| Disable | Removes SailOps-managed observability injection on resync |
If a workload shows the Missing token status, use Resync to restore it. The resync detects missing or revoked credentials and automatically provisions a new ingestion token when needed.
What appears in Grafana
Custom metrics and traces appear in managed Grafana after your application emits OTLP data. Traces with recent activity are also listed under Monitoring → Observability, ordered by start time. Metric names, labels, and resource attributes come from your OpenTelemetry SDK or collector configuration, plus the SailOps resource attributes injected by the platform.
If dashboards are empty:
- confirm Grafana is Ready
- confirm the project or workload is Configured
- resync the project if any workload shows Missing token
- verify your application exports OTLP metrics or traces
- check that the workload uses the SailOps endpoint and bearer header
Usage and quotas
SailOps tracks observability usage for metric samples, trace spans, active series, and retained telemetry storage. You can view usage under Monitoring → Usage → Telemetry.
The Usage page shows:
- metric samples ingested
- trace spans ingested
- retained storage in GiB-month
- active series average, p95, and maximum
- included quota for your plan
- current and projected overage
- per-project usage breakdown