Managed Observability
Investigate application telemetry and SailOps infrastructure signals from one managed Grafana workspace.
Managed observability gives each organization a SailOps-provisioned Grafana workspace with preconfigured datasources and dashboards. Use it when you need deeper analysis than the built-in Monitoring pages, especially for custom application metrics and full trace investigation.
What managed observability includes
| Signal | Source | Where to use it |
|---|---|---|
| Custom metrics | Your workloads export OTLP metrics to SailOps | Grafana Custom Metrics datasource |
| Traces | Your workloads export OTLP traces to SailOps | Grafana Traces datasource |
| Service dependencies | SailOps derives the service graph from your traces | Grafana Trace Dependencies datasource |
| Infrastructure metrics | SailOps collects resource usage from the cluster | Grafana infra dashboards and Monitoring metrics |
| Logs | SailOps collects workload stdout and stderr | Grafana logs datasource and Log Explorer |
The built-in Metrics page and managed Grafana are complementary. Built-in metrics focus on SailOps resource usage, and the built-in Observability page lists traces with recent activity, ordered by start time. Grafana is the place for custom OpenTelemetry metrics, span-level trace analysis, and cross-signal dashboards.
Activate Grafana
- Open Monitoring for your organization.
- Go to Grafana.
- Click Activate Grafana.
- Wait for the status to become Ready.
- Click Open Grafana.
Activation provisions an organization-scoped Grafana workspace and connects it to SailOps through authenticated access. Users open Grafana through SailOps; the backend datasources are not exposed directly. Activation is idempotent — you can safely click Activate Grafana or Retry activation again if the workspace gets stuck in Provisioning or enters an Error state.
You need organization maintainer or owner permissions to activate Grafana. Opening the workspace requires organization-level monitoring access; project-only permissions do not grant access.
Grafana access and security
The interface uses your SailOps session. SailOps maps your organization role to a Grafana role: owners and maintainers become Admin, developers become Editor, and readers and accountants become Viewer. Grafana then applies its native permissions.
The workspace contains a single Grafana organization and covers all of your SailOps organization. Project filters narrow results; they do not isolate access to a project. Service Account tokens follow Grafana permissions, independently of SailOps API keys and scopes.
The Grafana page shows workspace status, an Open Grafana action, connected projects, OpenTelemetry connections, and active signals.
The managed datasources are read-only, and their credentials are managed by SailOps. Do not copy these credentials out of Grafana or configure workloads with them; workloads use ingestion tokens instead. Datasource queries enforce organization state and tenant isolation.
The public Grafana hostname also accepts native Service Account tokens. Direct access bypasses the SailOps proxy's workspace and lifecycle checks, while managed datasource queries retain their organization checks. Organization suspension stops Grafana asynchronously.
The proxy and public Grafana route block the same paths for disabled features. The proxy applies additional restrictions to datasource and plugin administration. Grafana alerting, snapshots, public dashboard sharing, and additional Grafana organizations are disabled.
Service Accounts for API access
- Open Grafana as an organization owner or maintainer.
- Go to Administration > Users and access > Service accounts and create an account. Use Viewer for read-only queries.
- Add a token that expires within 90 days, then store it in your secret manager. Rotate tokens and revoke unused or compromised ones in Grafana.
- Send it as a Bearer token to your organization's SailOps Grafana URL:
curl --fail-with-body --max-time 20 \
--header "Authorization: Bearer ${GRAFANA_TOKEN}" \
"https://app.sailops.io/api/v1/organizations/${ORGANIZATION_ID}/observability/grafana/api/search"
Use the same URL prefix for query endpoints such as api/ds/query. SailOps checks workspace availability and organization state; Grafana validates the token and its permissions. An invalid token is rejected even when a browser session is present. Service Account tokens cannot open WebSocket connections through this proxy.
New tokens must expire within 90 days. Existing tokens, including tokens without an expiry, remain unchanged; rotate them to apply this limit.
If a Service Account is compromised, delete all of its tokens before changing its role or re-enabling it. Disabling or downgrading the account does not guarantee an immediate access cutoff because Grafana caches account state and permissions. To change permissions during rotation, revoke the old tokens first, change the role, then issue replacement tokens.
See Grafana Service Accounts for token and role management.
Dashboards and datasources
Managed Grafana is provisioned with datasources for:
- Metrics - SailOps-collected CPU, memory, storage, and resource usage
- Logs - SailOps-collected workload logs
- Custom Metrics - customer OTLP metrics stored in the managed metrics backend
- Traces - customer OTLP traces stored in the managed tracing backend
- Trace Dependencies - the service dependency graph derived from your traces
The APM dashboard combines TraceQL metrics and the Trace Dependencies graph to show request rate, error rate, p95 latency, and service relationships.
Grafana dashboards offer project and resource filters where available. Logs expose resource-oriented labels so you can filter by project, workload type, resource name, namespace, pod, container, and stream.
Traces on the Observability page
The Observability page includes a Traces panel below the metric charts. It shows the most recent traces received by SailOps for the current organization, ordered by start time (newest first).
Each trace displays:
| Column | Description |
|---|---|
| Start | Timestamp of the root span |
| Service | Service name from the root span |
| Root operation | Span name of the root span (e.g. GET /api/orders) |
| Duration | End-to-end trace duration |
| Trace ID | Truncated identifier (hover for the full ID) |
Up to 20 traces are shown per query. The time window and the Workload filter are shared with the metric charts — selecting a workload name narrows both metrics and traces. The Workload type filter applies only to metrics; when a workload type other than "All types" is selected, a note explains that traces follow the workload name filter only.
If OpenTelemetry is not activated for the organization, the panel shows an activation link instead. If traces fail to load, metrics remain visible and you can retry traces independently.
For span-level trace analysis and cross-signal investigation, open managed Grafana from the Grafana page.
Monitoring Center pages
The Monitoring Center provides built-in views around the same system:
| Page | Purpose |
|---|---|
| Overview | Organization-wide snapshot: open alert count and severity breakdown, projects in scope, OpenTelemetry setup status, alert preview with severity filtering and detail inspection, resource metrics sparklines (CPU, memory, storage, running instances) over a selectable period (up to 24 hours), managed telemetry usage (metric samples, trace spans, storage, overage), Grafana workspace status, and quick links to Logs, Events and Alert history. Each section is gated by the viewer's permissions. |
| Observability | Built-in workload, CPU, memory, and storage panels, plus traces with recent activity in the selected window (up to 24 hours), ordered by start time |
| Log Explorer | Server-side log search with project, workload, level, stream, time, and text filters |
| Project observability | OpenTelemetry injection status for projects and workloads |
| Grafana | Managed Grafana activation and access |
Telemetry and log usage, quotas, and projected overage are now in Organization Settings under dedicated Observability usage and Log usage pages in the Billing section.
Related Topics
Querying the Metrics datasource
The infrastructure Metrics datasource accepts a PromQL subset: metric{label="value"}, optionally wrapped in sum, avg, max or min, with optional by (labels). Matchers support =, !=, =~, !~ and single or double quotes. Functions, arithmetic and comparisons are unavailable in queries; use Grafana Reduce, Math or Threshold expressions.
Instant queries accept the SailOps extension [d], for example sum(sailops_gateway_latency_avg_ms{project="my-project"}[1h]). It evaluates over that window, rather than returning a Prometheus range vector. Counters and ratios default to a 5-minute window. [d] is unavailable on range queries; windows must be between 30 s and 93 days. Unsupported syntax, names or labels return 400 bad_data; limit exhaustion returns 422 execution, and backend failures return 500. Use project rather than the internal namespace.
Samples carry interval-start timestamps and typically lag collection by 30–45 s. Range queries end at the last complete step before now minus 30 s. Steps grow to multiples of 15 s to stay within 1,100 points. No data emits no sample; no traffic emits no ratio sample. Observed flat counters emit a measured zero. Instant gauges disappear about 45 s after ingestion stops.
Labels and gateway series
project is common to every family. Container/workload series expose workload, resource, workload_kind, pod, instance, product_id; storage adds its volume identity; engine series expose their cluster/instance labels. Gateway series expose gateway, route, route_kind and gateway_scope (project or shared). Scope describes the owning gateway, not the tenant: shared rows still belong to the route's project. Discovery is the authority for labels supported by each metric.
| Gateway metric | Meaning |
|---|---|
sailops_gateway_requests_per_sec | HTTP/gRPC request rate |
sailops_gateway_requests_2xx_per_sec, sailops_gateway_requests_4xx_per_sec, sailops_gateway_requests_5xx_per_sec | HTTP status-class rates |
sailops_gateway_requests_other_per_sec | 1xx and 3xx rate |
sailops_gateway_5xx_ratio | 5xx / HTTP and gRPC requests |
sailops_gateway_error_ratio | Deprecated alias of sailops_gateway_5xx_ratio; excludes 4xx |
sailops_gateway_latency_avg_ms | Pooled mean upstream response time |
sailops_gateway_latency_sum_ms_per_sec, sailops_gateway_latency_count_per_sec | Response-time sum and sample-count rates |
sailops_gateway_requests_le_500ms_per_sec | Completed upstream responses at or below 500 ms |
sailops_gateway_slow_requests_ratio | Completed upstream responses above 500 ms / sample count |
sailops_gateway_timeouts_per_sec | HTTP/gRPC timeout rate |
sailops_gateway_connection_failures_per_sec | Failed connection attempts, separated by route_kind |
sailops_gateway_tcp_connections_per_sec | TCPRoute and TLSRoute connection rate |
Weighted calculations and dashboards
Ratios pool counter deltas across Envoy sources and routes: sum of numerators / sum of denominators. sum, avg and no aggregation use this pooling; min and max compare pooled series ratios. An unweighted average of interval averages can misrepresent the whole window.
To calculate a mean response time, query sailops_gateway_latency_sum_ms_per_sec and sailops_gateway_latency_count_per_sec, apply Reduce(sum) to each, then Math $sum/$count. Alternatively use the instant window selector on sailops_gateway_latency_avg_ms. Request rate minus sailops_gateway_latency_count_per_sec estimates responses without a complete upstream sample over a window; histogram flush timing can briefly make the difference negative.
Gateways & Traffic separates Request rate (selected range), Error rate (5xx, selected range) and Avg upstream response time (selected range) from route time series. TCP/TLS connections is separate from HTTP traffic. The console error rate includes 4xx as well as 5xx.
Grafana alerting and SailOps alerts
Grafana alert rules, contact points, and notifications are unavailable in managed Grafana. SailOps alerts in the Monitoring Center are a separate feature and remain available.