Skip to main content

Managed Observability

Investigate application telemetry and SailOps infrastructure signals from one managed Grafana workspace.

Managed observability gives each organization a SailOps-provisioned Grafana workspace with preconfigured datasources and dashboards. Use it when you need deeper analysis than the built-in Monitoring pages, especially for custom application metrics and full trace investigation.

What managed observability includes​

SignalSourceWhere to use it
Custom metricsYour workloads export OTLP metrics to SailOpsGrafana Custom Metrics datasource
TracesYour workloads export OTLP traces to SailOpsGrafana Traces datasource
Service dependenciesSailOps derives the service graph from your tracesGrafana Trace Dependencies datasource
Infrastructure metricsSailOps collects resource usage from the clusterGrafana infra dashboards and Monitoring metrics
LogsSailOps collects workload stdout and stderrGrafana logs datasource and Log Explorer
info

The built-in Metrics page and managed Grafana are complementary. Built-in metrics focus on SailOps resource usage, and the built-in Observability page lists traces with recent activity, ordered by start time. Grafana is the place for custom OpenTelemetry metrics, span-level trace analysis, and cross-signal dashboards.

Activate Grafana​

  1. Open Monitoring for your organization.
  2. Go to Grafana.
  3. Click Activate Grafana.
  4. Wait for the status to become Ready.
  5. Click Open Grafana.

Activation provisions an organization-scoped Grafana workspace and connects it to SailOps through authenticated access. Users open Grafana through SailOps; the backend datasources are not exposed directly. Activation is idempotent — you can safely click Activate Grafana or Retry activation again if the workspace gets stuck in Provisioning or enters an Error state.

warning

You need organization maintainer or owner permissions to activate Grafana. Opening the workspace requires organization-level monitoring access; project-only permissions do not grant access.

Grafana access and security​

The interface uses your SailOps session. SailOps maps your organization role to a Grafana role: owners and maintainers become Admin, developers become Editor, and readers and accountants become Viewer. Grafana then applies its native permissions.

The workspace contains a single Grafana organization and covers all of your SailOps organization. Project filters narrow results; they do not isolate access to a project. Service Account tokens follow Grafana permissions, independently of SailOps API keys and scopes.

The Grafana page shows workspace status, an Open Grafana action, connected projects, OpenTelemetry connections, and active signals.

The managed datasources are read-only, and their credentials are managed by SailOps. Do not copy these credentials out of Grafana or configure workloads with them; workloads use ingestion tokens instead. Datasource queries enforce organization state and tenant isolation.

The public Grafana hostname also accepts native Service Account tokens. Direct access bypasses the SailOps proxy's workspace and lifecycle checks, while managed datasource queries retain their organization checks. Organization suspension stops Grafana asynchronously.

The proxy and public Grafana route block the same paths for disabled features. The proxy applies additional restrictions to datasource and plugin administration. Grafana alerting, snapshots, public dashboard sharing, and additional Grafana organizations are disabled.

Service Accounts for API access​

  1. Open Grafana as an organization owner or maintainer.
  2. Go to Administration > Users and access > Service accounts and create an account. Use Viewer for read-only queries.
  3. Add a token that expires within 90 days, then store it in your secret manager. Rotate tokens and revoke unused or compromised ones in Grafana.
  4. Send it as a Bearer token to your organization's SailOps Grafana URL:
curl --fail-with-body --max-time 20 \
--header "Authorization: Bearer ${GRAFANA_TOKEN}" \
"https://app.sailops.io/api/v1/organizations/${ORGANIZATION_ID}/observability/grafana/api/search"

Use the same URL prefix for query endpoints such as api/ds/query. SailOps checks workspace availability and organization state; Grafana validates the token and its permissions. An invalid token is rejected even when a browser session is present. Service Account tokens cannot open WebSocket connections through this proxy.

New tokens must expire within 90 days. Existing tokens, including tokens without an expiry, remain unchanged; rotate them to apply this limit.

If a Service Account is compromised, delete all of its tokens before changing its role or re-enabling it. Disabling or downgrading the account does not guarantee an immediate access cutoff because Grafana caches account state and permissions. To change permissions during rotation, revoke the old tokens first, change the role, then issue replacement tokens.

See Grafana Service Accounts for token and role management.

Dashboards and datasources​

Managed Grafana is provisioned with datasources for:

  • Metrics - SailOps-collected CPU, memory, storage, and resource usage
  • Logs - SailOps-collected workload logs
  • Custom Metrics - customer OTLP metrics stored in the managed metrics backend
  • Traces - customer OTLP traces stored in the managed tracing backend
  • Trace Dependencies - the service dependency graph derived from your traces

The APM dashboard combines TraceQL metrics and the Trace Dependencies graph to show request rate, error rate, p95 latency, and service relationships.

Grafana dashboards offer project and resource filters where available. Logs expose resource-oriented labels so you can filter by project, workload type, resource name, namespace, pod, container, and stream.

Traces on the Observability page​

The Observability page includes a Traces panel below the metric charts. It shows the most recent traces received by SailOps for the current organization, ordered by start time (newest first).

Each trace displays:

ColumnDescription
StartTimestamp of the root span
ServiceService name from the root span
Root operationSpan name of the root span (e.g. GET /api/orders)
DurationEnd-to-end trace duration
Trace IDTruncated identifier (hover for the full ID)

Up to 20 traces are shown per query. The time window and the Workload filter are shared with the metric charts — selecting a workload name narrows both metrics and traces. The Workload type filter applies only to metrics; when a workload type other than "All types" is selected, a note explains that traces follow the workload name filter only.

If OpenTelemetry is not activated for the organization, the panel shows an activation link instead. If traces fail to load, metrics remain visible and you can retry traces independently.

tip

For span-level trace analysis and cross-signal investigation, open managed Grafana from the Grafana page.

Monitoring Center pages​

The Monitoring Center provides built-in views around the same system:

PagePurpose
OverviewOrganization-wide snapshot: open alert count and severity breakdown, projects in scope, OpenTelemetry setup status, alert preview with severity filtering and detail inspection, resource metrics sparklines (CPU, memory, storage, running instances) over a selectable period (up to 24 hours), managed telemetry usage (metric samples, trace spans, storage, overage), Grafana workspace status, and quick links to Logs, Events and Alert history. Each section is gated by the viewer's permissions.
ObservabilityBuilt-in workload, CPU, memory, and storage panels, plus traces with recent activity in the selected window (up to 24 hours), ordered by start time
Log ExplorerServer-side log search with project, workload, level, stream, time, and text filters
Project observabilityOpenTelemetry injection status for projects and workloads
GrafanaManaged Grafana activation and access

Telemetry and log usage, quotas, and projected overage are now in Organization Settings under dedicated Observability usage and Log usage pages in the Billing section.

Querying the Metrics datasource​

The infrastructure Metrics datasource accepts a PromQL subset: metric{label="value"}, optionally wrapped in sum, avg, max or min, with optional by (labels). Matchers support =, !=, =~, !~ and single or double quotes. Functions, arithmetic and comparisons are unavailable in queries; use Grafana Reduce, Math or Threshold expressions.

Instant queries accept the SailOps extension [d], for example sum(sailops_gateway_latency_avg_ms{project="my-project"}[1h]). It evaluates over that window, rather than returning a Prometheus range vector. Counters and ratios default to a 5-minute window. [d] is unavailable on range queries; windows must be between 30 s and 93 days. Unsupported syntax, names or labels return 400 bad_data; limit exhaustion returns 422 execution, and backend failures return 500. Use project rather than the internal namespace.

Samples carry interval-start timestamps and typically lag collection by 30–45 s. Range queries end at the last complete step before now minus 30 s. Steps grow to multiples of 15 s to stay within 1,100 points. No data emits no sample; no traffic emits no ratio sample. Observed flat counters emit a measured zero. Instant gauges disappear about 45 s after ingestion stops.

Labels and gateway series​

project is common to every family. Container/workload series expose workload, resource, workload_kind, pod, instance, product_id; storage adds its volume identity; engine series expose their cluster/instance labels. Gateway series expose gateway, route, route_kind and gateway_scope (project or shared). Scope describes the owning gateway, not the tenant: shared rows still belong to the route's project. Discovery is the authority for labels supported by each metric.

Gateway metricMeaning
sailops_gateway_requests_per_secHTTP/gRPC request rate
sailops_gateway_requests_2xx_per_sec, sailops_gateway_requests_4xx_per_sec, sailops_gateway_requests_5xx_per_secHTTP status-class rates
sailops_gateway_requests_other_per_sec1xx and 3xx rate
sailops_gateway_5xx_ratio5xx / HTTP and gRPC requests
sailops_gateway_error_ratioDeprecated alias of sailops_gateway_5xx_ratio; excludes 4xx
sailops_gateway_latency_avg_msPooled mean upstream response time
sailops_gateway_latency_sum_ms_per_sec, sailops_gateway_latency_count_per_secResponse-time sum and sample-count rates
sailops_gateway_requests_le_500ms_per_secCompleted upstream responses at or below 500 ms
sailops_gateway_slow_requests_ratioCompleted upstream responses above 500 ms / sample count
sailops_gateway_timeouts_per_secHTTP/gRPC timeout rate
sailops_gateway_connection_failures_per_secFailed connection attempts, separated by route_kind
sailops_gateway_tcp_connections_per_secTCPRoute and TLSRoute connection rate

Weighted calculations and dashboards​

Ratios pool counter deltas across Envoy sources and routes: sum of numerators / sum of denominators. sum, avg and no aggregation use this pooling; min and max compare pooled series ratios. An unweighted average of interval averages can misrepresent the whole window.

To calculate a mean response time, query sailops_gateway_latency_sum_ms_per_sec and sailops_gateway_latency_count_per_sec, apply Reduce(sum) to each, then Math $sum/$count. Alternatively use the instant window selector on sailops_gateway_latency_avg_ms. Request rate minus sailops_gateway_latency_count_per_sec estimates responses without a complete upstream sample over a window; histogram flush timing can briefly make the difference negative.

Gateways & Traffic separates Request rate (selected range), Error rate (5xx, selected range) and Avg upstream response time (selected range) from route time series. TCP/TLS connections is separate from HTTP traffic. The console error rate includes 4xx as well as 5xx.

Grafana alerting and SailOps alerts​

Grafana alert rules, contact points, and notifications are unavailable in managed Grafana. SailOps alerts in the Monitoring Center are a separate feature and remain available.