Skip to main content

Alerts Overview

Stay informed with real-time alerts for all your resources.

What are Alerts?​

SailOps monitors your infrastructure 24/7 and notifies you when issues arise:

  • Container issues - Crashes, restarts, OOM kills, image pull errors
  • Resource usage - High CPU or memory exceeding thresholds
  • Deployment issues - Failed deployments, configuration errors
  • Security vulnerabilities - Critical and high CVEs detected by Trivy scans
  • Automatic resolution - Notifications when issues are fixed

Alert Center​

The Alert Center is the unified investigation hub for all your alerts across every project in an organization. It lives under Monitoring > Alerts and organizes alerts as incidents — persistent records that track a problem from detection through acknowledgment to resolution.

Incident Model​

Every detected problem creates an incident. An incident tracks:

  • The observed condition (active, recovering, unknown, suspended, resolved)
  • Acknowledgment — a shared team action recorded with the acknowledger's identity
  • Personal read state — each member tracks their own read/unread state independently
  • Evidence — metric samples, thresholds and affected instances captured at detection time
  • Timeline — chronological events including observations, team actions and notification outcomes
  • Deliveries — the outcome of each notification sent to configured destinations

Incident Views​

The incident list separates alerts into focused views:

ViewDescription
AttentionActive incidents nobody has acknowledged yet
AcknowledgedActive incidents a team member has taken responsibility for
UnknownIncidents where the observed condition is uncertain
HistoryResolved or closed incidents
ExecutionsTerminal job and cronjob execution records
AllEvery incident regardless of state

Incident Inspector​

Click an incident to open the inspector, which shows:

  • The observed condition, severity and triggering message
  • Acknowledgment status and who acknowledged it
  • Metric evidence chart with observed values and threshold series
  • Timeline of observations, team actions and notifications
  • Delivery outcomes for each notification destination
  • Investigation links to metrics and logs around the detection time

The inspector switches to full-width on viewports below 1280px.

Alert Types​

Container Alerts​

AlertDescriptionSeverity
Crash LoopContainer repeatedly crashingCritical
OOM KilledOut of memory terminationCritical
Container ErrorGeneral container failureCritical
Image Pull ErrorCannot pull container imageCritical
Config ErrorConfiguration problemCritical
Volume Mount ErrorStorage mounting failedCritical
Deployment FailedDeployment could not completeCritical

Resource Alerts​

AlertDescriptionSeverity
CPU HighCPU usage above thresholdWarning/Critical
Memory HighMemory usage above thresholdWarning/Critical

CPU and memory thresholds measure each instance's usage as a percentage of its configured limit, averaged over one minute, with each instance evaluated separately.

Instance Alerts​

AlertDescriptionSeverity
Instance RestartInstance has restartedWarning
Instance CrashInstance has crashedCritical

Capacity Alerts​

AlertDescriptionSeverity
Spot Capacity ExhaustedSpot pod cannot be scheduled — capacity fullWarning
Out Of StockWorkload stuck pending — no suitable node available for 10+ minutesCritical
GPU Stock ExhaustedNo GPU of the requested type available at the providerWarning

These alerts help detect resource availability issues at the cloud provider level:

  • Spot capacity alerts appear quickly (30 seconds) because spot pods don't trigger auto-scaling, making this the primary signal of full capacity
  • Out of Stock alerts fire after 10 minutes, allowing time for the cluster auto-scaler to provision nodes before determining capacity is truly exhausted
  • GPU stock alerts detect when GPU capacity is exhausted at the cloud provider level, indicating your workload is waiting for a free GPU slot

Security Alerts​

AlertDescriptionSeverity
Vulnerability ScanCritical or high CVEs found in deployed container imagesWarning/Critical

Security incidents include structured evidence listing affected images, individual findings (CVE ID, package, installed and fixed versions) and links to advisories.

Alert Rules (Thresholds)​

Alert rules define when metric-based incidents are created. Rules follow an inheritance chain: SailOps defaults → Organization → Project → Resource. Each level can override or inherit thresholds from its parent.

Threshold Configuration​

Each metric rule includes:

SettingDescription
Warning thresholdValue that triggers a warning incident
Critical thresholdValue that triggers a critical incident (must exceed warning)
Trigger durationHow long the condition must persist (30s–24h)
Recovery thresholdValue below which the incident recovers
Recovery durationHow long recovery must hold before resolving
EnabledWhether evaluation is active (disabling preserves the override)

Managing Rules​

  1. Go to Monitoring > Alerts > Thresholds
  2. Select the scope: organization defaults, project defaults, or a specific resource
  3. Click a rule to open the editor drawer
  4. Adjust thresholds and durations
  5. Review the change summary and click Save thresholds

Removing an override restores the parent's settings. Disabling a rule stops evaluation without removing the override.

Notification Activation​

Notifications for metric rules require explicit activation after seven days of observation with complete measurements and at least one enabled destination. This observation period prevents false positives on newly configured thresholds.

Incident Lifecycle​

Conditions​

ConditionDescription
ActiveThe problem is currently observed
RecoveringThe condition improved but has not stabilized
UnknownThe platform cannot determine the current state
SuspendedObservation is paused (e.g. during maintenance)
ResolvedThe issue is confirmed fixed

Viewing Alerts​

Monitoring Sidebar​

The global sidebar shows a Monitoring entry with an unread badge counting incidents across all authorized projects in the organization.

  1. Click Monitoring in the left sidebar
  2. Select Alerts to open the Alert Center
  3. Filter by project, resource, severity, or search
  4. Click an incident to investigate

Resource-Specific Alerts​

View alerts for a specific resource:

  1. Go to the resource (Container, Database, etc.)
  2. Click the Alerts tab
  3. See incidents related to that resource
  4. Switch to Thresholds to manage that resource's alert rules

Incident Details​

Each incident includes:

FieldDescription
TypeKind of issue detected
ResourceAffected container/database/etc. and whether it still exists
ProjectWhich project the resource belongs to
SeverityCritical, Warning, Error, or Info
ConditionActive, Recovering, Unknown, Suspended, or Resolved
Occurred atWhen the issue was first detected
Last observedMost recent observation
Resolved atWhen the issue was fixed (if applicable)
AcknowledgedWho acknowledged and when
ReadYour personal read timestamp
EvidenceMetric samples, thresholds, and affected instances
DiagnosticWhen available, a structured root-cause analysis with a summary, actionable suggestions, and sanitized technical details
TimelineChronological observations, actions and notification events (latest 500)
DeliveriesNotification outcomes per destination (latest 100)

Severity Levels​

SailOps uses four severity levels:

Critical​

Immediate attention required:

  • Service is down
  • Data at risk
  • Complete outage

Examples:

  • Container crash loop
  • OOM kill
  • Image pull failures
  • Deployment failures

Warning​

Issue needs attention soon:

  • Performance degradation
  • Resource constraints approaching limits
  • Configuration may need adjustment

Examples:

  • High CPU or memory usage
  • Instance restarts
  • Security vulnerabilities detected

Error​

An error condition observed by the platform.

Info​

Informational notices:

  • Status updates
  • Successful resolutions

Smart Detection​

SailOps uses intelligent detection to reduce noise:

Crash Loop Detection​

Container crashes are detected by:

  • Restart count - 3+ restarts in short period
  • Restart frequency - More than 3 restarts in 5 minutes

Stability Checking​

Incidents auto-resolve only after the condition stabilizes:

  • Configurable recovery duration (30s–24h)
  • Prevents false positive resolutions

Deduplication​

Prevents alert spam:

  • Same issue updates the existing incident instead of creating a new one
  • Duplicate diagnostic alerts are stored but do not trigger additional notifications

Real-Time Updates​

In-App Notifications​

  • Real-time incident updates via SSE (organization-wide)
  • Unread badge on the Monitoring sidebar entry
  • In-app toast notifications for new incidents (Critical: 8 seconds, others: 5 seconds)
  • Toasts are independent of external notification destinations

External Notifications​

Configure named destinations to receive alerts:

  • Slack - Team channels
  • Discord - Webhook integration
  • Microsoft Teams - Incoming webhooks
  • Mattermost - Incoming webhooks
  • Webhooks - Custom integrations
  • Email - Personal subscription per member

See Alert Channels for setup.

Incident Actions​

Mark as Read​

Read state is personal. Use the incident inspector or the API to mark incidents as read or unread for yourself. Reading an incident never acknowledges or resolves it.

Acknowledge​

Records that you take responsibility for an active incident:

  1. Open the incident
  2. Click Acknowledge

Acknowledgment is shared across the team — everyone sees who acknowledged and when.

Create Ticket​

For incidents that need escalation:

  1. Open the incident
  2. Click Create Ticket
  3. You'll be redirected to the Support page
Automatic Resolution

Most incidents auto-resolve when the issue is fixed. Unknown observations never imply recovery.

Alert Settings​

Thresholds​

Configure at organization, project, or resource level:

  1. Go to Monitoring > Alerts > Thresholds
  2. Select the scope
  3. Edit rules with their warning/critical thresholds and durations

Notifications​

Configure where alerts are delivered:

  1. Go to Monitoring > Alerts > Notifications
  2. Manage named destinations (Slack, Discord, Teams, etc.)
  3. Manage your personal email subscription

Maintenance Windows​

Pause notifications during planned work:

  1. Go to Monitoring > Alerts > Maintenance
  2. Schedule a pause with a reason, start and end time
  3. Optionally scope to a specific resource

Observation continues during maintenance — only notifications are suppressed.

Best Practices​

1. Set Up Destinations​

Don't rely only on in-app alerts:

  • Configure Slack or Discord as a primary destination
  • Add a backup destination for critical alerts
  • Test your destinations

2. Configure Appropriate Thresholds​

Avoid alert fatigue:

  • Set realistic CPU/memory thresholds
  • Consider normal usage patterns
  • Use the observation period before activating notifications

3. Use Acknowledgment​

Coordinate incident response:

  • Acknowledge incidents you're investigating
  • Other team members can see who is handling what
  • Acknowledged incidents move out of the Attention view

4. Schedule Maintenance​

Prevent noise during planned work:

  • Create maintenance windows before deployments
  • Scope to the affected resource when possible
  • Include a reason for audit purposes

5. Review Alert History​

Learn from past issues:

  • Look for patterns in resolved incidents
  • Review delivery outcomes to ensure notifications reach your team
  • Adjust thresholds based on experience

Troubleshooting​

Not Receiving Notifications​

  1. Check that destinations are configured and enabled
  2. Verify notifications are activated on the relevant rules (seven-day observation required)
  3. Test destinations from Monitoring > Alerts > Notifications
  4. Check that no maintenance window is suppressing delivery

Too Many Alerts​

  1. Review and adjust thresholds at the appropriate level
  2. Increase trigger durations to filter transient spikes
  3. Disable rules that are not relevant to specific resources

Incident Not Resolving​

  1. Check if issue is truly fixed
  2. Wait for the recovery duration to pass
  3. Check for related issues