Alerts Overview
Stay informed with real-time alerts for all your resources.
What are Alerts?
SailOps monitors your infrastructure 24/7 and notifies you when issues arise:
- Container issues - Crashes, restarts, OOM kills, image pull errors
- Resource usage - High CPU or memory exceeding thresholds
- Deployment issues - Failed deployments, configuration errors
- Security vulnerabilities - Critical and high CVEs detected by Trivy scans
- Automatic resolution - Notifications when issues are fixed
Alert Center
The Alert Center is the unified investigation hub for all your alerts across every project in an organization. It lives under Monitoring > Alerts and organizes alerts as incidents — persistent records that track a problem from detection through acknowledgment to resolution.
Incident Model
Every detected problem creates an incident. An incident tracks:
- The observed condition (active, recovering, unknown, suspended, resolved)
- Acknowledgment — a shared team action recorded with the acknowledger's identity
- Personal read state — each member tracks their own read/unread state independently
- Evidence — metric samples, thresholds and affected instances captured at detection time
- Timeline — chronological events including observations, team actions and notification outcomes
- Deliveries — the outcome of each notification sent to configured destinations
Incident Views
The incident list separates alerts into focused views:
| View | Description |
|---|---|
| Attention | Active incidents nobody has acknowledged yet |
| Acknowledged | Active incidents a team member has taken responsibility for |
| Unknown | Incidents where the observed condition is uncertain |
| History | Resolved or closed incidents |
| Executions | Terminal job and cronjob execution records |
| All | Every incident regardless of state |
Incident Inspector
Click an incident to open the inspector, which shows:
- The observed condition, severity and triggering message
- Acknowledgment status and who acknowledged it
- Metric evidence chart with observed values and threshold series
- Timeline of observations, team actions and notifications
- Delivery outcomes for each notification destination
- Investigation links to metrics and logs around the detection time
The inspector switches to full-width on viewports below 1280px.
Alert Types
Container Alerts
| Alert | Description | Severity |
|---|---|---|
| Crash Loop | Container repeatedly crashing | Critical |
| OOM Killed | Out of memory termination | Critical |
| Container Error | General container failure | Critical |
| Image Pull Error | Cannot pull container image | Critical |
| Config Error | Configuration problem | Critical |
| Volume Mount Error | Storage mounting failed | Critical |
| Deployment Failed | Deployment could not complete | Critical |
Resource Alerts
| Alert | Description | Severity |
|---|---|---|
| CPU High | CPU usage above threshold | Warning/Critical |
| Memory High | Memory usage above threshold | Warning/Critical |
CPU and memory thresholds measure each instance's usage as a percentage of its configured limit, averaged over one minute, with each instance evaluated separately.
Instance Alerts
| Alert | Description | Severity |
|---|---|---|
| Instance Restart | Instance has restarted | Warning |
| Instance Crash | Instance has crashed | Critical |
Capacity Alerts
| Alert | Description | Severity |
|---|---|---|
| Spot Capacity Exhausted | Spot pod cannot be scheduled — capacity full | Warning |
| Out Of Stock | Workload stuck pending — no suitable node available for 10+ minutes | Critical |
| GPU Stock Exhausted | No GPU of the requested type available at the provider | Warning |
These alerts help detect resource availability issues at the cloud provider level:
- Spot capacity alerts appear quickly (30 seconds) because spot pods don't trigger auto-scaling, making this the primary signal of full capacity
- Out of Stock alerts fire after 10 minutes, allowing time for the cluster auto-scaler to provision nodes before determining capacity is truly exhausted
- GPU stock alerts detect when GPU capacity is exhausted at the cloud provider level, indicating your workload is waiting for a free GPU slot
Security Alerts
| Alert | Description | Severity |
|---|---|---|
| Vulnerability Scan | Critical or high CVEs found in deployed container images | Warning/Critical |
Security incidents include structured evidence listing affected images, individual findings (CVE ID, package, installed and fixed versions) and links to advisories.
Alert Rules (Thresholds)
Alert rules define when metric-based incidents are created. Rules follow an inheritance chain: SailOps defaults → Organization → Project → Resource. Each level can override or inherit thresholds from its parent.
Threshold Configuration
Each metric rule includes:
| Setting | Description |
|---|---|
| Warning threshold | Value that triggers a warning incident |
| Critical threshold | Value that triggers a critical incident (must exceed warning) |
| Trigger duration | How long the condition must persist (30s–24h) |
| Recovery threshold | Value below which the incident recovers |
| Recovery duration | How long recovery must hold before resolving |
| Enabled | Whether evaluation is active (disabling preserves the override) |
Managing Rules
- Go to Monitoring > Alerts > Thresholds
- Select the scope: organization defaults, project defaults, or a specific resource
- Click a rule to open the editor drawer
- Adjust thresholds and durations
- Review the change summary and click Save thresholds
Removing an override restores the parent's settings. Disabling a rule stops evaluation without removing the override.
Notification Activation
Notifications for metric rules require explicit activation after seven days of observation with complete measurements and at least one enabled destination. This observation period prevents false positives on newly configured thresholds.
Incident Lifecycle
Conditions
| Condition | Description |
|---|---|
| Active | The problem is currently observed |
| Recovering | The condition improved but has not stabilized |
| Unknown | The platform cannot determine the current state |
| Suspended | Observation is paused (e.g. during maintenance) |
| Resolved | The issue is confirmed fixed |
Viewing Alerts
Monitoring Sidebar
The global sidebar shows a Monitoring entry with an unread badge counting incidents across all authorized projects in the organization.
- Click Monitoring in the left sidebar
- Select Alerts to open the Alert Center
- Filter by project, resource, severity, or search
- Click an incident to investigate
Resource-Specific Alerts
View alerts for a specific resource:
- Go to the resource (Container, Database, etc.)
- Click the Alerts tab
- See incidents related to that resource
- Switch to Thresholds to manage that resource's alert rules
Incident Details
Each incident includes:
| Field | Description |
|---|---|
| Type | Kind of issue detected |
| Resource | Affected container/database/etc. and whether it still exists |
| Project | Which project the resource belongs to |
| Severity | Critical, Warning, Error, or Info |
| Condition | Active, Recovering, Unknown, Suspended, or Resolved |
| Occurred at | When the issue was first detected |
| Last observed | Most recent observation |
| Resolved at | When the issue was fixed (if applicable) |
| Acknowledged | Who acknowledged and when |
| Read | Your personal read timestamp |
| Evidence | Metric samples, thresholds, and affected instances |
| Diagnostic | When available, a structured root-cause analysis with a summary, actionable suggestions, and sanitized technical details |
| Timeline | Chronological observations, actions and notification events (latest 500) |
| Deliveries | Notification outcomes per destination (latest 100) |
Severity Levels
SailOps uses four severity levels:
Critical
Immediate attention required:
- Service is down
- Data at risk
- Complete outage
Examples:
- Container crash loop
- OOM kill
- Image pull failures
- Deployment failures
Warning
Issue needs attention soon:
- Performance degradation
- Resource constraints approaching limits
- Configuration may need adjustment
Examples:
- High CPU or memory usage
- Instance restarts
- Security vulnerabilities detected
Error
An error condition observed by the platform.
Info
Informational notices:
- Status updates
- Successful resolutions
Smart Detection
SailOps uses intelligent detection to reduce noise:
Crash Loop Detection
Container crashes are detected by:
- Restart count - 3+ restarts in short period
- Restart frequency - More than 3 restarts in 5 minutes
Stability Checking
Incidents auto-resolve only after the condition stabilizes:
- Configurable recovery duration (30s–24h)
- Prevents false positive resolutions
Deduplication
Prevents alert spam:
- Same issue updates the existing incident instead of creating a new one
- Duplicate diagnostic alerts are stored but do not trigger additional notifications
Real-Time Updates
In-App Notifications
- Real-time incident updates via SSE (organization-wide)
- Unread badge on the Monitoring sidebar entry
- In-app toast notifications for new incidents (Critical: 8 seconds, others: 5 seconds)
- Toasts are independent of external notification destinations
External Notifications
Configure named destinations to receive alerts:
- Slack - Team channels
- Discord - Webhook integration
- Microsoft Teams - Incoming webhooks
- Mattermost - Incoming webhooks
- Webhooks - Custom integrations
- Email - Personal subscription per member
See Alert Channels for setup.
Incident Actions
Mark as Read
Read state is personal. Use the incident inspector or the API to mark incidents as read or unread for yourself. Reading an incident never acknowledges or resolves it.
Acknowledge
Records that you take responsibility for an active incident:
- Open the incident
- Click Acknowledge
Acknowledgment is shared across the team — everyone sees who acknowledged and when.
Create Ticket
For incidents that need escalation:
- Open the incident
- Click Create Ticket
- You'll be redirected to the Support page
Most incidents auto-resolve when the issue is fixed. Unknown observations never imply recovery.
Alert Settings
Thresholds
Configure at organization, project, or resource level:
- Go to Monitoring > Alerts > Thresholds
- Select the scope
- Edit rules with their warning/critical thresholds and durations
Notifications
Configure where alerts are delivered:
- Go to Monitoring > Alerts > Notifications
- Manage named destinations (Slack, Discord, Teams, etc.)
- Manage your personal email subscription
Maintenance Windows
Pause notifications during planned work:
- Go to Monitoring > Alerts > Maintenance
- Schedule a pause with a reason, start and end time
- Optionally scope to a specific resource
Observation continues during maintenance — only notifications are suppressed.
Best Practices
1. Set Up Destinations
Don't rely only on in-app alerts:
- Configure Slack or Discord as a primary destination
- Add a backup destination for critical alerts
- Test your destinations
2. Configure Appropriate Thresholds
Avoid alert fatigue:
- Set realistic CPU/memory thresholds
- Consider normal usage patterns
- Use the observation period before activating notifications
3. Use Acknowledgment
Coordinate incident response:
- Acknowledge incidents you're investigating
- Other team members can see who is handling what
- Acknowledged incidents move out of the Attention view
4. Schedule Maintenance
Prevent noise during planned work:
- Create maintenance windows before deployments
- Scope to the affected resource when possible
- Include a reason for audit purposes
5. Review Alert History
Learn from past issues:
- Look for patterns in resolved incidents
- Review delivery outcomes to ensure notifications reach your team
- Adjust thresholds based on experience
Troubleshooting
Not Receiving Notifications
- Check that destinations are configured and enabled
- Verify notifications are activated on the relevant rules (seven-day observation required)
- Test destinations from Monitoring > Alerts > Notifications
- Check that no maintenance window is suppressing delivery
Too Many Alerts
- Review and adjust thresholds at the appropriate level
- Increase trigger durations to filter transient spikes
- Disable rules that are not relevant to specific resources
Incident Not Resolving
- Check if issue is truly fixed
- Wait for the recovery duration to pass
- Check for related issues