Skip to content

Alerting Component

System alerting and notification management component that provides a unified interface for handling alerts from multiple sources and managing their lifecycle through Linear integration.

Overview

The alerting component centralizes alert management across different monitoring systems, automatically creating and tracking Linear issues for engineering teams. It supports multiple alert sources and provides a consistent workflow for alert resolution.

[!NOTE]
This component doesn't provide a public API: * alerts are defined through Eng Tools ⧉ * this component is responsible for executing alert checks and creating Linear issues following the configuration.

Key Features

Alert Sources

The component supports multiple alert sources:

  • Database Alerts - SQL-based alerts defined in the database
  • Datadog Alerts - Monitoring alerts from Datadog
  • Datadog Security Signals - Security events from Datadog
  • Turing Alerts - Custom alerts from internal Turing system

Linear Integration

Automatically creates and manages Linear issues for alerts:

  • Creates Linear tickets with appropriate labels and teams
  • Tracks alert resolution status
  • Configurable resolution times based on alert source
  • Supports oncall and intercom conversation workflows

Alert Lifecycle

  1. Detection - Alert sources periodically check for new alerts
  2. Creation - Alerts are created in the database with unique instance IDs
  3. Linear Issue - Corresponding Linear ticket is created with proper metadata
  4. Tracking - Alert state is tracked through Linear issue status
  5. Resolution - Alerts are resolved when Linear tickets are closed

Configuration

Linear Settings

Configured in public/config.py: - Linear label IDs for different source types - Default resolution times - Team assignments - State mapping for issue lifecycle

Key constants: - MAX_ALERTS_CREATED_PER_DEFINITION - Limits alerts created per run (default: 10) - ONCALL_LINEAR_ISSUE_STATE_NAME_WHEN_CLOSED - State name for closed issues

Alert Owner Types

Defined in public/entities/alert_owner_type.py: - oncall - Engineering oncall team - intercom - Support team handling Intercom conversations - claims_management - Claims processing team

Alert Severities

Defined in public/entities/alert_severity.py: - critical - Immediate attention required - high - High priority - medium - Normal priority - low - Low priority

Usage

Creating Alert Definitions

Alert definitions are typically created through Eng Tools ⧉. Each definition requires:

  1. Source Configuration - Source-specific settings (SQL query, Datadog monitor ID, etc.)
  2. Owner Assignment - Team responsible for handling the alert
  3. Severity Level - Priority of the alert
  4. Schedule - Cron expression for periodic checks
  5. Linear Integration - Source type for Linear issue creation