Skip to content

Alerting Component

System alerting and notification management component that provides a unified interface for handling alerts from multiple sources and managing their lifecycle through Linear integration.

Overview

The alerting component centralizes alert management across different monitoring systems, automatically creating and tracking Linear issues for engineering teams. It supports multiple alert sources and provides a consistent workflow for alert resolution.

[!NOTE]
This component doesn't provide a public API: * alerts are defined through Eng Tools ⧉ * this component is responsible for executing alert checks and creating Linear issues following the configuration.

Key Features

Alert Sources

The component supports multiple alert sources:

  • Database Alerts - SQL-based alerts defined in the database
  • Datadog Alerts - Monitoring alerts from Datadog
  • Datadog Security Signals - Security events from Datadog
  • Turing Alerts - Custom alerts from internal Turing system

Linear Integration

Automatically creates and manages Linear issues for alerts:

  • Creates Linear tickets with appropriate labels and teams
  • Tracks alert resolution status
  • Configurable resolution times based on alert source
  • Supports oncall and intercom conversation workflows

Alert Lifecycle

  1. Detection - Alert sources periodically check for new alerts
  2. Creation - Alerts are created in the database with unique instance IDs
  3. Linear Issue - Corresponding Linear ticket is created with proper metadata
  4. Tracking - Alert state is tracked through Linear issue status
  5. Resolution - Alerts are resolved when Linear tickets are closed

Configuration

Linear Settings

Configured in public/config.py: - Linear label IDs for different source types - Default resolution times - Team assignments - State mapping for issue lifecycle

Key constants: - MAX_ALERTS_CREATED_PER_DEFINITION - Limits alerts created per run (default: 10) - ONCALL_LINEAR_ISSUE_STATE_NAME_WHEN_CLOSED - State name for closed issues

Alert Owner Types

Defined in public/entities/alert_owner_type.py: - oncall - Engineering oncall team - intercom - Support team handling Intercom conversations - claims_management - Claims processing team

Alert Severities

Defined in public/entities/alert_severity.py: - critical - Immediate attention required - high - High priority - medium - Normal priority - low - Low priority

Templating & per-row overrides

SQL-backed sources (database_alert, turing_alert) can customize each Linear issue based on the row that triggered it — either by rendering Jinja templates against the row, or by emitting reserved _override_* columns that override the definition's defaults.

[!NOTE] Datadog-backed sources don't expose row columns, so templating and overrides only apply to database_alert and turing_alert.

Jinja templates in name, description, linear_scope_label_id

These three fields on an AlertDefinition are rendered as Jinja2 templates before the Linear issue is created.

Available context variables: - {{ instance_id }} — the row's id (postgres) / ID (turing) - {{ instance_title }} — human-readable title if the source supplies one - Any column returned by the SQL query — {{ company_name }}, {{ stuck_at }}, etc.

Example name template: Stuck claim for {{ company_name }}.

Syntax is validated at definition-save time (_validate_template_syntax in internal/business_logic/actions/alert_definition.py). Render failures at alert time (e.g. referencing a column the query didn't return) are logged and fall back to the raw template — the literal {{ var }} ends up in Linear, which is loud enough for a human to spot.

_override_* magic columns (SQL sources only)

The SQL query may emit any of these reserved columns to override the definition's defaults on a per-row basis. Missing or NULL values fall through to the definition.

Column Type Overrides
_override_owner_type AlertOwnerType enum string definition.owner_type (changes Linear team + labels)
_override_severity AlertSeverity enum string definition.severity (changes Linear priority)
_override_resolution_time_in_hours int definition.base_resolution_time_in_hours (SLA breach time)
_override_linear_scope_label_id label UUID string definition.linear_scope_label_id
_override_linear_project_id project UUID string definition.linear_project_id

Parsing rules (see internal/business_logic/actions/sources/overrides.py): - Invalid enum / non-int values are logged and silently ignored — the definition's default kicks back in. - Magic columns are stripped from the row data before it's displayed in the Linear comment footer.

[!WARNING] _override_owner_type parks a Linear issue on a different team than the definition's default. The auto-close job (close_resolved_alerts_linear_issue) only scans the definition's team, so issues created via this override are not auto-closed when the alert resolves.

Usage

Creating Alert Definitions

Alert definitions are typically created through Eng Tools ⧉. Each definition requires:

  1. Source Configuration - Source-specific settings (SQL query, Datadog monitor ID, etc.)
  2. Owner Assignment - Team responsible for handling the alert
  3. Severity Level - Priority of the alert
  4. Schedule - Cron expression for periodic checks
  5. Linear Integration - Source type for Linear issue creation