Skip to content

Async Exports Component

Table of Contents

Context

  • Scope: Global
  • Ownership: Health Insurance Affiliation & Global Customer Dashboard
  • Purpose: Provides asynchronous export capabilities for large datasets across countries
  • Architecture: Global component with scope-based routing (global APIs + legacy dependency injection)
  • Tech framing: Global async export component #30249 ⧉

Overview

The async exports component enables the asynchronous generation and processing of large data exports (CSV, Excel, etc.) without blocking web requests. It provides a complete workflow from request creation to file delivery via email notifications.

Usage

This is the preferred approach as we move toward globalization. Global exports call directly to global component public APIs:

  1. Add your export type to the ExportType enum and ExportConfig.EXPORT_DEFINITIONS
  2. Implement the export function in your global component's public API
  3. Add the function call to the global export handler in the worker
  4. Use the REST API to create export requests

For Local Components (Legacy Pattern)

Use this pattern only for country-specific functionality that are not globalized yet:

  1. Create a local dependency implementation by extending AsyncExportsDependency
  2. Implement the generate_export method with your country-specific export logic
  3. Register the dependency during app initialization
  4. Use the REST API to create export requests

Implementation Guide

Step 1: Add New Export Type Configuration

All export types must be configured in components/async_exports/public/enums/export_type.py:

class ExportType(AlanBaseEnum):
    employees = "employees"
    exemptions = "exemptions"
    your_new_export = "your_new_export"  # Add your export type here

class ExportConfig:
    EXPORT_DEFINITIONS: dict[ExportType, ExportConfiguration] = {
        # Existing exports...
        ExportType("your_new_export"): ExportConfiguration(
            scope="global",  # "global" for global components, "local" for country-specific
            retention_period_days=7  # How long to keep the export files before deletion
        ),
    }

For global exports:

  1. Implement function in components/your_global_component/public/actions/export.py
  2. Add import and call in components/async_exports/internal/workers/export_worker.py in the _generate_global_component_export function

Function signature must match the Interface Reference above.

Step 2B: Local Component Implementation (Legacy)

For local exports:

  1. Create dependency class extending AsyncExportsDependency in components/{country}/bootstrap/dependencies/async_exports.py
  2. Implement generate_export method matching the Interface Reference signature
  3. Register dependency in your app's components_config.py using set_app_dependency

See France implementation example: components/fr/bootstrap/dependencies/async_exports.py

Step 3: Adapt Email notifications

  1. Add your translation and link for the email notifications in components/async_exports/notification/fr/generator.py so that your new export type can be matched to the correct email content.

Step 4: API Usage Examples

Create Export Request (POST): 

{
  "export_type": "your_export_type",
  "file_format": "csv",
  "filters": {"status": "active"},
  "context_account_id": "account-uuid",
  "company_ids": ["123", "456"]
}

List Export Results (GET): 

GET /api/async_exports?export_type=your_export_type&limit=10

Architecture

Key Components

  • ExportRequest/ExportResult: Database models for tracking export lifecycle
  • GeneratedExportData: Interface data transfer object
  • Export Worker: Background job processor (Redis Queue)
  • S3 Service: File storage and URL generation
  • Email Notifications: User notifications for completed exports

Flow

  1. API Call → Create ExportRequest → Queue background job
  2. Worker → Determine scope → Call global API or local dependency
  3. Storage → Upload to S3 → Email notification → Auto-cleanup

Interface Reference

Export Function Signature (Both Global & Local)

Both global and local implementations must match this signature:

generate_export(filters, file_format, company_ids, operational_scope_ids) → GeneratedExportData

Parameters & Returns

Input Parameters: - filters: Optional filtering criteria as key-value pairs - file_format: Output format (CSV, Excel, etc.) - company_ids: List of company IDs for authorization context - operational_scope_ids: List of operational scope IDs for filtering

Returns GeneratedExportData: - file_content: File stream/buffer with the export data - mimetype: MIME type for the generated file - items_count: Number of items in the export

Implementation Requirements

  1. Authorization: Respect provided company and operational scope IDs
  2. Filtering: Handle the filters parameter for user-defined criteria
  3. Error Handling: Raise exceptions that will be caught by the worker
  4. File Formats: Support at least CSV, consider Excel for complex data

REST API Reference

Base URL: /api/async_exports

POST - Create Export

  • Creates new export request
  • Returns export ID for tracking

GET - List Exports

  • Lists user's export requests with filtering
  • Returns export metadata, processing status and outcome

Full API Documentation: Admin Tools API ⧉

Configuration

  • Queue: Redis Queue with retry policy and dedicated async_exports queue
  • Storage: S3 for temporary file storage
  • S3 Buckets: async-exports-prod, async-exports-demo, async-exports-acceptance,
  • Retention: Per-export-type retention periods in ExportConfig.EXPORT_DEFINITIONS
  • Email: Notifications sent for exports taking >45 seconds

Testing

Unit Tests: Test export implementation with mocked dependencies
Integration Tests: Test via REST API with real workflow

Monitoring

  • APM: Datadog monitoring ⧉
  • Logs: Structured with export_request_id context
  • Alerts: Sentry error tracking
  • Metrics: Success rates, processing times, file sizes

Common Pitfalls

  1. Memory: Stream data instead of loading everything into memory
  2. Timeouts: Implement chunking for very large exports
  3. Authorization: Always validate company/scope access
  4. Configuration: Add export types to ExportConfig.EXPORT_DEFINITIONS
  5. Scope Choice: Use "global" unless you have country-specific needs

References

  • Global Example: components/global_customer_dashboard/public/actions/employee
  • Local Example:
  • components/fr/bootstrap/dependencies/async_exports.py
  • apps/fr_api/components_config.py
  • Worker: components/async_exports/internal/workers/export_worker.py
  • Full Documentation ⧉ (available after next docs build)
  • Tech Framing: Global async export component #30249 ⧉