Search documentation of phoenix_kit

# PhoenixKit.Emails.Archiver (phoenix_kit v1.6.19)

View Source

Archive and compress old email tracking data for optimal storage.

Provides comprehensive data lifecycle management for email tracking:

- Body Compression - Compress full email bodies after configurable time
- S3 Archival - Move old logs to S3 cold storage
- Sampling Optimization - Apply sampling to reduce storage load
- Cleanup Integration - Work with cleanup tasks for complete lifecycle
- Performance Optimization - Batch operations for large datasets

## Storage Optimization Strategy

1. Recent Data (0-7 days): Full storage with all fields
2. Medium Data (7-30 days): Compress body_full, keep metadata
3. Old Data (30-90 days): Archive to S3, keep local summary
4. Ancient Data (90+ days): Delete after S3 confirmation

## Settings Integration

All archival settings stored in phoenix_kit_settings:

- `email_compress_body` - Days before compressing bodies (default: 30)
- `email_archive_to_s3` - Enable S3 archival (default: false)
- `email_s3_bucket` - S3 bucket name
- `email_sampling_rate` - Percentage to fully log (default: 100)
- `email_retention_days` - Total retention before deletion (default: 90)

## Usage Examples

    # Compress bodies older than 30 days
    {compressed_count, size_saved} = PhoenixKit.Emails.Archiver.compress_old_bodies(30)

    # Archive to S3 with automatic cleanup
    {:ok, archived_count} = PhoenixKit.Emails.Archiver.archive_to_s3(90,
      bucket: "my-email-archive",
      prefix: "email-logs/2025/"
    )

    # Apply sampling to reduce future storage
    sampled_email = PhoenixKit.Emails.Archiver.apply_sampling_rate(email)

    # Get storage statistics
    stats = PhoenixKit.Emails.Archiver.get_storage_stats()
    # => %{total_logs: 50000, compressed: 15000, archived: 10000, size_mb: 2341}

## S3 Integration

Supports multiple S3-compatible storage providers:

- Amazon S3
- DigitalOcean Spaces
- Google Cloud Storage
- MinIO
- Any S3-compatible service

## Compression Algorithm

Uses gzip compression for email bodies with fallback strategies:

1. Gzip - Primary compression for text content
2. Preview Only - Keep only first 500 chars for very old data
3. Metadata Only - Keep only delivery status and timestamps

## Batch Processing

All operations are designed for efficiency:

- Process in configurable batch sizes (default: 1000)
- Progress tracking for long operations
- Automatic retry on transient failures
- Memory-efficient streaming for large datasets

# Summary

## Functions

apply_sampling_rate(email_attrs, opts \\ [])

Apply sampling rate to email for storage optimization.

archive_to_s3(days_old, opts \\ [])

Archive old emails to S3 storage.

compress_old_bodies(days_old \\ nil, opts \\ [])

Compress email bodies older than specified days.

get_storage_breakdown()

Get detailed storage breakdown by time periods.

get_storage_stats()

Get comprehensive storage statistics.

# Functions

# apply_sampling_rate(email_attrs, opts \\ [])

Apply sampling rate to email for storage optimization.

Returns modified email with reduced storage footprint for non-critical emails.

## Sampling Strategy

- Always Full: Error emails, bounces, complaints
- Always Full: Transactional emails (password resets, etc.)
- Sampling Applied: Marketing emails, newsletters
- Metadata Only: Bulk emails when over limit

## Examples

    # Apply system sampling rate
    email = Archiver.apply_sampling_rate(original_email)

    # Force specific sampling
    email = Archiver.apply_sampling_rate(original_email, force_rate: 50)

# archive_to_s3(days_old, opts \\ [])

Archive old emails to S3 storage.

Returns `{:ok, archived_count}` on success or `{:error, reason}` on failure.

## Options

- `:bucket` - S3 bucket name (required)
- `:prefix` - S3 object key prefix
- `:batch_size` - Process in batches (default: 500)
- `:format` - Archive format: :json (default), :csv, :parquet
- `:delete_after_archive` - Delete from DB after successful archive
- `:include_events` - Include email events in archive

## Examples

    # Basic S3 archival
    {:ok, count} = Archiver.archive_to_s3(90,
      bucket: "email-archive",
      prefix: "logs/2025/"
    )

    # Archive with events and cleanup
    {:ok, count} = Archiver.archive_to_s3(90,
      bucket: "email-archive",
      include_events: true,
      delete_after_archive: true
    )

# compress_old_bodies(days_old \\ nil, opts \\ [])

Compress email bodies older than specified days.

Returns `{compressed_count, size_saved_bytes}`.

## Options

- `:batch_size` - Process in batches (default: 1000)
- `:dry_run` - Show what would be compressed without doing it
- `:preserve_errors` - Don't compress emails with errors/bounces

## Examples

    # Compress bodies older than 30 days
    {count, saved} = Archiver.compress_old_bodies(30)
    # => {1523, 45231040}

    # Dry run to see impact
    {count, estimated_saved} = Archiver.compress_old_bodies(30, dry_run: true)

# get_storage_breakdown()

Get detailed storage breakdown by time periods.

## Examples

    iex> Archiver.get_storage_breakdown()
    %{
      last_7_days: %{logs: 5000, size_mb: 145, compressed: false},
      last_30_days: %{logs: 15000, size_mb: 420, compressed: 8000},
      last_90_days: %{logs: 35000, size_mb: 980, compressed: 25000},
      older: %{logs: 70000, size_mb: 1200, archived: 45000}
    }

# get_storage_stats()

Get comprehensive storage statistics.

## Examples

    iex> Archiver.get_storage_stats()
    %{
      total_logs: 125000,
      total_events: 450000,
      compressed_bodies: 45000,
      archived_logs: 15000,
      storage_size_mb: 2341,
      oldest_log: ~U[2024-01-15 10:30:00Z],
      compression_ratio: 0.65,
      s3_archived_size_mb: 890
    }