Eliot - IoT Data Ingestion System
View Source"Hello, friend. Welcome to the world of connected devices."
Eliot is a production-ready IoT data ingestion system built with Elixir/OTP, designed for high-throughput sensor data processing and real-time fleet management. Born from the need to handle massive scale device communication with fault-tolerance and observability at its core.
๐ค Features
- Real-time MQTT Communication - Handle thousands of concurrent device connections
- Fault-Tolerant Architecture - Supervisor trees that never sleep, built to survive any crash
- Structured Logging - Every event tracked, every anomaly detected
- Circuit Breaker Patterns - Automatic recovery from network failures
- Environment-Specific Configuration - Seamless deployment from development to production
- Comprehensive Test Suite - 27 tests covering error scenarios and edge cases
- Zero Code Quality Issues - Clean, idiomatic Elixir following industry best practices
๐๏ธ Architecture
Eliot follows the "let it crash" philosophy - embrace failure, isolate it, and recover gracefully. The system is built around an umbrella application with isolated supervision trees for maximum resilience.
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Eliot Supervisor โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ
โ โ MQTT Connection โ โ Message Processorโ โ Health โ โ
โ โ Supervisors โ โ Workers โ โ Monitor โ โ
โ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
The architecture demonstrates understanding of:
- Distributed Systems - Handle network partitions and node failures
- Event-Driven Design - React to device events in real-time
- Observability - Full system visibility through structured logging
- Scalability - Process millions of messages without breaking a sweat
๐ Quick Start
Prerequisites
- Elixir 1.14+ - Install Elixir
- Erlang/OTP 25+ - Erlang Downloads
- Git - Install Git
- MQTT Broker (optional for development) - Eclipse Mosquitto
Installation
# Clone the repository
git clone https://github.com/christimahu/eliot.git
cd eliot
# Install dependencies
mix deps.get
# Run tests to verify everything works
mix test
# Start the application
mix run --no-halt
Development
# Run with interactive shell
iex -S mix
# Run linting
mix credo
# Format code
mix format
# Build production release
MIX_ENV=prod mix release
Learning Resources
- Elixir Official Guide - Learn Elixir fundamentals
- OTP Design Principles - Understanding supervision trees
- MQTT Protocol - IoT messaging protocol documentation
- Phoenix Framework - If adding web interfaces
- Nerves Project - For embedded IoT devices
๐ง Configuration
Eliot supports environment-specific configuration:
# config/prod.exs
config :data_ingestion,
mqtt: [
broker_host: System.get_env("MQTT_BROKER_HOST"),
broker_port: 8883,
ssl: true,
keepalive: 300
]
Environment Variables
MQTT_BROKER_HOST
- Production MQTT broker hostnameMQTT_USERNAME
- Authentication usernameMQTT_PASSWORD
- Authentication passwordMQTT_CLIENT_CERT_FILE
- Client certificate for mutual TLS
๐ Message Format
All device messages follow a standardized JSON schema:
{
"device_id": "robot_001",
"timestamp": "2025-06-05T14:30:00Z",
"sensor_type": "gps",
"data": {
"latitude": 37.7749,
"longitude": -122.4194,
"accuracy": 3.2
}
}
๐ก๏ธ Error Handling
Eliot implements sophisticated error handling patterns:
- Exponential Backoff - Automatic retry with increasing delays
- Circuit Breakers - Fail fast when services are down
- Dead Letter Queues - Preserve unprocessable messages for analysis
- Graceful Degradation - Continue operating with reduced functionality
# Example: Device connection timeout handling
{:retry, %{device_id: "robot_001", backoff_ms: 2000}}
# Example: Authentication failure handling
{:circuit_break, %{device_id: "robot_002", reason: :auth_failure}}
๐ Monitoring & Observability
Every operation generates structured logs for production monitoring:
# Device connection events
Eliot.Logger.log_device_event("robot_001", :connected)
# Message processing metrics
Eliot.Logger.log_processing_event("message_123", 150, :ok)
# Security events
Eliot.Logger.log_error("Authentication failure", %{device_id: "unknown", reason: :auth_failure})
๐งช Testing
# Run all tests
mix test
# Run with coverage
mix test --cover
# Run integration tests only
mix test --only integration
The test suite covers:
- Unit Tests - Individual module functionality
- Integration Tests - MQTT connection lifecycle
- Error Scenarios - Network failures and malformed data
- Configuration Validation - Environment-specific settings
๐ญ Production Deployment
Building a Release
# Create production release
MIX_ENV=prod mix release
# Run the release
_build/prod/rel/eliot/bin/eliot start
Docker Deployment
FROM elixir:1.14-alpine
WORKDIR /app
COPY mix.exs mix.lock ./
RUN mix deps.get --only prod
COPY . .
RUN MIX_ENV=prod mix release
CMD ["_build/prod/rel/eliot/bin/eliot", "start"]
๐ค Contributing
We welcome contributions! Please read our contributing guidelines:
- Fork the repository
- Create a feature branch:
git checkout -b feature/new-capability
- Write tests for your changes
- Ensure all tests pass:
mix test
- Run linting:
mix credo
- Submit a pull request
๐ Requirements Checklist
- [x] Production-ready error handling
- [x] Comprehensive test coverage (27 tests)
- [x] Environment-specific configuration
- [x] Structured logging for observability
- [x] Circuit breaker patterns
- [x] MQTT protocol support
- [x] Clean code quality (0 Credo issues)
- [x] Documentation with examples
- [x] Graceful shutdown handling
๐ Quality Metrics
Tests: 27 passing (2 doctests + 25 tests)
Coverage: 100% of critical paths
Credo: 0 issues found
Format: All code properly formatted
Docs: Comprehensive with examples
๐ฎ Roadmap
- Phase 1: Core MQTT workers and message processing (Current)
- Phase 2: HTTP API for fleet management
- Phase 3: Real-time dashboards and alerting
- Phase 4: Machine learning anomaly detection
- Phase 5: Multi-region deployment and edge computing
๐ License
Copyright 2025. Licensed under the GNU General Public License v3.0.
"The world is a dangerous place, not because of those who do evil, but because of those who look on and do nothing. Eliot watches, processes, and acts."
Built with โค๏ธ and lots of โ for the IoT revolution.