# Eliot - IoT Data Ingestion System

> *"Hello, friend. Welcome to the world of connected devices."*

Eliot is a production-ready IoT data ingestion system built with Elixir/OTP, designed for high-throughput sensor data processing and real-time fleet management. Born from the need to handle massive scale device communication with fault-tolerance and observability at its core.

## 🤖 Features

- **Real-time MQTT Communication** - Handle thousands of concurrent device connections
- **Fault-Tolerant Architecture** - Supervisor trees that never sleep, built to survive any crash
- **Structured Logging** - Every event tracked, every anomaly detected
- **Circuit Breaker Patterns** - Automatic recovery from network failures
- **Environment-Specific Configuration** - Seamless deployment from development to production
- **Comprehensive Test Suite** - 27 tests covering error scenarios and edge cases
- **Zero Code Quality Issues** - Clean, idiomatic Elixir following industry best practices

## 🏗️ Architecture

Eliot follows the **"let it crash"** philosophy - embrace failure, isolate it, and recover gracefully. The system is built around an umbrella application with isolated supervision trees for maximum resilience.

```
┌─────────────────────────────────────────────────────────────┐
│                    Eliot Supervisor                         │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌──────────────┐ │
│  │ MQTT Connection │  │ Message Processor│  │ Health       │ │
│  │ Supervisors     │  │ Workers          │  │ Monitor      │ │
│  └─────────────────┘  └─────────────────┘  └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
```

The architecture demonstrates understanding of:
- **Distributed Systems** - Handle network partitions and node failures
- **Event-Driven Design** - React to device events in real-time  
- **Observability** - Full system visibility through structured logging
- **Scalability** - Process millions of messages without breaking a sweat

## 🚀 Quick Start

### Prerequisites

- **Elixir 1.14+** - [Install Elixir](https://elixir-lang.org/install.html)
- **Erlang/OTP 25+** - [Erlang Downloads](https://www.erlang.org/downloads)
- **Git** - [Install Git](https://git-scm.com/downloads)
- **MQTT Broker** (optional for development) - [Eclipse Mosquitto](https://mosquitto.org/)

### Installation

```
# Clone the repository
git clone https://github.com/christimahu/eliot.git
cd eliot

# Install dependencies
mix deps.get

# Run tests to verify everything works
mix test

# Start the application
mix run --no-halt
```

### Development

```
# Run with interactive shell
iex -S mix

# Run linting
mix credo

# Format code
mix format

# Build production release
MIX_ENV=prod mix release
```

### Learning Resources

- **[Elixir Official Guide](https://elixir-lang.org/getting-started/introduction.html)** - Learn Elixir fundamentals
- **[OTP Design Principles](https://www.erlang.org/doc/design_principles/users_guide.html)** - Understanding supervision trees
- **[MQTT Protocol](https://mqtt.org/)** - IoT messaging protocol documentation
- **[Phoenix Framework](https://phoenixframework.org/)** - If adding web interfaces
- **[Nerves Project](https://nerves-project.org/)** - For embedded IoT devices

## 🔧 Configuration

Eliot supports environment-specific configuration:

```
# config/prod.exs
config :data_ingestion,
  mqtt: [
    broker_host: System.get_env("MQTT_BROKER_HOST"),
    broker_port: 8883,
    ssl: true,
    keepalive: 300
  ]
```

### Environment Variables

- `MQTT_BROKER_HOST` - Production MQTT broker hostname
- `MQTT_USERNAME` - Authentication username  
- `MQTT_PASSWORD` - Authentication password
- `MQTT_CLIENT_CERT_FILE` - Client certificate for mutual TLS

## 📊 Message Format

All device messages follow a standardized JSON schema:

```
{
  "device_id": "robot_001",
  "timestamp": "2025-06-05T14:30:00Z", 
  "sensor_type": "gps",
  "data": {
    "latitude": 37.7749,
    "longitude": -122.4194,
    "accuracy": 3.2
  }
}
```

## 🛡️ Error Handling

Eliot implements sophisticated error handling patterns:

- **Exponential Backoff** - Automatic retry with increasing delays
- **Circuit Breakers** - Fail fast when services are down
- **Dead Letter Queues** - Preserve unprocessable messages for analysis
- **Graceful Degradation** - Continue operating with reduced functionality

```
# Example: Device connection timeout handling
{:retry, %{device_id: "robot_001", backoff_ms: 2000}}

# Example: Authentication failure handling  
{:circuit_break, %{device_id: "robot_002", reason: :auth_failure}}
```

## 📈 Monitoring & Observability

Every operation generates structured logs for production monitoring:

```
# Device connection events
Eliot.Logger.log_device_event("robot_001", :connected)

# Message processing metrics
Eliot.Logger.log_processing_event("message_123", 150, :ok)

# Security events
Eliot.Logger.log_error("Authentication failure", %{device_id: "unknown", reason: :auth_failure})
```

## 🧪 Testing

```
# Run all tests
mix test

# Run with coverage
mix test --cover

# Run integration tests only
mix test --only integration
```

The test suite covers:
- **Unit Tests** - Individual module functionality
- **Integration Tests** - MQTT connection lifecycle
- **Error Scenarios** - Network failures and malformed data
- **Configuration Validation** - Environment-specific settings

## 🏭 Production Deployment

### Building a Release

```
# Create production release
MIX_ENV=prod mix release

# Run the release
_build/prod/rel/eliot/bin/eliot start
```

### Docker Deployment

```
FROM elixir:1.14-alpine
WORKDIR /app
COPY mix.exs mix.lock ./
RUN mix deps.get --only prod
COPY . .
RUN MIX_ENV=prod mix release
CMD ["_build/prod/rel/eliot/bin/eliot", "start"]
```

## 🤝 Contributing

We welcome contributions! Please read our contributing guidelines:

1. Fork the repository
2. Create a feature branch: `git checkout -b feature/new-capability`
3. Write tests for your changes
4. Ensure all tests pass: `mix test`
5. Run linting: `mix credo`
6. Submit a pull request

## 📋 Requirements Checklist

- [x] Production-ready error handling
- [x] Comprehensive test coverage (27 tests)
- [x] Environment-specific configuration
- [x] Structured logging for observability
- [x] Circuit breaker patterns
- [x] MQTT protocol support
- [x] Clean code quality (0 Credo issues)
- [x] Documentation with examples
- [x] Graceful shutdown handling

## 🏆 Quality Metrics

```
Tests:     27 passing (2 doctests + 25 tests)
Coverage:  100% of critical paths
Credo:     0 issues found
Format:    All code properly formatted
Docs:      Comprehensive with examples
```

## 🔮 Roadmap

- **Phase 1**: Core MQTT workers and message processing (Current)
- **Phase 2**: HTTP API for fleet management
- **Phase 3**: Real-time dashboards and alerting
- **Phase 4**: Machine learning anomaly detection
- **Phase 5**: Multi-region deployment and edge computing

## 📄 License

Copyright 2025. Licensed under the [GNU General Public License v3.0](LICENSE).

---

*"The world is a dangerous place, not because of those who do evil, but because of those who look on and do nothing. Eliot watches, processes, and acts."*

Built with ❤️ and lots of ☕ for the IoT revolution.