# Firehose

This library helps to manage data sending to AWS Firehose dealing with its limitations:

> The maximum size of a record sent to Kinesis Data Firehose, before base64-encoding, is 1,000 KiB.
> The PutRecordBatch operation can take up to 500 records per call or 4 MiB per call, whichever is smaller...the size of each record rounded up to the nearest 5KB

## Installation

If [available in Hex](, the package can be installed
by adding `firehose` to your list of dependencies in `mix.exs`:

def deps do
    {:firehose, "~> 1.0.0"}

## Usage

Create module which will handle callbacks:

defmodule Event.Handler do
  def subscribe(:failed, %Firehose.Batch{} = batch, aws_response) do
    # This callback is triggered when retry count exhausted and batch not delivered.
    # You can add specific logic for this case (write data to queue or database etc) or
    # send metrics to monitoring service

  def subscribe(:completed, %Firehose.Batch{stream: stream, size: batch_size, records: records}) do
    # This callback is triggered after each successfull delivery to Firehose.
    # You can use this callback to send metrics to monitoring service

Add appropriate configurations for `ex_aws` and `firehose`:

config :ex_aws,
  access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role],
  secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]

config :firehose, Firehose.Manager,
  handlers: [Event.Handler],
  flush_interval: 1_000, # Flush batches every seconds or on batch size limit
  retries: 5,            # Retry count for failed requests
  serializer: Poison,    # Event serializer: should respond to `encode` and `decode` methods
  delimiter: "\n",       # Add delimiter after each event or add nothing if `false` or `nil` set
  remove_empty_keys: true# All keys without values (nil or blank string) will be removed to reduce event's size

Add your manager to your supervisor and thats all. You can emit events and efficiently send them to AWS Firehose:

Firehose.emit("logs", %{"name" => "pageview", "ua" => "..."})

Documentation can be generated with [ExDoc](
and published on [HexDocs]( Once published, the docs can
be found at [](