# High Dynamic Range (HDR) Histogram for Elixir

Get percentile values from stream of input. Each histogram has a configurable
min, max and precision value which control both the accuracy and memory
requirements.

Learn more about [HDR Histograms](http://hdrhistogram.org/).

Writes (which ought to substantially outnumber reads) are in constant time
regardless of the configuration. Querying a histogram for a specific percentile,
or other metric (max value, min value) is slightly slower as the range and/or
precision grows. However, the implementation relies on write-optimized ETS
tables and thus operations are not serialized (beyond the locks used by ETS).

A histogram with a range of 1..1_000_000 and a precision of 3 takes roughly
16000 bytes.

## Usage

First, add `histogrex` to your`mix.exs` file:

```
def deps do
  [{:doma_histogrex, "~> 0.0.5"}]
end
```

Next, create a registry:

```
defmodule MyApp.Stats do
  use Histogrex

  histogrex :load_user, min: 1, max: 10_000_000, precision: 3
  histogrex :db_save_settings, min: 1, max: 10_000, precision: 2
  ...
end
```

And then add this module as a `worker` to your application's supervisor
tree:

```
worker(MyApp.Stats, [])
```

Values can then be recorded via the `record!` or `record` functions:

```
alias MyApp.Stats

Stats.record!(:load_user, 233)
Stats.record!(:db_save_settings, 84)
```

`min`, `max`, `total_count` and `value_at_quantile` are used to query the
histogram:

```
alias MyApp.Stats
Stats.mean(:load_user)
Stats.max(:db_save_settings)
Stats.total_count(:db_save_settings)
Stats.value_at_quantile(:load_user, 99.9)
```

It would be reasonable to have a GenServer dump these statistics to some log
ingestor every X seconds (10? 60?). This would be the only reader (though
concurrent reads are fully supported).

## Query Multiple Metrics at Once
If you're querying multiple metrics from an histogram at once, you can get a
slight performance boost by first getting an iterator and querying that.
There is no need to reset the iterator between calls:

```
it = Stats.iterator(:load_user)
Stats.mean(it)
Stats.max(it)
Stats.total_count(it)
Stats.value_at_quantile(it, 99.9)
```

## Dynamic Metrics
You won't always be able to define histograms upfront. In these cases, use templates:

```
defmodule MyApp.Stats do
  use Histogrex

  template :http_ms, min: 1, max: 10_000, precision: 2
  ...
end
```

The `template` macro takes the same arguments as the `histogrex` macro and, the
two can live side by side. However, all your metric names must be unique.

Overloaded functions take the template name and your dynamic metric name::

```
alias MyApp.Stats
Stats.record!(:http_ms, "about", 82)

Stats.mean(:http_ms, "about")
Stats.max(:http_ms, "about")
Stats.total_count(:http_ms, "about")
Stats.value_at_quantile(:http_ms, "about", 99.9)
```

The dynamic metric name can be a string or atom.

You can also get an iterator and use that for efficient multi-queries:

```
it = Stats.iterator(:http_ms, "about")
Stats.max(it)
Stats.mean(it)
```

## Collecting All Data
Use your registry's `reduce/2` function to collect the data:

```
Stats.reduce(%{}, fn {name, iterator}, metrics ->
  collected = {Stats.max(iterator), Stats.value_at_quantile(iterator, 99.9)}
  Map.put(metrics, name, collected)
end)
```

## Implementation

The core histogram implementation is taken from [the Go version](https://github.com/codahale/hdrhistogram).

In order to maintain high write throughput, data access is not serialized through
a single process. Instead, write-optimized (via the `write_concurrency: true` option)
ETS tables are used. Functions are fully executed by the calling process. A write consists of as single `:ets.update_counter` call. A read consists of a single `:ets.lookup`.

It is possible to use multiple registries, though I suspect this will have no impact
unless thousands or metrics with high load are used. Benchmark it.

You can get the memory requirements of a registry by calling:

```
:ets.info(MyApp.Stats)[:memory]
```

There's a bit of additional overhead for each histogram, but that will give you
the bulk of it.