README.md

# RaftKV

An Elixir library to store key-value pairs in a distributed, fault-tolerant, self-adjusting data structure.

- [API Documentation](https://hexdocs.pm/raft_kv/)
- [Hex package information](https://hex.pm/packages/raft_kv)

[![Hex.pm](http://img.shields.io/hexpm/v/raft_kv.svg)](https://hex.pm/packages/raft_kv)
[![Build Status](https://travis-ci.org/skirino/raft_kv.svg)](https://travis-ci.org/skirino/raft_kv)
[![Coverage Status](https://coveralls.io/repos/github/skirino/raft_kv/badge.svg?branch=master)](https://coveralls.io/github/skirino/raft_kv?branch=master)

## Feature & Design

- Each value can be arbitrary data structure.
    - Operation on a stored value must be given as an implementation of `RaftKV.ValuePerKey` behaviour.
- Built on top of [raft_fleet](https://github.com/skirino/raft_fleet).
    - Key-value pairs are sharded into multiple Raft consensus groups by hash partitioning.
- Based on number of keys, data size and current load, shards are automatically split/merged in a transparent manner.
- Designed for many key-value pairs and throughput.

## Usage

Suppose we have the following callback module for simple key-value store.
```ex
defmodule KV do
  alias RaftKV.ValuePerKey
  @behaviour ValuePerKey

  @impl true
  def command(_previous_value, _size, _key, {:set, value}) do
    {:ok, 5, value, 0}
  end
  def command(_previous_value, _size, _key, :unset) do
    {:ok, 5, nil, 0}
  end

  @impl true
  def query(value, _size, _key, :get) do
    {value, 1}
  end

  #
  # API
  #
  def get(k) do
    case RaftKV.query(:kv, k, :get) do
      {:ok, v}                 -> v
      {:error, :key_not_found} -> nil
    end
  end

  def set(k, v) do
    {:ok, :ok} = RaftKV.command(:kv, k, {:set, v})
    :ok
  end

  def unset(k) do
    {:ok, :ok} = RaftKV.command(:kv, k, :unset)
    :ok
  end
end
```

Let's initialize `:raft_kv` and then register a keyspace.
```ex
RaftFleet.activate("some_zone")
RaftKV.init()
RaftKV.register_keyspace(:kv, KV, nil, %RaftKV.SplitMergePolicy{max_shards: 16, max_keys_per_shard: 100})
```

Now we can get/set values with arbitrary keys.
```ex
KV.get("foo")        # => nil
KV.set("foo", "bar") # => :ok
KV.get("foo")        # => "bar"
```

Initially there's only one shard.
```ex
RaftKV.reduce_keyspace_shard_names(:kv, [], &[&1 | &2]) # => [:kv_0]
```

When the number of key-value pairs exceeds the limit per shard (`100`) as follows...
```ex
Enum.each(1..200, fn i -> KV.set("#{i}", i) end)
```

then shards are automatically split.
(Depending on the configurations it may take several minutes. See `RaftKV.Config` for more detail.)
```ex
RaftKV.reduce_keyspace_shard_names(:kv, [], &[&1 | &2]) # => [:kv_100663296, :kv_67108864, :kv_0]
```

Similarly, when you remove key-value pairs...
```ex
Enum.each(1..200, fn i -> KV.unset("#{i}") end)
```

the shards are automatically merged.
```ex
RaftKV.reduce_keyspace_shard_names(:kv, [], &[&1 | &2]) # => [:kv_0]
```

Shard splits and merges are transparent from client processes interacting with key-value pairs.