<p align="center">
<img src="https://raw.githubusercontent.com/stefanzvkvc/chord/main/assets/chord.png" alt="Chord Logo" width="300">
</p>
Welcome to **Chord** - a flexible and powerful Elixir library designed to simplify context management and delta tracking in your distributed or real-time applications.
[](https://hex.pm/packages/chord) [](https://hexdocs.pm/chord)
## Why Chord?
When you need a solution for real-time state synchronization, partial updates, and efficient cleanup, Chord strikes the perfect note! Here’s what makes Chord special:
- **Seamless state sync**: Keep your clients up-to-date with full context or delta-based updates.
- **Customizable backend**: Use ETS, Redis, or your own backend implementation.
- **Flexible delta formatting**: Define how your updates are structured.
- **Periodic cleanup**: Automatically clear stale contexts or deltas.
- **Developer-friendly APIs**: Simple, consistent, and easy-to-use APIs.
- **Context export and restore**: Export contexts to or restore them from external providers.
- **Partial updates**: Apply updates to specific fields within a context.
- **Delta Tracking**: Efficiently track and retrieve state changes.
- **Flexible architecture**: Chord works in both stateful (via GenServer) and stateless modes (direct calls to backends like Redis or ETS). This flexibility makes it easier to adapt Chord to a variety of use cases.
---
## 🚀 Getting started
### Install the library
Add Chord to your Mix dependencies:
```elixir
def deps do
[
{:chord, "~> 0.1.4"}
]
end
```
Run:
```bash
mix deps.get
```
### Configure Chord
Add your desired configuration in `config/config.exs`:
```elixir
config :chord,
backend: Chord.Backend.ETS, # Choose your backend (Redis, ETS, etc.)
context_auto_delete: false, # Enable or disable auto-deletion of old contexts
context_ttl: 6 * 60 * 60, # Time-to-live for contexts
delta_ttl: 24 * 60 * 60, # Time-to-live for deltas
delta_threshold: 100, # Number of deltas to retain
delta_formatter: Chord.Delta.Formatter.Default, # Format for deltas
time_provider: Chord.Utils.Time, # Time provider for consistent timestamps
export_callback: nil, # Optional: Define a callback for exporting contexts
context_external_provider: nil # Optional: Define a function for fetching external contexts
```
---
## How to use Chord
In Chord, a **context** is basically a container for state. The term **“context”** might mean different things in various fields, but in Chord, it specifically means a **container for state**. Here are some examples to explain this idea:
- In a **chat application**, a context could be a group chat, including its details (e.g., participants, topic), and messages.
- In a **game session**, a context might hold the game’s state, like player positions, scores, and progress.
- In a **collaborative document editor**, a context could be the document’s state, keeping track of edits, updates, and collaborators.
With this understanding of the term, let's look at some practical examples.
### Setting a context
Define the global context and track changes with deltas.
```elixir
Chord.set_context("user:123", %{status: "online", metadata: %{theme: "light", language: "en-US"}})
{:ok,
%{
context: %{
version: 1,
context: %{
status: "online",
metadata: %{language: "en-US", theme: "light"}
},
inserted_at: 1737570501,
context_id: "user:123"
},
delta: %{
version: 1,
delta: %{
status: %{value: "online", action: :added},
metadata: %{value: %{language: "en-US", theme: "light"}, action: :added}
},
inserted_at: 1737570501,
context_id: "user:123"
}
}}
```
### Updating a context
Updates a portion of the global context associated with a specific identifier.
This function allows for partial modifications without affecting the entire context.
```elixir
Chord.update_context("user:123", %{metadata: %{theme: "dark"}})
{:ok,
%{
context: %{
version: 2,
context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
inserted_at: 1737570583,
context_id: "user:123"
},
delta: %{
version: 2,
delta: %{
metadata: %{
theme: %{value: "dark", action: :modified, old_value: "light"}
}
},
inserted_at: 1737570583,
context_id: "user:123"
}
}}
```
### Getting a context
Fetches the current state for a specified identifier.
```elixir
Chord.get_context("user:123")
{:ok,
%{
version: 2,
context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
inserted_at: 1737570583,
context_id: "user:123"
}}
```
### Synchronizing state
Synchronize the state for a given identifier.
Depending on the version the client has, it will receive either the full context, only the changes (deltas), or a notification that there are no updates.
```elixir
Chord.sync_context("user:123", nil)
{:full_context,
%{
version: 2,
context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
inserted_at: 1737570583,
context_id: "user:123"
}}
Chord.sync_context("user:123", 1)
{:delta,
%{
version: 2,
delta: %{
metadata: %{theme: %{value: "dark", action: :modified, old_value: "light"}}
},
inserted_at: 1737570583,
context_id: "user:123"
}}
Chord.sync_context("user:123", 2)
{:no_change, 2}
```
### Exporting a context
Save the current context for a specific identifier to external storage using the configured export callback.
#### Defining the export callback
To enable the export functionality, you need to define a callback function in your application. This function will handle how the context is exported (e.g., saving it to a database). Here’s an example:
```elixir
defmodule MyApp.ContextExporter do
@moduledoc """
Handles exporting contexts to external storage.
"""
@spec export_context(map()) :: :ok | {:error, term()}
def export_context(context_data) do
%{context_id: context_id, version: verion, context: context} = context_data
# Example: Save context_data to an external database or storage
case ExternalStorage.save(context_id, context, version) do
:ok -> :ok
{:error, reason} -> {:error, reason}
end
end
end
```
#### Configure the export callback
Next, configure the export callback in your application’s environment. This tells Chord how to handle context exports.
```elixir
# config/config.exs
config :chord, :export_callback, &MyApp.ContextExporter.export_context/1
```
#### Use Chord.export_context/1
Once the callback is configured, you can use function to export a specific context to external storage:
```elixir
Chord.export_context("user:123")
:ok
```
### Deleting a context
Removes the entire context and its associated deltas.
```elixir
Chord.delete_context("user:123")
:ok
```
### Restoring a context
Retrieve and restore a context from an external provider to the current backend.
#### Define the restore callback
First, define a module and function that will handle the logic for retrieving a context. For example:
```elixir
defmodule MyApp.ContextRestorer do
@moduledoc """
Handles restoring contexts from external storage.
"""
@spec restore_context(String.t()) :: {:ok, map()} | {:error, term()}
def restore_context(context_id) do
# Example: Retrieve the context from a database or other storage system
case ExternalStorage.get(context_id) do
{:ok, %{context: context, version: version}} -> {:ok, %{context: context, version: version}}
{:error, reason} -> {:error, reason}
end
end
end
```
#### Configure the restore callback
Next, configure the restore callback in your application’s environment. This tells Chord how to handle context restoration:
```elixir
# config/config.exs
config :chord, :context_external_provider, &MyApp.ContextRestorer.restore_context/1
```
#### Use Chord.restore_context/1
Once the callback is configured, you can use function to retrieve and restore a specific context:
```elixir
Chord.restore_context("user:1234")
{:ok,
%{
version: 10,
context: %{source: "external storage provider"},
inserted_at: 1737464001,
context_id: "user:1234"
}}
```
### Cleanup operations
Chord provides cleanup functionality to remove stale contexts and deltas. To enable and configure this feature, add the following settings to your application configuration:
#### Configuration options
```elixir
config :chord,
context_auto_delete: true, # Enable or disable auto-deletion of old contexts
context_ttl: 6 * 60 * 60, # Time-to-live for contexts (in seconds)
delta_ttl: 24 * 60 * 60, # Time-to-live for deltas (optional, in seconds)
delta_threshold: 100 # Number of delta versions to retain (optional)
```
#### How it works
- Context cleanup:
- Set **context_auto_delete: true** to enable context cleanup.
- Configure **context_ttl** to define how long contexts should remain in memory before being deleted. The value must be in seconds, as timestamps are created with second-level precision.
- When a context is deleted, all associated deltas are automatically cleaned up as well.
- Delta cleanup:
- To clean deltas by age, set **delta_ttl** to specify the maximum time deltas should remain in memory.
- To clean deltas by number, set **delta_threshold** to define the maximum number of deltas to retain.
#### Example usage
Run the cleanup process manually with:
```elixir
Chord.cleanup(limit: 50)
```
#### Future plans
In the future, you’ll be able to configure the time unit (e.g., seconds, milliseconds), which will be used when generating timestamps.
This configuration will apply to the function responsible for providing the current time, primarily used when setting inserted_at during data storage in memory.
If the configured time unit is set to seconds, related configurations such as context_ttl and delta_ttl will also need to be specified in seconds to ensure consistency.
### Managing the cleanup server
Start and manage the Cleanup Server for automated periodic cleanup:
```elixir
{:ok, _pid} = Chord.start_cleanup_server(interval: :timer.minutes(30))
Chord.update_cleanup_interval(:timer.minutes(60))
Chord.update_cleanup_backend_opts(limit: 100)
Chord.stop_cleanup_server()
```
---
## Customization
### Backends
Chord supports multiple backends out-of-the-box:
- **ETS** (In-Memory): No additional setup is required.
- **Redis** (Distributed): Requires a Redis instance and some configuration.
#### Using Redis as a backend
To use Redis as the backend for Chord, follow these steps:
1. **Start Redis**: Ensure a Redis server is running.
2. **Set up the Redis connection**: Start a Redis connection process using the Redix library, which is included with Chord:
```elixir
{:ok, _} = Redix.start_link("redis://localhost:6379", name: :my_redis)
```
3. **Configure Chord to use Redis**: Set the Redis client and backend in your application’s
```elixir
# config/config.exs
config :chord,
backend: Chord.Backend.Redis,
redis_client: :my_redis
```
You can also implement your own backend by adhering to the `Chord.Backend.Behaviour`.
### Delta formatters
Chord provides the ability to define custom delta formatters by implementing the `Chord.Delta.Formatter.Behaviour`. This feature is useful for tailoring how deltas (changes) are formatted to suit your application’s requirements.
#### Defining a custom delta formatter
To define a custom delta formatter, create a module that implements the `Chord.Delta.Formatter.Behaviour`:
```elixir
defmodule MyApp.CustomFormatter do
@moduledoc """
A custom delta formatter for Chord, demonstrating how to implement the behavior.
"""
@behaviour Chord.Delta.Formatter.Behaviour
@impl true
def format(delta, context_id) do
timestamp = DateTime.utc_now() |> DateTime.to_iso8601()
Enum.map(delta, fn {key, change} ->
base = %{
timestamp: timestamp,
context_id: context_id,
field: key,
action: change.action
}
case change.action do
:added ->
Map.put(base, :new_value, change.value)
:modified ->
Map.merge(base, %{old_value: change.old_value, new_value: change.value})
:removed ->
base
end
end)
end
end
```
#### Configuring Chord to use your delta formatter
Once you’ve defined your custom formatter, configure Chord to use it by setting it in the application environment:
```elixir
# config/config.exs
config :chord, :delta_formatter, MyApp.CustomFormatter
```
#### Example usage
```elixir
delta = %{
a: %{action: :added, value: 1},
b: %{action: :modified, old_value: 2, value: 3},
c: %{action: :removed}
}
context_id = "game:42"
formatted_delta = MyApp.CustomFormatter.format(delta, context_id)
IO.inspect(formatted_delta)
[
%{
timestamp: "2025-01-22T15:00:00Z",
context_id: "game:42",
field: :a,
action: :added,
new_value: 1
},
%{
timestamp: "2025-01-22T15:00:00Z",
context_id: "game:42",
field: :b,
action: :modified,
old_value: 2,
new_value: 3
},
%{
timestamp: "2025-01-22T15:00:00Z",
context_id: "game:42",
field: :c,
action: :removed
}
]
```
### Custom time provider
Chord allows you to define custom time provider by implementing the `Chord.Utils.Time.Behaviour`. This feature is useful for customizing time-based operations, such as timestamp generation and for mocking time in tests.
#### Defining a custom time provider
To define your custom time provider, create a module that implements the `Chord.Utils.Time.Behaviour`:
```elixir
defmodule MyApp.CustomTimeProvider do
@moduledoc """
A custom time provider for Chord, demonstrating how to implement the behavior.
"""
@behaviour Chord.Utils.Time.Behaviour
@impl true
def current_time(:second) do
# Example: Use a custom logic for time in seconds
DateTime.utc_now() |> DateTime.to_unix(:second)
end
@impl true
def current_time(:millisecond) do
# Example: Use a custom logic for time in milliseconds
DateTime.utc_now() |> DateTime.to_unix(:millisecond)
end
end
```
#### Configuring Chord to use your time provider
```elixir
# config/config.exs
config :chord, :time_provider, MyApp.CustomTimeProvider
```
---
## Benchmark results: Redis and ETS performance
Chord has been tested to ensure solid performance in both Redis (single-node setup for now, with plans for distributed scenarios) and ETS (in-memory, single-node applications). Here’s how it performs under various scenarios:
## Scenarios tested
### 1. Stateless operations
These scenarios simulate operations without maintaining a dedicated process per context. All updates, syncs and state modifications happen directly through the library’s API.
- **Single context (50 participants):** Represents a single group chat or meeting with 50 participants frequently updating their status, typing indicators, or syncing state.
- **Multiple contexts (100 contexts):** Simulates 100 independent group chats or meetings being updated simultaneously.
### 2. Stateful operations
These scenarios introduce a process per context (e.g., a GenServer for each group chat). Each participant interacts with this stateful process and the process uses Chord’s API to manage context.
- **Single context (50 participants):** A single group chat or meeting managed by a GenServer, handling frequent updates and syncs from 50 participants.
- **Multiple contexts (100 contexts):** Simulates 100 group chats or meetings, each managed by its own GenServer, handling participant interactions.
## Results
### Redis backend (single node)
> **Note:** Redis benchmarks were conducted in a single-node configuration to evaluate baseline performance. While Redis is designed for distributed systems, a fully distributed environment is not yet implemented in the benchmark script. Plans are underway to expand the benchmarking script to support distributed scenarios.
| **Scenario** | **Operations/sec** | **Average Time** | **Notes** |
|------------------------------------|--------------------|------------------|--------------------------------------------------|
| Stateless - Single Context (50) | 92.89 ops/s | 10.77 ms | Handles concurrent operations efficiently. |
| Stateful - Single Context (50) | 18.80 ops/s | 53.19 ms | Performance impacted by GenServer overhead. |
| Stateful - Multiple Contexts (100) | 1.72 ops/s | 581.46 ms | Slower due to process sync overhead. |
| Stateless - Multiple Contexts (100)| 1.57 ops/s | 635.29 ms | Poor throughput under high multi-context load. |
### ETS Backend (In-Memory, Single-Node)
| **Scenario** | **Operations/sec** | **Average Time** | **Notes** |
|------------------------------------|--------------------|------------------|--------------------------------------------------|
| Stateless - Single Context (50) | 230.61 ops/s | 4.34 ms | Extremely fast for single-node setups. |
| Stateful - Single Context (50) | 54.34 ops/s | 18.40 ms | GenServer overhead slows performance. |
| Stateful - Multiple Contexts (100) | 5.66 ops/s | 176.81 ms | Scales well but slower with 100 contexts. |
| Stateless - Multiple Contexts (100)| 4.69 ops/s | 213.18 ms | Limited scalability for multi-context updates. |
## Key insights
### Redis:
- **Single-node performance**: Reflects the baseline for Redis's capability, with potential for distributed scaling in the future.
- **Stateless operations** Outperform stateful ones when multiple clients update a single context concurrently. The absence of GenServer synchronization overhead makes Redis particularly well-suited for high-frequency, multi-client updates to shared contexts.
- **Stateful performance bottlenecks**: Syncing multiple contexts (100) causes significant performance degradation due to process synchronization and network overhead.
- **Future improvements**: A distributed Redis setup would allow benchmarking its true scalability and potential for handling high-throughput, multi-context scenarios.
### ETS:
- **Optimal for single-node applications**: Outshines Redis in single-node scenarios, particularly in stateless operations where 50 participants concurrently update a single context, achieving 230.61 ops/sec with just 4.34 ms latency.
- **GenServer overhead**: Stateful operations see reduced performance due to process-based synchronization, especially with many contexts (e.g., 100).
- **Scalability limitations:**: While ETS is efficient for localized, single-node setups, it struggles with multi-context workloads, where its performance drops to 4.69 ops/sec for stateless and 5.66 ops/sec for stateful scenarios.
## Choosing between stateless and stateful
### Stateless:
- Directly interacts with Chord’s API, bypassing the need for per-context processes.
- **Best for**: High-concurrency scenarios where multiple clients update a single shared context. Performance degrades under high multi-context workloads, particularly in Redis.
### Stateful:
- Manages a dedicated GenServer per context (e.g., per group chat or meeting).
- **Best for**: Scenarios requiring additional application-level state or business logic. However, high-concurrency and multi-context workloads may lead to significant performance degradation.
### Device Information
| Property | Value |
|----------------------------|--------------------------|
| **Operating System** | macOS |
| **CPU Information** | Apple M4 Pro |
| **Number of Cores** | 12 |
| **Available Memory** | 24 GB |
| **Elixir Version** | 1.17.3 |
| **Erlang Version** | 27.1.2 |
| **JIT Enabled** | True |
**Benchmark Suite Configuration:**
- **Warmup:** 2 seconds
- **Execution Time:** 5 seconds
- **Parallel:** 1
- **Inputs:** Data
---
## Contributing
Contributions from the community are welcome to make Chord even better! Whether it's fixing bugs, improving documentation, or adding new features, your help is greatly appreciated.
### How to contribute
1. Fork the repository.
2. Create a new branch for your changes.
3. Make your changes and test them thoroughly.
4. Submit a pull request with a clear description of your changes.
Feel free to open issues for discussion or if you need help. Together, we can build something amazing!
---
## Testing
Chord comes with a robust suite of tests to ensure reliability. Run tests with:
```bash
mix test
```
---
🎵 *"Let Chord orchestrate your state management with precision and elegance."*
