# BroadwayRedis

An _opinionated_ Redis connector for [Broadway]( to process work from a Redis list structure.

Documentation can be found at [](

This project provides:

  * `BroadwayRedis.Producer` - A GenStage producer that continuously pops items from a Redis list and acknowledges them after being successfully processed.
  * `BroadwayRedis.RedisClient` - A generic behaviour to implement Redis clients.
  * `BroadwayRedis.RedixClient` - Default Redis client used by `BroadwayRedis.Producer`.

## What is opinionated about this library?

Because Redis lists do not support the concept of acknowledgements, this project utilizes the [RPOPLPUSH]( command available in Redis to atomically pop an item from the list while moving the item to a 'working' or 'processing' list for a later pseudo-acknowledgement using the `LREM` command.

This idea follows the blueprint of the _Reliable Queue_ pattern outlined in the Redis documentation found [here](

Because `RPOPLPUSH` is used, the other assumption is that the head of your list will be on the right side, so you will likely want to push work into your list using `LPUSH` (for FIFO processing). If you want to prioritize an item to be processed next, you could push that to the right (head) by using a `RPUSH`.

## Redis Client

The default Redis client uses the [Redix]( library. See that project for other features.

I have not attempted to use any other Redis libraries in the community at this point. I expect there may need to be changes made to this producer to accomodate others.

## Caveats

* You are responsible for maintaining your own named connection to Redis outside the scope of this library. See the [Real-World Usage]( docs for Redix for setting up a named instance/connection.
* At this point, no testing has been done with a pooling strategy around Redis. I am using a single connection dedicated for my broadway pipeline in a small system. Ideally I would like to improve this to the point where just the Redis host, port, and credentials are provided to this provider for handling it's own connections/pooling.
* You are responsible for monitoring your working/processing list in Redis. If something goes wrong and an acknowledgement (`LREM`) is not handled - you will want some logic or process in place to move an item from the working list back to the main list.
* The Redis `LREM` command is _O(N)_ - so the performance on this operation during acknowledgement will be based on the length of the list. I have been using this pattern for a number of years without problem, but be aware and do research on your own use-case to ensure this is not going to be a problem for you.


## Installation

Add `:broadway_redis` to the list of dependencies in `mix.exs`:

def deps do
    {:broadway_redis, "~> 0.1.0"}

## Usage

Configure Broadway with one or more producers using `BroadwayRedis.Producer`:

  name: MyBroadway,
  producers: [
    default: [
      module: {
        redis_instance: :some_redis_instance,
        list_name: "some_list",
        working_list_name: "some_list_processing"


## Other Info

This library was created using the [Broadway Custom Producers documentation]( for reference. I would encourage you to view that as well as the [Broadway Architecture documentation]( for more information.


## License

MIT License

See the [license file](LICENSE.txt) for details.