JC
====
##Erlang, Distributable, In-Memory Cache
### Pub/Sub; JSON-Query; Consistency Assist; and Simple, TCP Interoperability Protocol.
[![Build Status](https://travis-ci.org/jr0senblum/jc.svg)](https://travis-ci.org/jr0senblum/jc)
[![hex.pm version](https://img.shields.io/hexpm/v/jc.svg)](https://hex.pm/packages/jc)
###Features
* Cache entries are Map, Key, Value, [TTL], [Sequence]
* Maps represent a name-space for Keys - similar to the notion
of 'bucket'
in other caching systems
* Maps, Keys and Values can be any Erlang term
* TTL is time-to-live in seconds
* Consistency assist through sequence numbers: An alternative API
allows for a sequence-number parameter on the put/x, evict/x,
match/x and remove/x operations. Operations whose sequence
number is lower than the current, per-map max are disallowed
thereby ensuring, for example, that stale puts do not
overwrite newer ones due to the newer one beating the stale
ones to the cache.
* JSON query support
* Query by JSON: When Values are JSON, evict_match/2,
evict_all_match/1 and values_match/2 will search or evict
keys whose JSON value, at a location specificed by a java-style,
dot-path string, equals the given value. That is,
jc:values_match(bed, "id.type=3") would return all values, in the given
map (bed), where that value was a JSON object, id, with a "type":3
at its top-level.
* Ad-hoc, index support: In order to support faster
operations, (2-3 orders of magnitude), each map can have up to four,
dot-path, strings configured (or added at run-time) for which jc will
provide index support.
* Auto index recognition - Frequently used JSON querries will be
automatically detected and indexing initiated.
* User controlled eviction
* Map-level TTL: A job runs at configured intervals and removes
items whose create-date is older than a map-specific, configured
number of seconds.
* Item-level TTL: PUTS can include a TTL which defines when the
item should be evicted. Used for shorter TTLs, as an exception
to a Map-level TTL, or when more precision is required than that
offered by the Map-level TTL.
* Pub/Sub
* Clients can subscribe to Map events for a specific key or
for any key, and for write, delete or either operations
* Clients can create and subscribe to arbitrary 'topics' and
broadcast arbitrary messages under those topic names
* Clients can subscribe to node-up and node-down events
* Interopability
* Binary string over TCP returning JSON
* Bridge process that accepts messages from a client indicating
cache operations, executes the cache operations and returns the
results to the client. This has been used with JInterface to
interoperate with CLOJURE and Java clients
* Fine-grained logging via Lager
###Cache Functions (jc)
* Create
* put(Map, Key, Value, [TTLSecs]) -> {ok, Key} | {error, badarg}
* put_all(Map, [{K,V},{K,V},...], [TTLSecs]) -> {ok, CountOfSuccessfulPuts} |
{error, badarg}
* Delete
* evict(Map, Key) -> ok
* evict_all_match(Criteria = "Json.Path.Match=Value") -> ok
* evict_map_since(Map, Age_In_Secs) -> ok
* evict_match(Map, Criteria = "Json.Path.Match=Value") -> ok
* remove_items(Map, Keys) -> {ok, [{K, V}, ...]} for each {K, V} deleted.
* Retrieve
* get(Map, Key) -> {ok, Value} | miss
* get_all(Map, [K1, K2, ...]) -> {ok, {Found=[{K1,V1},...], Misses=[K2,...]}}
* key_set(Map) -> {ok, [K1, K2, ...]} for each Key in the Map
* values(Map) -> {ok, [V1, V2, ...]} for each Value in the Map
* values_match(Map, Criteria="JSon.Path.Match=Value") ->
{ok, [{K1,V1}, {K2,V2}, ...]}
* Flush
* clear(Map) -> ok
* flush() -> ok
* flush(silent) -> ok, Does not send alerts to subscribers
* Predicates
* contains_key(Map, Key) -> true | false.
* Meta
* cache_nodes() -> {{active, [Node1,... ]}, {configured, [Node1,... ]}}
* cache_size() -> {size, [{TableName, RecordCnt, Words}],...}
* map_size(Map) -> {records, Count}
* maps() -> {maps, [Map1, Map2,...]}
* up() -> {uptime, [{up_at, String},{now, String},
{up_time, {D, {H, M, S}}}]
### Consistency Support Functions (jc_s)
Identical to the Create and Evict family of functions of the jc module
(see above), except:
* An additional sequence parameter, which is expected to be a monotonically
incresing integer (with respect to a given Map), used to disalow
"out of sequence" operations
* Functions return {error, out_of_seq} if out of sequence operation is attempted
* evict(Map, Key, Seq)
* evict_all_match(Criteria = "Json.Path.Match=Value", Seq)
* evict_match(Map, Criteria = "Json.Path.Match=Value", Seq)
* put(Map, Key, Value, [TTLSecs], Seq)
* put_all(Map, [{K,V},{K,V},...], [TTLSecs], Seq)
* remove_items(Map, Keys, Seq)
* Meta functions
* sequence() -> [{Map, HighestNnumber},... ]
* sequence(Map) -> HightestNumber
###Eviction Manager Functions (jc_eviction_manager)
* set_max_ttl(Map, Secs) -> ok | {error, badarg}
* get_max_ttls() -> [{Map, Secs}, ...]
###Pub/Sub Functions (jc_psub)
* map_subscribe(Pid, Map, Key|any, write|delete|any) -> ok | {error, badarg}
* map_unsubscribe(Pid, Map, Key|any, write|delete|any) -> ok | {error, badarg}
* client receives
`{map_event, {Map, Key, delete}}`
or
`{map_event, {Map, key, write, Value}`
* topic_subscribe(Pid, Topic, Value) -> ok | {error, badarg}
* topic_unsubscribe(Pid, Topic, Value) -> ok | {error, badarg}
* client receives: {topic_event, {Topic, Value}}
* topic_event(Topic, Value) -> ok
* Broadcasts Value to all
subscribers of Topic
* topic_subscribe(Pid, jc_node_events, any) -> ok
* subscribtes the user
to node up and node down events:
`{jc_node_events, {nodedown, DownedNode, [ActiveNodes],[ConfiguredNodes]}}`
`{jc_node_events, {nodeup, UppedNode, [ActiveNodes],[ConfiguredNodes]}}`
###Indexing Functions (jc_store)
* start_indexing(Map, Path={bed,"menu.id"}) -> ok |
{error, no_indexes_available} |
{error, Term}
* stop_indexing(Map, Path={bed,"menu.id"}) -> ok
* indexes(Map) -> {indexes, [{{Map, Path}, Position},...]} for all indexes
of given Map
* indexes() -> {indexes, [{{Map, Path}, Position},...]} for all indexes
###Interoperability: Bridge (jc_bridge)
* All functions from the jc, jc_s, jc_eviction_manager, jc_psub
and jc_store are supported and are of the form:
`{From, {Fn, P1, P2,...}}`
for each paramater, as in
`jc_bridge ! {Pid, {put, Map, Key, Value}}`
* Additionally,
{From, {node_topic_sub}} -> ok | {error, badarg},
client will recieve:
`{jc_node_events, {nodedown, DownedNode, [ActiveNodes],[ConfiguredNodes]}}`
or
`{jc_node_events, {nodeup, UppedNode, [ActiveNodes],[ConfiguredNodes]}}`
{From, {node_topic_unsub}} -> ok.
### Interoperability: Socket Protocol
Binary-encoded, string protocol used to provide socket-based
interoperability with JC.
All messages to JC are string representations of a tuple. All
messages form the caching system to the client are JSON
The protocol defines three message types: CONNECT, CLOSE and COMMAND all
of which are binary strings consisting of an 8-byte size followed by the
actual command messages.
Responses are also binary strings with an 8-byte size prefix.
The CONNECT command initiates a session,
M = <<"{connect,{version,\"1.0\"}}">>
Size is 25, so the CONNECT message is:
<<25:8, M/binary>> =
<<25,40,58,99,111,110,110,101,99,116,32,123,58,118,101,
114,115,105,111,110,32,49,46,48,125,41>>
The server will respond to a CONNECT command with either an error or
the encoded version of {"version":" "1.0"}
<<15:8, <<"{\"version\":\"1.0\"}">> =
<<15,123,34,118,101,114,115,105,111,110,34,58,49,46,48,125>>
The CLOSE command closes the socket, ending the session
M = <<"{close}">>
Size is 7 so the CLOSE message is:
<<7,123,99,108,111,115,101,125>>
COMMAND messages are string versions of the tuple-messages, which
jc_bridge uses, without the self() parameter. For example
{self(), {put, Map, Key, Value}} becomes
{put, Map, Key, Value}
The return will be an encoded version of a JSON string. A client session
might look as follows:
client:send("{put, evs, \"1\", \"{\\\"value:\\\":true}\"}")
<<"{\"ok\":\"1\"}">>
client:send("{get, evs, 1}"),
<<"{\"ok\":"{\\\"value\\\":true}\"}">>
###Configuration
* Application configuration is in sys.config which is heavily
commented
* Cookie, node-name and auto-restart of VM controlled by vm.args
###Application Modules
* jc_cluster
* Simple, mnesia-based, cluster creation and management
* jc, jc_s, jc_store, jc_eviction_manager
* Caching operations, Key-specific and Map-level TTLs
* jc_sequence
* Singleton service enforcing strictly monotonic sequence
numbers on jc_s operations
* jc_analyzer
* Analysis and indexing inititation of JSON query strings
* jc_protocol
* Socket processing of messages and Erlang -> JSON
* jc_psub:
* Pub / Sub of cache write and delete events
* On-demand, ad-hoc topic events
* Predefined, *jc_node_events* topic provides subscribers
node-up and node-down notifications
* jc_bridge
* Server that acts as a proxy between an external process and
jc functionality
###Build Instructions
* Ensure that Erlang 17 or higher is installed
* Get the Source Code from Stash
`[root@db01] git clone https://github.com/jr0senblum/jc.git`
* Edit the sys.config and vm.args files in ./config
* vm.args: Indicate the correct node names and cookie
* sys.config: Adjust prarameters as neccesary.
`[root@db01] ./rebar3 release`
or
`[root@db01] ./rebar3 prod release`
###Documentation
`[root@dbo1] ./rebar3 edoc`
## Component and Sequence Diagrams
### Components
![](https://cloud.githubusercontent.com/assets/2043491/10463722/1aff2856-71b4-11e5-8e0a-5fcbee0c3ea3.png)
![](https://cloud.githubusercontent.com/assets/2043491/10463734/2446743c-71b4-11e5-8c6a-6de2da844fbf.png)
![](https://cloud.githubusercontent.com/assets/2043491/10463736/290403f4-71b4-11e5-8b94-bd7273d4c7fa.png)
### Sequence Diagrams
![](https://cloud.githubusercontent.com/assets/2043491/10463819/9c59b7f4-71b4-11e5-9db9-a82fa762240c.png)