Filezcache - cache s3 and other files on disk
This is a file caching system optimized for use with cowmachine
but then also be used with other services.
It has some unique properties:
* Lookups return file references (if possible) which can be used with sendfile
* Cached files can be streamed to requestors while they are being filled
* On startup the cache is repopulated with existing files
* Lookups return data formats directly useable for cowmachine serving
The system uses the file system to store files and a disk log. The files are stored
in `priv/data/` and the disk log in `priv/journal`.
The disk log is used to rebuild the cache after a start. All files are checked against
the checksum from the disk log, non-matching files are deleted from the cache.
$ erl -pa ebin
Erlang R15B03 (erts-188.8.131.52) [source] [64-bit] [smp:4:4] [async-threads:0] [kernel-poll:false]
Eshell V184.108.40.206 (abort with ^G)
4> filezcache:insert(mykey, <<"foobar">>).
6> filezcache:insert(mykey, <<>>).
* `max_bytes` Maximum size of the cache in bytes, defaults to 10GiB (10737418240 bytes)
* `journal_dir` Directory for the disk log, defaults to `priv/journal`
* `data_dir` Directory for the cached files, defaults to `priv/data`
There are some known issues that need to be resolved:
* On startup delete files that are unknown to the disk log
* Add timeouts to `filezcache_entry` states `wait_for_data` and `streaming`
* Extra intelligence in filezcache_entry to prevent evicting active entries during garbage collection