# ExPmtiles
An Elixir library for working with PMTiles files - a single-file format for storing tiled map data.
## Overview
PMTiles is an efficient single-file format for storing tiled map data, designed for cloud storage and CDN delivery. This library provides a complete Elixir implementation for reading and accessing tiles from PMTiles files stored either locally or on Amazon S3.
## Features
- **Multi-storage support**: Read PMTiles files from local storage or Amazon S3
- **Efficient caching**: Caching system with directory caching
- **Concurrent access**: Safe concurrent access with request deduplication
- **Compression support**: Built-in support for gzip compression
- **Tile type support**: MVT, PNG, JPEG, WebP, and AVIF tile formats
- **Performance optimized**: Background directory pre-loading and persistent cache storage
- **Production ready**: Configurable timeouts, connection pooling, and error handling
## Installation
Add `ex_pmtiles` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:ex_pmtiles, "~> 0.2.1"}
]
end
```
## Quick Start
### Basic Usage
```elixir
# Open a local PMTiles file
instance = ExPmtiles.new("path/to/file.pmtiles", :local)
# Open a PMTiles file from S3
instance = ExPmtiles.new("my-bucket", "path/to/file.pmtiles", :s3)
# Get a tile by coordinates
case ExPmtiles.get_zxy(instance, 10, 512, 256) do
{{offset, length, data}, updated_instance} ->
# Use the tile data
data
{nil, updated_instance} ->
# Tile not found
nil
end
```
### Using the Cache
For production applications, use the caching layer for better performance:
```elixir
# Start the cache for an S3 PMTiles file
{:ok, cache_pid} = ExPmtiles.Cache.start_link(
name: :cache_one,
bucket: "maps",
path: "world.pmtiles",
max_entries: 100_000, # Maximum tiles to cache (default: 100,000)
enable_dets: true # Enable persistent cache (default: true)
)
# Get tiles with automatic caching
case ExPmtiles.Cache.get_tile(:cache_one, 10, 512, 256) do
{:ok, tile_data} ->
# Handle tile data
tile_data
{:error, reason} ->
# Handle error
nil
end
# Get cache statistics
stats = ExPmtiles.Cache.get_stats(:cache_one)
# Returns: %{hits: 150, misses: 25}
```
## API Reference
### Core Functions
#### `ExPmtiles.new/2` and `ExPmtiles.new/4`
Create a new PMTiles instance:
```elixir
# Local file
instance = ExPmtiles.new("data/world.pmtiles", :local)
# S3 file
instance = ExPmtiles.new("my-bucket", "maps/world.pmtiles", :s3)
```
#### `ExPmtiles.get_zxy/4`
Get a tile by zoom level and coordinates:
```elixir
case ExPmtiles.get_zxy(instance, 10, 512, 256) do
{{offset, length, data}, updated_instance} ->
# Tile found
data
{nil, updated_instance} ->
# Tile not found
nil
end
```
#### `ExPmtiles.zxy_to_tile_id/3` and `ExPmtiles.tile_id_to_zxy/1`
Convert between coordinates and tile IDs:
```elixir
# Convert coordinates to tile ID
tile_id = ExPmtiles.zxy_to_tile_id(10, 512, 256)
# Convert tile ID back to coordinates
{z, x, y} = ExPmtiles.tile_id_to_zxy(tile_id)
```
### Cache Functions
#### `ExPmtiles.Cache.start_link/1`
Start a cache process:
```elixir
{:ok, pid} = ExPmtiles.Cache.start_link(
bucket: "maps",
path: "world.pmtiles",
max_entries: 100_000, # Optional: max tiles to cache (default: 100,000)
enable_dets: true # Optional: enable persistent cache (default: true)
)
```
**Options:**
- `:bucket` - S3 bucket name (or `nil` for local files)
- `:path` - Path to the PMTiles file
- `:name` - Optional custom name for the cache process (atom)
- `:max_entries` - Maximum number of tiles to cache (default: 100,000)
- `:enable_dets` - Enable DETS persistence for directory cache (default: true)
#### `ExPmtiles.Cache.get_tile/4`
Get a tile with caching:
```elixir
case ExPmtiles.Cache.get_tile(pid, 10, 512, 256) do
{:ok, data} -> data
{:error, reason} -> nil
end
```
## Configuration
### S3 Configuration
For S3 access, ensure you have ExAws configured in your application:
```elixir
# In config/config.exs
config :ex_aws,
access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"},
region: "us-east-1"
```
### Cache Configuration
Configure cache behavior:
```elixir
# In config/config.exs
config :ex_pmtiles,
max_entries: 100_000, # Maximum tiles to cache
cache_dir: "priv/pmtiles_cache", # Directory for persistent cache (when enabled)
http_pool_size: 100, # Maximum HTTP connections for S3 (default: 100)
http_timeout: 15_000 # HTTP timeout in milliseconds (default: 15_000)
```
### DETS Persistence
By default, the cache persists directory structures to disk using DETS (Disk-based Erlang Term Storage). This allows the cache to warm up instantly on restart, avoiding expensive S3 fetches.
**Enable/Disable DETS** (per-cache instance):
```elixir
# With DETS persistence (default) - faster restarts
{:ok, pid} = ExPmtiles.Cache.start_link(
bucket: "maps",
path: "world.pmtiles",
enable_dets: true # Default
)
# Without DETS persistence - useful for ephemeral environments
{:ok, pid} = ExPmtiles.Cache.start_link(
bucket: "maps",
path: "world.pmtiles",
enable_dets: false
)
```
**Configure DETS storage location**:
```elixir
# In config/config.exs
config :ex_pmtiles,
cache_dir: "/var/cache/pmtiles" # Default: System.tmp_dir!() <> "/ex_pmtiles_cache"
```
**When to disable DETS**:
- Ephemeral containers (Docker/Kubernetes) with no persistent volumes
- Serverless environments (AWS Lambda, etc.)
- Testing environments where you want a fresh cache each time
- Memory-constrained environments where disk I/O should be minimized
## Performance Features
### Multi-Level Caching
The library implements a sophisticated caching system:
1. **Directory Cache (ETS)**: Hot cache - deserialized PMTiles directory structures in memory
2. **Tile Cache (ETS)**: Individual tile data with LRU eviction
3. **Persistent Cache (DETS)**: Warm cache - saves directories to disk for faster restarts (optional, enabled by default)
### Background Pre-loading
The cache automatically pre-loads directories for zoom levels 0-4 in the background, improving response times for common zoom levels.
### Concurrent Request Handling
Multiple processes can safely access the same cache without duplicate requests. The system coordinates requests and broadcasts results to all waiting processes.
## Supported Formats
### Compression Types
- `:none` - No compression
- `:gzip` - Gzip compression
- `:unknown` - Unknown compression type
### Tile Types
- `:mvt` - Mapbox Vector Tiles
- `:png` - PNG images
- `:jpg` - JPEG images
- `:webp` - WebP images
- `:avif` - AVIF images
## Testing
The library includes test support with mock implementations:
```elixir
# In test/test_helper.exs
Mox.defmock(ExPmtiles.CacheMock, for: ExPmtiles.Behaviour)
# Mock implementations are automatically configured for test environment
```
You can also test against a pmtiles file stored in an s3 bucket using `export $(cat .env | xargs) && elixir test_s3.exs`
This expects you to define environment variables in `.env`:
```
AWS_ACCESS_KEY_ID=<>
AWS_REGION=<>
AWS_SECRET_ACCESS_KEY=<>
BUCKET=<>
OBJECT=<>
```
### Performance Testing
The library includes performance tests to ensure O(n) linear complexity for directory deserialization:
```bash
# Run performance tests
mix test test/ex_pmtiles/performance_test.exs
# Run detailed benchmarks
mix run bench/deserialize_directory_bench.exs
```
The performance tests verify that:
- Directory deserialization scales linearly with the number of entries
- Large directories (10,000+ entries) deserialize in under 500ms
- Time per entry remains roughly constant regardless of directory size
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Documentation
Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc) and published on [HexDocs](https://hexdocs.pm). Once published, the docs can be found at <https://hexdocs.pm/ex_pmtiles>.
