README.md
# Crawler [![Travis](https://img.shields.io/travis/fredwu/crawler.svg)](https://travis-ci.org/fredwu/crawler) [![Coverage](https://coveralls.io/repos/github/fredwu/crawler/badge.svg?branch=master)](https://coveralls.io/github/fredwu/crawler?branch=master) [![Hex.pm](https://img.shields.io/hexpm/v/crawler.svg)](https://hex.pm/packages/crawler)
A high performance web crawler in Elixir.
## Usage
```elixir
Crawler.crawl("http://elixir-lang.org", max_levels: 2)
```
## Configurations
| Option | Type | Default Value | Description |
|-------------|---------|---------------|-------------|
| :max_levels | integer | 3 | Maximum nested level of pages to crawl.
## Features Backlog
Crawler is under active development, below is a non-comprehensive list of features to be implemented.
- [x] Set the maximum crawl level.
- [ ] Save to disk.
- [ ] Set timeouts.
- [ ] The ability to manually stop/pause/restart the crawler.
- [ ] Restrict crawlable domains, paths or file types.
- [ ] Limit concurrent crawlers.
- [ ] Limit rate of crawling.
- [ ] Set crawler's user agent.
- [ ] The ability to retry a failed crawl.
- [ ] DSL for scraping page content.
## Changelog
Please see [CHANGELOG.md](CHANGELOG.md).
## License
Licensed under [MIT](http://fredwu.mit-license.org/).