# Crawly

[![Coverage Status](](
[![Hex pm](]( [![ downloads](](

## Overview

Crawly is an application framework for crawling web sites and
extracting structured data which can be used for a wide range of
useful applications, like data mining, information processing or
historical archival.

## Requirements

1. Elixir "~> 1.10"
2. Works on Linux, Windows, OS X and BSD

## Quickstart

1. Add Crawly as a dependencies:
   # mix.exs
   defp deps do
         {:crawly, "~> 0.13.0"},
         {:floki, "~> 0.26.0"}
2. Fetch dependencies: `$ mix deps.get`
3. Create a spider

   # lib/crawly_example/esl_spider.ex
   defmodule EslSpider do
     use Crawly.Spider
     alias Crawly.Utils

     @impl Crawly.Spider
     def base_url(), do: ""

     @impl Crawly.Spider
     def init(), do: [start_urls: [""]]

     @impl Crawly.Spider
     def parse_item(response) do
       {:ok, document} = Floki.parse_document(response.body)
       hrefs = document |> Floki.find("a.btn-link") |> Floki.attribute("href")

       requests =
         Utils.build_absolute_urls(hrefs, base_url())
         |> Utils.requests_from_urls()

       title = document |> Floki.find("") |> Floki.text()

         :requests => requests,
         :items => [%{title: title, url: response.request_url}]

4. Configure Crawly
   - By default, Crawly does not require any configuration. But obviously you will need a configuration for fine tuning the crawls:
   # in config.exs
   config :crawly,
     closespider_timeout: 10,
     concurrent_requests_per_domain: 8,
     middlewares: [
       {Crawly.Middlewares.UserAgent, user_agents: ["Crawly Bot"]}
     pipelines: [
       {Crawly.Pipelines.Validate, fields: [:url, :title]},
       {Crawly.Pipelines.DuplicatesFilter, item_id: :title},
       {Crawly.Pipelines.WriteToFile, extension: "jl", folder: "/tmp"}
5. Start the Crawl:
   - `$ iex -S mix`
   - `iex(1)> Crawly.Engine.start_spider(EslSpider)`
6. Results can be seen with: `$ cat /tmp/EslSpider.jl`

## Need more help?
I have decided to create a public telegram channel, so it's now possible to be connected, and it's possible to ask questions
and get answers faster!

Please join me on:

## Browser rendering

Crawly can be configured in the way that all fetched pages will be browser rendered,
which can be very useful if you need to extract data from pages which has lots
of asynchronous elements (for example parts loaded by AJAX).

You can read more here:
- [Browser Rendering](

## Experimental UI

The CrawlyUI project is an add-on that aims to provide an interface for managing and rapidly developing spiders.

Checkout the code from [GitHub]( 
or try it online [CrawlyUIDemo](

See more at [Experimental UI](

## Documentation

- [API Reference](
- [Quickstart](
- [Tutorial](

## Roadmap

1. [x] Pluggable HTTP client
2. [x] Retries support
3. [x] Cookies support
4. [x] XPath support - can be actually done with meeseeks
5. [ ] Project generators (spiders)
6. [ ] UI for jobs management

## Articles

1. Blog post on Erlang Solutions website:
2. Blog post about using Crawly inside a machine learning project with Tensorflow (Tensorflex):
3. Web scraping with Crawly and Elixir. Browser rendering:
4. Web scraping with Elixir and Crawly. Extracting data behind authentication:
5. [What is web scraping, and why you might want to use it?](
6. [Using Elixir and Crawly for price monitoring](

## Example projects

1. Blog crawler:
2. E-commerce websites:
3. Car shops:
4. JavaScript based website (Splash example):

## Contributors

We would gladly accept your contributions! 

## Documentation
Please find documentation on the [HexDocs](

## Production usages

Using Crawly on production? Please let us know about your case!