Euros
Euros is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.
Installation
def deps do
[
{:euros, "~> 0.3.0"}
]
endUsage
iex> url = "https://euros-test.blogspot.jp/"
iex> Euros.Core.crawl(url, fn(page) -> IO.puts(inspect(page)) end)
%HTTPoison.Response{body: "<!DOCTYPE html><html dir='ltr'><head><meta content='width=device-width, initial-scale=1' name='viewport'/><title>euros test blog</title>....set depth limit
iex> url = "https://euros-test.blogspot.jp/"
iex> option = %Euros.CrawlOption{depth_limit: 2}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option)set cookie
iex> url = "https://euros-test.blogspot.jp/"
iex> http_option = %Euros.HTTPOption{cookie: "foo=bar;"}
iex> option = %Euros.CrawlOption{http_option: http_option}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option)cusotm url pattern
iex> url = "https://euros-test.blogspot.jp/"
iex> option = %Euros.HTTPOption{cookie: "foo=bar;"}
iex> pattern = ~r/test1/
iex> option = %Euros.CrawlOption{http_option: http_option, pattern: pattern}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option, pattern)License
Copyright (c) 2017 kytiken
Released under the MIT license
https://github.com/kytiken/euros/blob/master/LICENSEDocumentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/euros.