kreuzcrawl

Elixir bindings for kreuzcrawl — a high-performance Rust web crawling engine. Uses Rustler NIFs for native BEAM integration with OTP-compatible error tuples and ResourceArc handles.

Installation

def deps do
  [{:kreuzcrawl, "~> 0.1.0-rc.2"}]
end

Quick Start

# Create engine with default settings
{:ok, engine} = Kreuzcrawl.create_engine(nil)

# Scrape a single page
{:ok, result} = Kreuzcrawl.scrape(engine, "https://example.com")
IO.puts("Title: #{result.metadata.title}")
IO.puts("Status: #{result.status_code}")
IO.puts("Links: #{length(result.links)}")

API Reference

Full API documentation is available at docs.kreuzcrawl.kreuzberg.dev.

Key functions:

create_engine(config?) — Create a crawl engine with optional configuration
scrape(engine, url) — Scrape a single URL
crawl(engine, url) — Crawl a website following links
map_urls(engine, url) — Discover all pages on a site
batch_scrape(engine, urls) — Scrape multiple URLs concurrently
batch_crawl(engine, urls) — Crawl multiple seed URLs concurrently

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

License

This project is licensed under Elastic License 2.0.