RobotsTxt

An RFC 9309 robots.txt parser and matcher for Elixir, with Google-compatible matching behavior and no runtime dependencies.

Installation

Add robots_txt to the dependencies in mix.exs:

def deps do
  [
    {:robots_txt, "~> 0.1"}
  ]
end

Or depend on the Git repository:

{:robots_txt, git: "https://github.com/ivan-podgurskiy/robots_txt.git"}

Quick start

robots =
  RobotsTxt.parse("""
  User-agent: *
  Disallow: /private
  Allow: /private/preview
  Sitemap: https://example.com/sitemap.xml
  """)

RobotsTxt.allowed?(robots, "ExampleBot", "/private/report")
#=> false

RobotsTxt.matched_rule(robots, "ExampleBot", "/private/preview")
#=> {:allow, "/private/preview", 3}

RobotsTxt.sitemaps(robots)
#=> ["https://example.com/sitemap.xml"]

Why

• Total, byte-oriented parsing accepts arbitrary binary input, including malformed UTF-8.
• Longest-match precedence, allow-on-tie behavior, wildcard matching, and specific-versus-global group selection follow Google-compatible behavior.
• Match decisions include the original pattern and source line number for diagnostics.
• The package has no runtime dependencies.

What this library does not do

RobotsTxt does not fetch robots.txt files, follow redirects, cache results, enforce crawl delays, or schedule crawler work. It also does not escape or normalize request targets. Network policy and crawler coordination remain with the caller.

The caller is responsible for fetching /robots.txt from the correct origin and for applying the parsed rules only to that origin. Although absolute URLs are accepted for matching convenience, their scheme, host, and port are not used to determine scope.

For example, an application using Req can classify the final response before parsing it. Req remains an application dependency, not a dependency of this package:

case Req.get("https://example.com/robots.txt") do
  {:ok, %{status: status, body: body}} ->
    case RobotsTxt.fetch_semantics(status) do
      :parse_body -> RobotsTxt.parse(body)
      :allow_all -> :allow_all
      :disallow_all -> :disallow_all
    end

  {:error, reason} ->
    {:fetch_error, reason}
end

Redirects must be resolved before passing the final status to RobotsTxt.fetch_semantics/1.

API

• RobotsTxt.parse/1 and RobotsTxt.parse/2 parse a binary document. Version 0.1 accepts only an empty options list.
• RobotsTxt.allowed?/3 returns the access decision for a crawler token and an escaped target.
• RobotsTxt.matched_rule/3 returns {:allow | :disallow, pattern, line} or :default.
• RobotsTxt.sitemaps/1 returns sitemap values in file order.
• RobotsTxt.crawl_delay/2 returns the first parseable non-negative delay from the selected groups, but does not enforce it.
• RobotsTxt.extensions/2 returns unknown directives for selected groups or global scope.
• RobotsTxt.fetch_semantics/1 classifies a final HTTP response.
• RobotsTxt.valid_user_agent?/1 validates a crawler product token.

See the HexDocs API reference for complete contracts and examples.

Escaped-path contract

Targets passed to allowed?/3 and matched_rule/3 must already be RFC 3986-escaped. The matcher does not decode percent escapes or normalize paths, so /a/b and /a%2Fb remain distinct.

Matching uses the path, parameters, and query. It ignores the fragment and, for absolute URLs, the scheme, host, and port. A missing path is treated as /.

The crawler argument is a product token, not a complete HTTP User-Agent header. It is compared case-insensitively with the leading valid product token extracted from each file-side User-agent value.

HTTP semantics cheat sheet

fetch_semantics/1 accepts only a final response status:

Resolve 1xx and 3xx responses before classification. Passing them to fetch_semantics/1 raises FunctionClauseError.

Extensions

Sitemap directives are collected globally in file order. Crawl-delay is available as optional metadata and is never enforced by the library.

Unknown directive names are lowercased and their values retain file order. extensions(robots, user_agent) merges all selected specific groups, falling back to selected global groups only when no specific group matches. extensions(robots, :global) returns directives found outside groups.

Compliance

The implementation passes all 378 standard cases and all 22 Google-specific cases in the pinned Google robots.txt compliance harness. The exact harness revision, prerequisites, invocation, and recorded result are in test/compliance/README.md.

Google-derived compatibility tests are identified in NOTICE.

Roadmap

Planned follow-up work is tracked in the repository-only ROADMAP.md.

Development

mix deps.get
mix format --check-formatted
mix compile --warnings-as-errors
mix test --seed 0
mix credo --strict
mix dialyzer --format github
mix docs

License

MIT © Ivan Podgurskiy. See LICENSE.