PlugRobots

An Elixir plug serving robots.txt.

Installation

Add plug_robots to your list of dependencies in mix.exs:

def deps do
  [{:plug_robots, "~> 0.1.0"}]
end

Usage

• Add it to your endpoint before the router

endpoint.ex

plug PlugRobots, rules: :allow_all

plug MyApp.Router

Examples

• Deny all crawler

plug PlugRobots, rules: :deny_all

plug MyApp.Router

• Returns different rules by environment

case Mix.env do
    :prod -> plug PlugRobots, rules: :allow_all
    _     -> plug PlugRobots, rules: :deny_all
end

plug MyApp.Router

• Returns custom rules

@googlebot_rule """
User-agent: GoogleBot
Disallow: /images/dogs.jpg
"""

plug PlugRobots, rules: @googlebot_rule

plug MyApp.Router

Options

• rules
  • :allow_all: Allow all crawlers to all path
  • :deny_all: Disallow all crawlers to all path
  • string: any other custom rules