ExAbby

ExAbby is a minimal A/B testing library for Elixir/Phoenix.

Caveat: This was created primarily over a weekend with the help of Chat GPT/Claude. The code is working but still needs a lot of cleanup and optimizations, which I'll do as I run into problems. As it stands, it is working under low load in production settings providing a super easy way to AB test Phoenix and Liveview using assigns.

Why Ex Abby?

I have found there are no super simple ways to get ab testing working for smaller sites in Elixir. You have to pay $$ or use a complex system. And everything has moved to feature tagging. This experiment framework is based on something we built in-house for a previous company that reached virality co-efficients of 1.0 a few times. And the goal is to make it super easy to use in Liveview environments.

This is really early and the API is 100% likely to change. Feedback is appreciated!

It supports:

• Ecto-based storage (Experiments, Variations, Trials)
• Session-based or User-based assignment
• Linking session trials to users - Track user performance from session-based experiments
• Archive experiments with optional winner declaration
• Flexible ID support: structs, integers, or strings
• Weighted randomization
• Recording success events
• LiveView helpers (checking connected?/1 and storing assigned variation)
• Admin LiveViews with experiment filtering (Active/Archived/All)
• Upserting experiments/variations with optional weight updates
• Reviewing results over different time periods
• Control-relative p-values for experiments with multiple treatments
• Ability to toggle variations by user or session for testing

Coming in the future

• armed bandits
• optimizations / caching
• better UX of admin screens
• So much cleanup
• Likely changes to the API.

Table of Contents

1. Installation
2. Configuration
3. Migrations
4. Upgrading from v0.1 to v0.2
5. Upserting Experiments and Updating Weights
6. Session Setup
7. Admin Routes
8. Usage in Controllers
9. Usage in LiveView
10. Linking Sessions to Users
11. Archiving Experiments
12. Troubleshooting

Installation

1. Addex_abby as a dependency in your Phoenix (host) app’s mix.exs.

   If it's a public Hex package (or if you plan to publish it):

   defp deps do
     [
       {:ex_abby, "~> 0.4.0"}
     ]
   end
   

   If it's a GitHub repo (private or public):

   defp deps do
     [
       {:ex_abby, github: "grahac/ex_abby", tag: "v0.4.0"}
     ]
   end
   

2. Run:

   mix deps.get
   

Configuration

In your host app’s config/config.exs (or dev.exs, etc.), set:

config :ex_abby,
  repo: MyApp.Repo,
  control_variation_name: "control",
  bot_detection: [
    enabled: true,
    detectors: [ExAbby.BotDetector.UserAgent],
    fallback_variation: "control"
  ]

Where MyApp.Repo is your Ecto Repo module.

control_variation_name identifies the variation used as the baseline for significance comparisons in the admin results view. It defaults to "control" when omitted.

Bot detection is enabled by default. The built-in detector classifies common search, AI, social-preview, monitoring, SEO, and generic crawler user agents. It is intentionally user-agent based: applications can add request-aware detectors for Cloudflare or other signals, but custom detectors must return a stable atom such as {:bot, :cloudflare_verified_bot} and must never return a user agent, IP address, or other request-derived value.

Migrations

ExAbby provides Ecto migrations that create three tables:

• ex_abby_experiments
• ex_abby_variations
• ex_abby_trials

In your host app, generate a new migration:

mix ecto.gen.migration create_ex_abby_tables

Open priv/repo/migrations/2025xxxxxx_create_ex_abby_tables.exs, and add:

defmodule MyApp.Repo.Migrations.CreateExAbbyTables do
  use Ecto.Migration

  def up do
    ExAbby.Migrations.create_tables()
  end

  def down do
    ExAbby.Migrations.drop_tables()
  end
end

Then run:

mix ecto.migrate

Upgrading from v0.1 to v0.2

Version 0.2.0 adds experiment archiving with optional winner declaration. To upgrade:

1. Update your dependency in mix.exs:

{:ex_abby, "~> 0.2.0"}

1. Create a new migration:

mix ecto.gen.migration ex_abby_v2

1. Add to the generated migration file:

defmodule MyApp.Repo.Migrations.ExAbbyV2 do
  use Ecto.Migration

  def up, do: ExAbby.Migrations.v1_to_v2()
  def down, do: ExAbby.Migrations.v2_to_v1()
end

1. Run the migration:

mix ecto.migrate

This adds two new columns to ex_abby_experiments:

• archived_at - Timestamp when experiment was archived
• winner_variation_id - Reference to the winning variation (optional)

Upgrading to trial exclusion fields

Bot-exclusion audit fields are added with the next ExAbby migration:

defmodule MyApp.Repo.Migrations.ExAbbyBotExclusion do
  use Ecto.Migration

  def up, do: ExAbby.Migrations.v2_to_v3()
  def down, do: ExAbby.Migrations.v3_to_v2()
end

The migration adds nullable excluded_at and exclusion_reason fields to ex_abby_trials; it preserves rows for auditing and restoration.

Upserting Experiments and Updating Weights

If you have a function like:

ExAbby.upsert_experiment_and_update_weights(
  "landing_page_test",
  "Testing different landing pages",
  [
    {"Original", 1.0},
    {"Variation A", 1.0},
    {"Variation B", 2.0}
  ],
  success1_label: "Signup",
  success2_label: "Purchase"
)

Then:

• If "landing_page_test"does not exist, the library creates a new experiment with that name + description, and 3 variations with the specified weights.
• If the experiment already exists, we do not change its weights. We update all the other info if
• you can optionally add labels to label success. This is just for readability and is optional.

Seeding Experiments

Create a file priv/repo/seeds/experiments.exs to define your experiments:

experiments = [
  {
    "button_color_test",
    "Testing different button colors for signup",
    [
      {"control", 0.33},
      {"green", 0.33}, 
      {"blue", 0.33}
    ],
    [success1_label: "Signup", success2_label: "Purchase", update_weights: false]
  }
]

# Seed or update experiments without modifying weights
Enum.each(experiments, fn {name, description, variations, opts} ->
  ExAbby.upsert_experiment_and_update_weights(name, description, variations, opts)
end)

Then in your priv/repo/seeds.exs, add:

Code.require_file("seeds/experiments.exs", __DIR__)

You can run the seeds in different ways:

Development:

mix run priv/repo/seeds.exs

Session Setup

To enable session-based A/B testing, add ExAbby.SessionPlug to your endpoint or router pipeline:

# In  your router pipeline (recommended):

pipeline :browser do
  # ... other plugs ...
  plug ExAbby.SessionPlug
end

# In lib/your_app_web/endpoint.ex
plug Plug.Session,
  store: :cookie,
  key: "_your_app_key",
  signing_salt: "your_signing_salt"

plug ExAbby.SessionPlug

This plug creates a unique "ex_abby_session_id" for tracking A/B test variations across requests. It also evaluates bot detection on every request and stores only compact :human or {:bot, bot_name} status in conn.assigns[:ex_abby_bot] and the session's "ex_abby_bot" key. It never persists a raw user agent or IP address.

Bot behavior and custom detectors

For bot requests, the controller and LiveView assignment helpers return the configured fallback variation without creating or changing a trial. Each such assignment emits [:ex_abby, :assignment, :excluded] telemetry with the experiment, :bot reason, and static bot name only.

To add a detector, implement detect/1 against ExAbby.BotDetector.Context, which includes transient user_agent and an optional conn:

defmodule MyApp.BotDetector do
  def detect(%ExAbby.BotDetector.Context{conn: conn}) do
    if conn.assigns[:verified_bot], do: {:bot, :verified_bot}, else: :human
  end
end

config :ex_abby,
  bot_detection: [detectors: [ExAbby.BotDetector.UserAgent, MyApp.BotDetector]]

The stored and emitted bot name must be a stable atom; invalid custom detector responses are treated as :human so request evidence cannot leak into the session or telemetry.

Excluding historical trials

Use the batch APIs only from privileged server-side maintenance code. They are transactional and idempotent, retain the original variation, and can be safely reversed:

{:ok, excluded_count} = ExAbby.exclude_session_trials(session_ids, reason: "bot")
{:ok, restored_count} = ExAbby.restore_session_trials(session_ids)

Excluded trials are omitted from experiment summaries, conversion rates, and significance calculations, while the admin experiment view displays their total separately. An eligible human request restores only the trial for the experiment it revisits, preserving unrelated exclusions.

ExAbby.Experiments persistence functions and public raw session-ID/user-ID overloads do not have request bot status and therefore bypass this eligibility check. Normal Phoenix and LiveView integrations should use the Plug.Conn and socket public APIs after ExAbby.SessionPlug; host code that uses low-level assignment must perform bot-first eligibility itself.

Admin Routes

ExAbby includes a simple admin interface for viewing and managing experiments. To use it:

1. Add the routes to your router:

defmodule MyAppWeb.Router do
  use MyAppWeb, :router
  import ExAbby.Router  # add this line 

  scope "/admin", MyAppWeb do
    pipe_through [:browser, :admin_auth]
    ex_abby_admin_routes()
  end
end

1. Visit /admin/ab_tests to see a clean, Tailwind-styled interface showing:
   • List of all experiments
   • Experiment details and descriptions
   • Quick links to view individual experiments

Statistical significance

The experiment results view compares every treatment with the variation named by control_variation_name. It calculates two-sided, anytime-valid p-values by inverting beta-binomial mixture confidence sequences for the binary conversion rate in each arm. The calculation uses unique converters and trials in the selected date range; excluded trials do not contribute to either count.

ExAbby applies Holm's adjustment to reduce false positives from making several comparisons. Adjusted values below 0.05 are highlighted. The correction family is every configured treatment arm across every displayed success metric, including any arm that currently has no data. Either success metric can identify a winner; both do not need to be significant. The family is deliberately fixed by the experiment's design rather than by how many arms happen to have data at the moment you look: sizing it from the arms that are currently measurable would make the correction depend on the observed data, and would make p-values jump whenever an unrelated empty arm recorded its first trial. The cost is that a permanently empty arm — one with weight 0, or one created after the selected range began — still consumes a comparison for each displayed metric.

Each treatment also shows its absolute lift and a pairwise 95% anytime-valid confidence interval. These intervals are not Holm-adjusted. They answer "how large is this arm's lift" rather than "which arms survive the family-wise decision", so with several treatments an interval can exclude zero while the same row's adjusted p-value sits above 0.05. Read the highlighted p-value for the decision and the interval for the magnitude.

Sparse conversion counts produce wide intervals rather than unreliable approximations; Not enough data appears only when the control or treatment has no eligible trials.

The p-values and confidence intervals remain valid when results are checked repeatedly and the experiment is stopped based on them. This guarantee assumes the experiment's start date, metric, and eligibility rules were fixed independently of the observed results. Trying several date ranges or metrics and reporting only the best result is a separate multiple-testing problem.

Usage in Controllers

ExAbby supports multiple ways to identify users and sessions:

• Plug.Conn - For session-based experiments in controllers
• Phoenix.LiveView.Socket - For session-based experiments in LiveView
• User struct - Any struct/map with an id field (e.g., %{id: 123})
• Integer ID - Pass user ID directly (e.g., 123)
• String ID - Pass session ID directly (e.g., "session_abc123")

Session-based Example

In a controller action (e.g., PageController):

def index(conn, _params) do
  # Single variation example
  {conn, _variation} = ExAbby.get_variation(conn, "landing_page_test")
  
  # Multiple variations example
  {conn, _variations} = ExAbby.get_variations(conn, ["landing_page_test", "button_color_test"])
  render(conn, "index.html")
end

def record_conversion(conn, _params) do
  # Single experiment success recording
  ExAbby.record_success(conn, "landing_page_test")
  
  # Multiple experiment success recording
  ExAbby.record_successes(conn, ["landing_page_test", "button_color_test"])
  
  # Record with options (works for both single and multiple)
  ExAbby.record_successes(conn, ["landing_page_test", "button_color_test"], 
    amount: 99.99,
    success_type: :success1
  )
  
  redirect(conn, to: "/thank_you")
end

User-based Example

If you have a current_user:

def show(conn, _params) do
  user = conn.assigns.current_user
  
  # Single variation
  variation = ExAbby.get_variation(user, "dashboard_experiment")
  
  # Multiple variations
  variations = ExAbby.get_variations(user, ["dashboard_experiment", "feature_test"])
  render(conn, "show.html", ab_variations: variations)
end

def record_dashboard_success(conn, _params) do
  user = conn.assigns.current_user
  
  # Record multiple successes
  ExAbby.record_successes(user, ["dashboard_experiment", "feature_test"])
  redirect(conn, to: "/thanks")
end

Direct ID Usage

ExAbby now supports passing IDs directly without wrapping in a struct:

# Using integer user IDs directly
user_id = 12345
variation = ExAbby.get_variation(user_id, "experiment_name")
variations = ExAbby.get_variations(user_id, ["exp1", "exp2"])

# Record success with user ID
ExAbby.record_success(user_id, "experiment_name")
ExAbby.record_successes(user_id, ["exp1", "exp2"], amount: 99.99)

# Using session IDs directly (strings)
session_id = "abc123xyz"
variation = ExAbby.get_variation(session_id, "experiment_name")
variations = ExAbby.get_variations(session_id, ["exp1", "exp2"])

# Record success with session ID
ExAbby.record_success(session_id, "experiment_name")
ExAbby.record_successes(session_id, ["exp1", "exp2"], success_type: :success2)

This is useful when:

• You only have the user ID (not the full user struct)
• You're working with session IDs from external systems
• You want to run A/B tests in background jobs or processes without full context

Usage in LiveView

1. Ensure your endpoint/pipeline sets up a session and calls ExAbby.SessionPlug or something similar to create "ex_abby_session_id".
2. In your LiveView:

defmodule MyAppWeb.ButtonTestLive do
  use MyAppWeb, :live_view

  def mount(_params, session, socket) do
    # Get multiple variations at once
    socket = ExAbby.get_variations(socket, session, ["landing_page_test", "button_color_test"])
    {:ok, assign(socket, session: session)}
  end

  def render(assigns) do
    ~H"""
    <div class="max-w-md mx-auto mt-10 p-6 bg-white rounded-lg shadow-lg">
      <%= case @ex_abby_trials["landing_page_test"] do %>
        <% "hello_world" -> %>
          <div>Hello World!</div>
        <% _ -> %>
          <div>This is the control</div>
      <% end %>

      <button 
        phx-click="convert" 
        class={get_button_class(@ex_abby_trials["button_color_test"])}
      >
        Click Me!
      </button>
    </div>
    """
  end

  def handle_event("convert", _params, socket) do
    case ExAbby.record_successes(socket, ["landing_page_test", "button_color_test"]) do
      {:ok, _trial} ->
        {:noreply, put_flash(socket, :info, "Conversion recorded!")}
      {:error, _reason} ->
        {:noreply, put_flash(socket, :error, "Failed to record conversion")}
    end
  end

  # Helper function for button styling based on variation
  defp get_button_class("blue"), do: "bg-blue-500 text-white rounded hover:bg-blue-600"
  defp get_button_class("green"), do: "bg-green-500 text-white rounded hover:bg-green-600"
  defp get_button_class(_), do: "bg-gray-500 text-white rounded hover:bg-gray-600"
end

The variations are stored in @ex_abby_trials as a map where:

• Keys are experiment names (e.g., "landing_page_test")
• Values are variation names (e.g., "hello_world", "control")

Recording Conversions with Options

You can record conversions with additional options:

# Record a conversion with an amount
ExAbby.record_success(socket, "button_color_test",
  amount: 100.0,
  success_type: :success2
)

# Record multiple conversions at once
ExAbby.record_successes(socket, ["landing_page_test", "button_color_test"])

Available options:

• :amount - Optional numeric value to track with the success (default: 0.0)
• :success_type - Type of success to record, either :success1 or :success2 (default: :success1)

Linking Sessions to Users

When a user signs up or logs in, you can link their session-based experiment trials to their user account. This allows you to track how users who signed up through different experiment variations perform over time.

Why Link Sessions to Users?

Session-based experiments are great for testing landing pages and signup flows. But once a user creates an account, you want to track their long-term behavior (purchases, engagement, retention) tied to the original experiment variation they saw.

Usage

In Controllers (Plug.Conn):

def create(conn, %{"user" => user_params}) do
  case Accounts.create_user(user_params) do
    {:ok, user} ->
      # Link all session experiments to the new user
      conn = ExAbby.link_session_to_user(conn, user)

      # Or link specific experiments only
      conn = ExAbby.link_session_to_user(conn, user, ["signup_flow_test", "landing_page_test"])

      conn
      |> put_flash(:info, "Account created!")
      |> redirect(to: "/dashboard")

    {:error, changeset} ->
      render(conn, "new.html", changeset: changeset)
  end
end

In LiveView:

def handle_event("register", %{"user" => user_params}, socket) do
  case Accounts.create_user(user_params) do
    {:ok, user} ->
      # Link all session experiments to the new user
      socket = ExAbby.link_session_to_user(socket, user)

      # Or link specific experiments
      socket = ExAbby.link_session_to_user(socket, user, ["signup_flow_test"])

      {:noreply, push_navigate(socket, to: "/dashboard")}

    {:error, changeset} ->
      {:noreply, assign(socket, changeset: changeset)}
  end
end

Flexible User Identification

You can pass the user in different ways:

# Pass user struct (must have :id field)
ExAbby.link_session_to_user(conn, user)

# Pass user ID directly
ExAbby.link_session_to_user(conn, 12345)

What Happens When You Link

1. The session trial's user_id field is updated to the provided user ID
2. Future success recordings for that user will be associated with the original variation
3. You can now analyze user-based metrics (lifetime value, retention) by experiment variation

Archiving Experiments

Once an experiment has concluded and you've determined a winner (or decided to end it), you can archive it. Archived experiments:

• Stop accepting new trials - New users won't be assigned to the experiment
• Preserve existing data - All historical trials and conversions remain
• Can declare a winner - Optionally mark which variation won
• Are hidden by default - Admin UI shows Active experiments by default

Archiving via Admin UI

1. Navigate to /admin/ex_abby/:id (experiment detail page)
2. Select an optional winner from the dropdown
3. Click "Archive Experiment"

To unarchive, click the "Unarchive" button on the archived experiment.

Archiving via Seeds

You can archive experiments directly in your seeds file:

experiments = [
  # Active experiment
  {
    "current_test",
    "Currently running experiment",
    [{"control", 0.5}, {"variant", 0.5}],
    [success1_label: "Signup", update_weights: false]
  },

  # Archived experiment with winner
  {
    "old_test",
    "Completed experiment",
    [{"control", 0.5}, {"variant", 0.5}],
    [archived: true, winner: "variant", update_weights: false]
  }
]

Enum.each(experiments, fn {name, description, variations, opts} ->
  ExAbby.upsert_experiment_and_update_weights(name, description, variations, opts)
end)

Important: If you archive an experiment via the Admin UI and then run seeds without archived: true, the experiment will remain archived. Seeds only modify the archived status when explicitly specified.

Archiving Programmatically

# Archive without winner
ExAbby.Experiments.archive_experiment(experiment_id)

# Archive with winner (by variation name)
ExAbby.Experiments.archive_experiment(experiment_id, "variant_a")

# Archive with winner (by variation ID)
ExAbby.Experiments.archive_experiment(experiment_id, 123)

# Unarchive
ExAbby.Experiments.unarchive_experiment(experiment_id)

Filtering Experiments

# List only active experiments (default in Admin UI)
ExAbby.Experiments.list_experiments(status: :active)

# List only archived experiments
ExAbby.Experiments.list_experiments(status: :archived)

# List all experiments
ExAbby.Experiments.list_experiments(status: :all)

Troubleshooting

• No Ecto repo configured for :ex_abby
  Add config :ex_abby, repo: MyApp.Repo in your host app’s config.exs.

• No Experiment Found
  If you see a warning for no experiment found, make sure you have seeded the database wtih experiments nd variations.

Enjoy A/B testing with ExAbby! Feel free to customize it further for bandit algorithms, Bayesian stats, or other advanced features.