ExAbby

ExAbby is a minimal A/B testing library for Elixir/Phoenix.

Caveat: This was created primarily over a weekend with the help of Chat GPT/Claude. The code is working but still needs a lot of cleanup and optimizations, which I'll do as I run into problems. As it stands, it is working under low load in production settings providing a super easy way to AB test Phoenix and Liveview using assigns.

Why Ex Abby?

I have found there are no super simple ways to get ab testing working for smaller sites in Elixir. You have to pay $$ or use a complex system. And everything has moved to feature tagging. This experiment framework is based on something we built in-house for a previous company that reached virality co-efficients of 1.0 a few times. And the goal is to make it super easy to use in Liveview environments.

This is really early and the API is 100% likely to change. Feedback is appreciated!

It supports:

• Ecto-based storage (Experiments, Variations, Trials)
• Session-based or User-based assignment
• Linking session trials to users - Track user performance from session-based experiments
• Archive experiments with optional winner declaration
• Flexible ID support: structs, integers, or strings
• Weighted randomization
• Recording success events
• LiveView helpers (checking connected?/1 and storing assigned variation)
• Admin LiveViews with experiment filtering (Active/Archived/All)
• Upserting experiments/variations with optional weight updates
• Reviewing results over different time periods
• Ability to toggle variations by user or session for testing

Coming in the future

• armed bandits
• optimizations / caching
• statistical significance
• better UX of admin screens
• So much cleanup
• Likely changes to the API.

Table of Contents

1. Installation
2. Configuration
3. Migrations
4. Upgrading from v0.1 to v0.2
5. Upserting Experiments and Updating Weights
6. Session Setup
7. Admin Routes
8. Usage in Controllers
9. Usage in LiveView
10. Linking Sessions to Users
11. Archiving Experiments
12. Production Deployment
13. Troubleshooting

Installation

1. Addex_abby as a dependency in your Phoenix (host) app’s mix.exs.

   If it's a public Hex package (or if you plan to publish it):

   defp deps do
     [
       {:ex_abby, "~> 0.2.0"}
     ]
   end

   If it's a GitHub repo (private or public):

   defp deps do
     [
       {:ex_abby, github: "grahac/ex_abby", tag: "0.2.0"}
     ]
   end

2. Run:

   mix deps.get

Configuration

In your host app’s config/config.exs (or dev.exs, etc.), set:

config :ex_abby,
  repo: MyApp.Repo

Where MyApp.Repo is your Ecto Repo module.

Migrations

ExAbby provides Ecto migrations that create three tables:

• ex_abby_experiments
• ex_abby_variations
• ex_abby_trials

In your host app, generate a new migration:

mix ecto.gen.migration create_ex_abby_tables

Open priv/repo/migrations/2025xxxxxx_create_ex_abby_tables.exs, and add:

defmodule MyApp.Repo.Migrations.CreateExAbbyTables do
  use Ecto.Migration

  def up do
    ExAbby.Migrations.create_tables()
  end

  def down do
    ExAbby.Migrations.drop_tables()
  end
end

Then run:

mix ecto.migrate

Upgrading from v0.1 to v0.2

Version 0.2.0 adds experiment archiving with optional winner declaration. To upgrade:

1. Update your dependency in mix.exs:

   {:ex_abby, "~> 0.2.0"}

2. Create a new migration:

   mix ecto.gen.migration ex_abby_v2

3. Add to the generated migration file:

   defmodule MyApp.Repo.Migrations.ExAbbyV2 do
   use Ecto.Migration
   
   def up, do: ExAbby.Migrations.v1_to_v2()
   def down, do: ExAbby.Migrations.v2_to_v1()
   end

4. Run the migration:

   mix ecto.migrate

This adds two new columns to ex_abby_experiments:

• archived_at - Timestamp when experiment was archived
• winner_variation_id - Reference to the winning variation (optional)

Upserting Experiments and Updating Weights

If you have a function like:

ExAbby.upsert_experiment_and_update_weights(
  "landing_page_test",
  "Testing different landing pages",
  [
    {"Original", 1.0},
    {"Variation A", 1.0},
    {"Variation B", 2.0}
  ],
  success1_label: "Signup",
  success2_label: "Purchase"
)

Then:

• If "landing_page_test"does not exist, the library creates a new experiment with that name + description, and 3 variations with the specified weights.
• If the experiment already exists, we do not change its weights. We update all the other info if
• you can optionally add labels to label success. This is just for readability and is optional.

Seeding Experiments

Create a file priv/repo/seeds/experiments.exs to define your experiments:

experiments = [
  {
    "button_color_test",
    "Testing different button colors for signup",
    [
      {"control", 0.33},
      {"green", 0.33}, 
      {"blue", 0.33}
    ],
    [success1_label: "Signup", success2_label: "Purchase", update_weights: false]
  }
]

# Seed or update experiments without modifying weights
Enum.each(experiments, fn {name, description, variations, opts} ->
  ExAbby.upsert_experiment_and_update_weights(name, description, variations, opts)
end)

Then in your priv/repo/seeds.exs, add:

Code.require_file("seeds/experiments.exs", __DIR__)

You can run the seeds in different ways:

Development:

mix run priv/repo/seeds.exs

Session Setup

To enable session-based A/B testing, add ExAbby.SessionPlug to your endpoint or router pipeline:

# In  your router pipeline (recommended):

pipeline :browser do
  # ... other plugs ...
  plug ExAbby.SessionPlug
end

# In lib/your_app_web/endpoint.ex
plug Plug.Session,
  store: :cookie,
  key: "_your_app_key",
  signing_salt: "your_signing_salt"

plug ExAbby.SessionPlug

This plug creates a unique "ex_abby_session_id" for tracking A/B test variations across requests.

Admin Routes

ExAbby includes a simple admin interface for viewing and managing experiments. To use it:

1. Add the routes to your router:

defmodule MyAppWeb.Router do
  use MyAppWeb, :router
  import ExAbby.Router  # add this line 

  scope "/admin", MyAppWeb do
    pipe_through [:browser, :admin_auth]
    ex_abby_admin_routes()
  end
end

1. Visit /admin/ab_tests to see a clean, Tailwind-styled interface showing:
   • List of all experiments
   • Experiment details and descriptions
   • Quick links to view individual experiments

Usage in Controllers

ExAbby supports multiple ways to identify users and sessions:

• Plug.Conn - For session-based experiments in controllers
• Phoenix.LiveView.Socket - For session-based experiments in LiveView
• User struct - Any struct/map with an id field (e.g., %{id: 123})
• Integer ID - Pass user ID directly (e.g., 123)
• String ID - Pass session ID directly (e.g., "session_abc123")

Session-based Example

In a controller action (e.g., PageController):

def index(conn, _params) do
  # Single variation example
  {conn, _variation} = ExAbby.get_variation(conn, "landing_page_test")
  
  # Multiple variations example
  {conn, _variations} = ExAbby.get_variations(conn, ["landing_page_test", "button_color_test"])
  render(conn, "index.html")
end

def record_conversion(conn, _params) do
  # Single experiment success recording
  ExAbby.record_success(conn, "landing_page_test")
  
  # Multiple experiment success recording
  ExAbby.record_successes(conn, ["landing_page_test", "button_color_test"])
  
  # Record with options (works for both single and multiple)
  ExAbby.record_successes(conn, ["landing_page_test", "button_color_test"], 
    amount: 99.99,
    success_type: :success1
  )
  
  redirect(conn, to: "/thank_you")
end

User-based Example

If you have a current_user:

def show(conn, _params) do
  user = conn.assigns.current_user
  
  # Single variation
  variation = ExAbby.get_variation(user, "dashboard_experiment")
  
  # Multiple variations
  variations = ExAbby.get_variations(user, ["dashboard_experiment", "feature_test"])
  render(conn, "show.html", ab_variations: variations)
end

def record_dashboard_success(conn, _params) do
  user = conn.assigns.current_user
  
  # Record multiple successes
  ExAbby.record_successes(user, ["dashboard_experiment", "feature_test"])
  redirect(conn, to: "/thanks")
end

Direct ID Usage

ExAbby now supports passing IDs directly without wrapping in a struct:

# Using integer user IDs directly
user_id = 12345
variation = ExAbby.get_variation(user_id, "experiment_name")
variations = ExAbby.get_variations(user_id, ["exp1", "exp2"])

# Record success with user ID
ExAbby.record_success(user_id, "experiment_name")
ExAbby.record_successes(user_id, ["exp1", "exp2"], amount: 99.99)

# Using session IDs directly (strings)
session_id = "abc123xyz"
variation = ExAbby.get_variation(session_id, "experiment_name")
variations = ExAbby.get_variations(session_id, ["exp1", "exp2"])

# Record success with session ID
ExAbby.record_success(session_id, "experiment_name")
ExAbby.record_successes(session_id, ["exp1", "exp2"], success_type: :success2)

This is useful when:

• You only have the user ID (not the full user struct)
• You're working with session IDs from external systems
• You want to run A/B tests in background jobs or processes without full context

Usage in LiveView

1. Ensure your endpoint/pipeline sets up a session and calls ExAbby.SessionPlug or something similar to create "ex_abby_session_id".
2. In your LiveView:

defmodule MyAppWeb.ButtonTestLive do
  use MyAppWeb, :live_view

  def mount(_params, session, socket) do
    # Get multiple variations at once
    socket = ExAbby.get_variations(socket, session, ["landing_page_test", "button_color_test"])
    {:ok, assign(socket, session: session)}
  end

  def render(assigns) do
    ~H"""
    <div class="max-w-md mx-auto mt-10 p-6 bg-white rounded-lg shadow-lg">
      <%= case @ex_abby_trials["landing_page_test"] do %>
        <% "hello_world" -> %>
          <div>Hello World!</div>
        <% _ -> %>
          <div>This is the control</div>
      <% end %>

      <button 
        phx-click="convert" 
        class={get_button_class(@ex_abby_trials["button_color_test"])}
      >
        Click Me!
      </button>
    </div>
    """
  end

  def handle_event("convert", _params, socket) do
    case ExAbby.record_successes(socket, ["landing_page_test", "button_color_test"]) do
      {:ok, _trial} ->
        {:noreply, put_flash(socket, :info, "Conversion recorded!")}
      {:error, _reason} ->
        {:noreply, put_flash(socket, :error, "Failed to record conversion")}
    end
  end

  # Helper function for button styling based on variation
  defp get_button_class("blue"), do: "bg-blue-500 text-white rounded hover:bg-blue-600"
  defp get_button_class("green"), do: "bg-green-500 text-white rounded hover:bg-green-600"
  defp get_button_class(_), do: "bg-gray-500 text-white rounded hover:bg-gray-600"
end

The variations are stored in @ex_abby_trials as a map where:

• Keys are experiment names (e.g., "landing_page_test")
• Values are variation names (e.g., "hello_world", "control")

Recording Conversions with Options

You can record conversions with additional options:

# Record a conversion with an amount
ExAbby.record_success(socket, "button_color_test",
  amount: 100.0,
  success_type: :success2
)

# Record multiple conversions at once
ExAbby.record_successes(socket, ["landing_page_test", "button_color_test"])

Available options:

• :amount - Optional numeric value to track with the success (default: 0.0)
• :success_type - Type of success to record, either :success1 or :success2 (default: :success1)

Linking Sessions to Users

When a user signs up or logs in, you can link their session-based experiment trials to their user account. This allows you to track how users who signed up through different experiment variations perform over time.

Why Link Sessions to Users?

Session-based experiments are great for testing landing pages and signup flows. But once a user creates an account, you want to track their long-term behavior (purchases, engagement, retention) tied to the original experiment variation they saw.

Usage

In Controllers (Plug.Conn):

def create(conn, %{"user" => user_params}) do
  case Accounts.create_user(user_params) do
    {:ok, user} ->
      # Link all session experiments to the new user
      conn = ExAbby.link_session_to_user(conn, user)

      # Or link specific experiments only
      conn = ExAbby.link_session_to_user(conn, user, ["signup_flow_test", "landing_page_test"])

      conn
      |> put_flash(:info, "Account created!")
      |> redirect(to: "/dashboard")

    {:error, changeset} ->
      render(conn, "new.html", changeset: changeset)
  end
end

In LiveView:

def handle_event("register", %{"user" => user_params}, socket) do
  case Accounts.create_user(user_params) do
    {:ok, user} ->
      # Link all session experiments to the new user
      socket = ExAbby.link_session_to_user(socket, user)

      # Or link specific experiments
      socket = ExAbby.link_session_to_user(socket, user, ["signup_flow_test"])

      {:noreply, push_navigate(socket, to: "/dashboard")}

    {:error, changeset} ->
      {:noreply, assign(socket, changeset: changeset)}
  end
end

Flexible User Identification

You can pass the user in different ways:

# Pass user struct (must have :id field)
ExAbby.link_session_to_user(conn, user)

# Pass user ID directly
ExAbby.link_session_to_user(conn, 12345)

What Happens When You Link

1. The session trial's user_id field is updated to the provided user ID
2. Future success recordings for that user will be associated with the original variation
3. You can now analyze user-based metrics (lifetime value, retention) by experiment variation

Archiving Experiments

Once an experiment has concluded and you've determined a winner (or decided to end it), you can archive it. Archived experiments:

• Stop accepting new trials - New users won't be assigned to the experiment
• Preserve existing data - All historical trials and conversions remain
• Can declare a winner - Optionally mark which variation won
• Are hidden by default - Admin UI shows Active experiments by default

Archiving via Admin UI

1. Navigate to /admin/ex_abby/:id (experiment detail page)
2. Select an optional winner from the dropdown
3. Click "Archive Experiment"

To unarchive, click the "Unarchive" button on the archived experiment.

Archiving via Seeds

You can archive experiments directly in your seeds file:

experiments = [
  # Active experiment
  {
    "current_test",
    "Currently running experiment",
    [{"control", 0.5}, {"variant", 0.5}],
    [success1_label: "Signup", update_weights: false]
  },

  # Archived experiment with winner
  {
    "old_test",
    "Completed experiment",
    [{"control", 0.5}, {"variant", 0.5}],
    [archived: true, winner: "variant", update_weights: false]
  }
]

Enum.each(experiments, fn {name, description, variations, opts} ->
  ExAbby.upsert_experiment_and_update_weights(name, description, variations, opts)
end)

Important: If you archive an experiment via the Admin UI and then run seeds without archived: true, the experiment will remain archived. Seeds only modify the archived status when explicitly specified.

Archiving Programmatically

# Archive without winner
ExAbby.Experiments.archive_experiment(experiment_id)

# Archive with winner (by variation name)
ExAbby.Experiments.archive_experiment(experiment_id, "variant_a")

# Archive with winner (by variation ID)
ExAbby.Experiments.archive_experiment(experiment_id, 123)

# Unarchive
ExAbby.Experiments.unarchive_experiment(experiment_id)

Filtering Experiments

# List only active experiments (default in Admin UI)
ExAbby.Experiments.list_experiments(status: :active)

# List only archived experiments
ExAbby.Experiments.list_experiments(status: :archived)

# List all experiments
ExAbby.Experiments.list_experiments(status: :all)

Production Deployment

ExAbby experiments can be seeded automatically during your migration process.

1. Update Mix Release Configuration

In your mix.exs, ensure you have the releases configuration:

def releases do
  [
    memoir: [
      include_erts: true,
      include_executables_for: [:unix],
      applications: [runtime_tools: :permanent],
      overlays: ["priv/repo/seeds"]
    ]
  ]
end

1. Add Release Module Function

In lib/your_app/release.ex:

defmodule YourApp.Release do
  # ... existing release module code ...

  def seed_experiments do
    load_app()
    repo = Application.get_env(:ex_abby, :repo)
    
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, fn _repo ->
      seed_path = Application.app_dir(@app, "priv/repo/seeds/experiments.exs")
      Code.eval_file(seed_path)
    end)
  end
end

1. Update Migration Script

Your existing rel/overlays/bin/migrate script will now run both migrations and seeds:

#!/bin/sh

./memoir eval "Memoir.Release.migrate"
./memoir eval "Memoir.Release.seed_experiments"

Now your experiments will be automatically seeded whenever you run migrations using:

bin/migrate

This will create or update your experiments while preserving existing weights for any experiments that already exist.

Troubleshooting

• No Ecto repo configured for :ex_abby
  Add config :ex_abby, repo: MyApp.Repo in your host app’s config.exs.

• No Experiment Found
  If you see a warning for no experiment found, make sure you have seeded the database wtih experiments nd variations.

Enjoy A/B testing with ExAbby! Feel free to customize it further for bandit algorithms, Bayesian stats, or other advanced features.