Alike 〰️

Semantic similarity testing for Elixir.

Alike lets you test if two sentences convey the same meaning using the expressive wave operator<~>. Perfect for testing LLM outputs, chatbots, NLP pipelines, or any application that generates natural language.

assert "The cat is sleeping" <~> "A feline is taking a nap"

Features

• Wave operator (<~>) - Beautiful, expressive test assertions
• Semantic understanding - Detects meaning, not just string matches
• Contradiction detection - Catches logical contradictions like "The sky is blue" vs "The sky is red"
• Local models - Runs entirely on your machine, no API keys needed
• Configurable - Tune thresholds for your use case

Installation

Add alike to your test dependencies in mix.exs:

def deps do
  [
    {:alike, "~> 0.1.0", only: :test}
  ]
end

Then run:

mix deps.get

Note: On first use, Alike downloads the required models (~460MB total). This only happens once and models are cached in ~/.cache/bumblebee/.

Quick Start

defmodule MyAppTest do
  use ExUnit.Case
  import Alike.WaveOperator

  test "chatbot responds appropriately" do
    response = MyChatbot.greet("Hello!")

    assert response <~> "Hi there! How can I help you today?"
  end

  test "summary captures key points" do
    summary = MySummarizer.summarize(long_article)

    assert summary <~> "The article discusses climate change impacts on agriculture"
  end
end

Usage

The Wave Operator

The <~> operator is the recommended way to test semantic similarity:

import Alike.WaveOperator

# Similar sentences pass
assert "The quick brown fox jumps over the lazy dog"
       <~> "A fast auburn fox leaps over a sleeping canine"

# Different meanings fail
refute "The weather is nice today" <~> "I enjoy reading books"

# Contradictions are detected
refute "The sky is blue" <~> "The sky is red"

Direct Function Calls

For more control, use Alike.alike?/3 directly:

# Basic usage
Alike.alike?("The cat is sleeping", "A feline is taking a nap")
# => true

# Custom similarity threshold
Alike.alike?("Hello world", "Hi there", threshold: 0.6)
# => true or false depending on threshold

# Disable contradiction checking for speed
Alike.alike?("Some text", "Other text", check_contradiction: false)

Raw Similarity Scores

Get the underlying similarity score (0.0 to 1.0):

{:ok, score} = Alike.similarity("Hello world", "Hi there")
# => {:ok, 0.623}

NLI Classification

Classify the relationship between sentences:

{:ok, result} = Alike.classify("The sky is blue", "The sky is red")
# => {:ok, %{label: "contradiction", score: 0.998}}

{:ok, result} = Alike.classify("The cat is sleeping", "An animal is resting")
# => {:ok, %{label: "entailment", score: 0.892}}

NLI Labels:

• "entailment" - The second sentence logically follows from the first (e.g., "A cat is sleeping" entails "An animal is resting")
• "contradiction" - The sentences cannot both be true (e.g., "The sky is blue" contradicts "The sky is red")
• "neutral" - The sentences are related but neither entails nor contradicts the other

Configuration

Customize thresholds in your config/config.exs:

config :alike,
  # Minimum similarity score to consider sentences "alike" (0.0 to 1.0)
  similarity_threshold: 0.45,

  # Minimum confidence to flag a contradiction (0.0 to 1.0)
  contradiction_threshold: 0.8

How It Works

Alike uses two models that run locally on your machine, powered by Bumblebee and Nx:

1. Sentence Embeddings (all-MiniLM-L12-v2) - Converts sentences into 384-dimensional vectors and computes cosine similarity

2. NLI Model (nli-distilroberta-base) - Detects contradictions that embeddings alone might miss

The combination catches both semantic similarity AND logical contradictions.

Interactive Tutorial

Click the "Run in Livebook" badge above to try the interactive tutorial, or check the livebook directory.

Examples

Testing LLM Outputs

test "LLM generates appropriate response" do
  prompt = "Explain photosynthesis in simple terms"
  response = MyLLM.generate(prompt)

  assert response <~> "Plants convert sunlight into energy through a process called photosynthesis"
end

Testing Translations

test "translation preserves meaning" do
  original = "The weather is beautiful today"
  translated = MyTranslator.translate(original, to: :spanish) |> MyTranslator.translate(to: :english)

  assert original <~> translated
end

Testing Summarization

test "summary captures main idea" do
  article = "Long article about renewable energy..."
  summary = MySummarizer.summarize(article)

  assert summary <~> "The article discusses the benefits and challenges of renewable energy adoption"
end

Speeding Up CI

Cache the Bumblebee models directory to avoid re-downloading on every CI run:

# GitHub Actions example
- name: Cache Bumblebee models
  uses: actions/cache@v3
  with:
    path: ~/.cache/bumblebee
    key: ${{ runner.os }}-bumblebee-${{ hashFiles('mix.lock') }}

Requirements

• Elixir 1.18+
• ~460MB disk space for models (downloaded on first use)
• Models are cached in ~/.cache/bumblebee/

License

Apache License 2.0 - see LICENSE for details.

Author

George Guimarães