Omni

Universal Elixir client for LLM APIs. Streaming text generation, tool use, and structured output.

Features

• Multi-provider — supports many LLM providers out of the box (see table below)
• Streaming-first — all requests stream by default; generate_text is built on stream_text
• Tool use — define tools with schemas and handlers; the loop auto-executes and feeds results back
• Structured output — JSON Schema validation with constrained decoding and automatic retries
• Pluggable model catalog — bundled models.dev snapshot by default, with optional live fetching or the llm_db package as alternative sources
• Extensible — add custom providers by implementing a behaviour

Installation

Add Omni to your dependencies:

def deps do
  [
    {:omni, "~> 1.6"}
  ]
end

Each built-in provider reads its API key from a standard environment variable by default — if your keys are set, no configuration is needed:

A local-Ollama provider (Omni.Providers.Ollama) is also included; it needs no API key — configure it with the models you have pulled locally.

All built-in providers are loaded by default. To limit what loads at startup, list provider modules (:builtins names all built-ins and may appear in the list):

config :omni, :models, providers: [Omni.Providers.Anthropic, Omni.Providers.OpenAI]

# or: everything built-in plus a custom provider
config :omni, :models, providers: [:builtins, MyApp.Providers.Acme]

Quick start

Text generation

Pass a model tuple and a string:

{:ok, response} = Omni.generate_text({:anthropic, "claude-sonnet-4-5-20250514"}, "Hello!")

response.message
#=> %Omni.Message{role: :assistant, content: [%Omni.Content.Text{text: "Hello! How can..."}]}

For multi-turn conversations, build a context with a system prompt and messages:

context = Omni.context(
  system: "You are a helpful assistant.",
  messages: [
    Omni.message(role: :user, content: "What is Elixir?"),
    Omni.message(role: :assistant, content: "Elixir is a functional programming language..."),
    Omni.message(role: :user, content: "How does it handle concurrency?")
  ]
)

{:ok, response} = Omni.generate_text({:anthropic, "claude-sonnet-4-5-20250514"}, context)

Streaming

stream_text returns a StreamingResponse that you consume with event handlers:

{:ok, stream} = Omni.stream_text({:anthropic, "claude-sonnet-4-5-20250514"}, "Tell me a story")

{:ok, response} =
  stream
  |> Omni.StreamingResponse.on(:text_delta, fn %{delta: text} -> IO.write(text) end)
  |> Omni.StreamingResponse.complete()

For simple cases where you just need the text chunks:

stream
|> Omni.StreamingResponse.text_stream()
|> Enum.each(&IO.write/1)

Structured output

Pass a schema via the :output option to get validated, decoded output:

schema = Omni.Schema.object(%{
  name: Omni.Schema.string(description: "The capital city"),
  population: Omni.Schema.integer(description: "Approximate population")
}, required: [:name, :population])

{:ok, response} =
  Omni.generate_text(
    {:anthropic, "claude-sonnet-4-5-20250514"},
    "What is the capital of France?",
    output: schema
  )

response.output
#=> %{name: "Paris", population: 2161000}

Tool use

Define tools with schemas and handlers — the loop automatically executes tool uses and feeds results back to the model:

weather_tool = Omni.tool(
  name: "get_weather",
  description: "Gets the current weather for a city",
  input_schema: Omni.Schema.object(
    %{city: Omni.Schema.string(description: "City name")},
    required: [:city]
  ),
  handler: fn input -> "72°F and sunny in #{input.city}" end
)

context = Omni.context(
  messages: [Omni.message("What's the weather in London?")],
  tools: [weather_tool]
)

{:ok, response} = Omni.generate_text({:anthropic, "claude-sonnet-4-5-20250514"}, context)

Model catalog

Model data loads at startup from a pluggable source. The default reads a bundled models.dev snapshot — no network access required. The snapshot is refreshed with every release, with data-only patch releases for notable catalog changes (major model launches, pricing changes) and at least one release per month. If you need fresher data than that, fetch the catalog at boot instead (disk-cached, degrading gracefully to the bundled snapshot when models.dev is unreachable):

config :omni, :models,
  source: {Omni.Sources.ModelsDev, live: true}

Or source the catalog from the optional llm_db package (requires {:llm_db, "~> 2026.7"} in your deps):

config :omni, :models, source: Omni.Sources.LLMDB

Sources can also be pinned per provider (config :omni, Omni.Providers.OpenAI, source: ...), and custom sources implement the Omni.Source behaviour. See the HexDocs for details.

Documentation

Full API documentation is available on HexDocs.

License

This package is open source and released under the Apache-2 License.

© Copyright 2024-2026 Push Code Ltd.