AgentObs

Hex.pmDocumentationCI

An Elixir library for LLM agent observability.

AgentObs provides a simple, powerful, and idiomatic interface for instrumenting LLM agentic applications with telemetry events. It supports multiple observability backends through a pluggable handler architecture.

Features

Architecture

AgentObs uses a two-layer architecture:

Layer 1: Core Telemetry API (Backend-Agnostic)

Layer 2: Pluggable Backend Handlers

Installation

Add agent_obs to your list of dependencies in mix.exs:

def deps do

  [

    {:agent_obs, "~> 0.1.0"}

  ]

end

Quick Start

1. Configure AgentObs

# config/config.exs

config :agent_obs,

  enabled: true,

  handlers: [AgentObs.Handlers.Phoenix]



# config/runtime.exs (for Arize Phoenix)

config :opentelemetry,

  span_processor: :batch,

  resource: [service: [name: "my_llm_agent"]]



config :opentelemetry_exporter,

  otlp_protocol: :http_protobuf,

  otlp_endpoint: System.get_env("ARIZE_PHOENIX_OTLP_ENDPOINT", "http://localhost:6006"),

  otlp_headers: []

# Note: /v1/traces is automatically appended by the exporter

2. Instrument Your Agent

defmodule MyApp.WeatherAgent do

  def get_forecast(city) do

    AgentObs.trace_agent("weather_forecast", %{input: "What's the weather in #{city}?"}, fn ->

      # Call LLM to determine tool to use

      {:ok, tool_call, _metadata} = call_llm_for_planning(city)



      # Execute the tool

      {:ok, weather_data} = AgentObs.trace_tool("get_weather_api", %{

        arguments: %{city: city}

      }, fn ->

        {:ok, %{temp: 72, condition: "sunny"}}

      end)



      # Return final result

      {:ok, "The weather in #{city} is #{weather_data.condition}", %{

        tools_used: ["get_weather_api"],

        iterations: 1

      }}

    end)

  end



  defp call_llm_for_planning(city) do

    AgentObs.trace_llm("gpt-4o", %{

      input_messages: [%{role: "user", content: "Get weather for #{city}"}]

    }, fn ->

      # Simulate LLM API call

      response = call_openai(...)



      {:ok, response, %{

        output_messages: [%{role: "assistant", content: response}],

        tokens: %{prompt: 50, completion: 25, total: 75},

        cost: 0.00012

      }}

    end)

  end

end

3. View Traces in Arize Phoenix

Start a local Phoenix instance:

docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest

Navigate to http://localhost:6006 to view your traces with:

Handlers

Phoenix Handler (OpenInference)

Translates events to OpenInference semantic conventions for Arize Phoenix:

config :agent_obs,

  handlers: [AgentObs.Handlers.Phoenix]

Generic Handler (Basic OpenTelemetry)

Creates basic OpenTelemetry spans without OpenInference:

config :agent_obs,

  handlers: [AgentObs.Handlers.Generic]

Multiple Handlers

Use multiple backends simultaneously:

config :agent_obs,

  handlers: [

    AgentObs.Handlers.Phoenix,  # For detailed LLM observability

    AgentObs.Handlers.Generic   # For APM integration

  ]

ReqLLM Integration (Optional)

For applications using ReqLLM, AgentObs provides high-level helpers that automatically instrument LLM calls with full observability:

# Add to your deps

{:req_llm, "~> 1.0.0-rc.7"}



# Non-streaming text generation

{:ok, response} =

  AgentObs.ReqLLM.trace_generate_text(

    "anthropic:claude-3-5-sonnet",

    [%{role: "user", content: "Hello!"}]

  )



text = ReqLLM.Response.text(response)



# Streaming text generation

{:ok, stream_response} =

  AgentObs.ReqLLM.trace_stream_text(

    "anthropic:claude-3-5-sonnet",

    [%{role: "user", content: "Tell me a story"}]

  )



stream_response.stream

|> Stream.filter(&(&1.type == :content))

|> Stream.each(&IO.write(&1.text))

|> Stream.run()



# Structured data generation

schema = [name: [type: :string, required: true], age: [type: :pos_integer]]



{:ok, response} =

  AgentObs.ReqLLM.trace_generate_object(

    "anthropic:claude-3-5-sonnet",

    [%{role: "user", content: "Generate a person"}],

    schema

  )



object = ReqLLM.Response.object(response)

#=> %{name: "Alice", age: 30}

Benefits:

See the demo agent and ReqLLM integration guide for complete examples.

Jido Integration (Optional)

For applications using Jido, AgentObs provides AgentObs.JidoTracer — a drop-in Jido.Observe.Tracer implementation that automatically instruments all composer events with OpenTelemetry spans.

# Add to your deps

{:jido, "~> 2.0"}



# Configure Jido to use the tracer

config :jido, :observability,

  tracer: AgentObs.JidoTracer

That's it. All [:jido, :composer, :agent|:llm|:tool] events are automatically mapped to AgentObs event types and traced with OpenInference semantic conventions. Parent-child span nesting is preserved, so you get a full trace tree in Phoenix:

weather_assistant (agent)

  ├── gpt-4o #1 (llm)

  ├── get_weather (tool)

  └── gpt-4o #2 (llm)

See the Jido integration guide for details on event mapping, metadata translation, and advanced usage.

API Reference

High-Level Instrumentation

ReqLLM Helpers (Optional)

Text Generation:

Structured Data Generation:

Tool Execution:

Stream Helpers:

Low-Level API

See the full documentation for detailed API reference and examples.

Testing

Running Tests

# Run all tests (unit tests only, 99 tests)

mix test



# Include integration tests (requires API keys)

mix test --include integration



# Run only integration tests

mix test --only integration

ReqLLM Integration Tests

The ReqLLM module includes comprehensive test coverage with 193 tests:

Unit Tests (185 tests) - Run by default, use mocked streams:

Integration Tests (8 tests) - Excluded by default, require real LLM API calls:

To run integration tests, set one of these environment variables:

export ANTHROPIC_API_KEY=your_key  # Uses claude-3-5-haiku-latest

# OR

export OPENAI_API_KEY=your_key     # Uses gpt-4o-mini

# OR

export GOOGLE_API_KEY=your_key     # Uses gemini-2.0-flash-exp



mix test --include integration

If no API key is configured, integration tests gracefully skip without failing.

Development

Quick Commands

# Install dependencies

mix deps.get



# Run pre-commit checks (format, test, credo)

mix precommit



# Run CI checks (format check, test, credo)

mix ci

Individual Commands

# Run tests

mix test



# Format code

mix format



# Check if code is formatted

mix format --check-formatted



# Run Credo (code quality)

mix credo



# Run Credo in strict mode

mix credo --strict



# Generate documentation

mix docs



# Run Dialyzer (type checking)

mix dialyzer

Pre-commit Hook

For automatic code quality checks before commits, you can run:

mix precommit

This will:

  1. Format your code
  2. Run all tests
  3. Run Credo in strict mode

CI Pipeline

The mix ci command is designed for continuous integration and will:

  1. Check that code is properly formatted (fails if not)
  2. Run all tests
  3. Run Credo in strict mode

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details.

Copyright (c) 2025 Edgar Gomes

References