AITrace logo

GitHub: AITraceLicense: MIT

AITrace

The unified observability layer for the AI Control Plane.

AITrace provides the unified observability layer for the AI Control Plane, transforming opaque, non-deterministic AI processes into fully interpretable and debuggable execution traces.

Its mission is to create an Elixir-native instrumentation library and a corresponding data model that captures the complete causal chain of an AI agent's reasoning process—from initial prompt to final output, including all thoughts, tool calls, and state changes—enabling a true "Execution Cinema" experience for developers and operators.

Standalone tracing may use the :aitrace, :exporters application config. Governed callers must pass explicit exporters to AITrace.export/2; ambient application env must not select trace export sinks for governed evidence.

Stack Position

AITrace is an evidence and observability library, not the authority layer. In the ranked stack it provides trace refs, spans, events, replay bundles, export receipts, and execution-cinema data that other owners can join to their own truth:

products, AppKit, Mezzanine, Citadel, Jido Integration, StackLab

  -> AITrace spans/events/export receipts

      -> Mezzanine owns durable audit truth

      -> Citadel owns authority truth

      -> StackLab owns assembled proof joins

This distinction matters. A trace can prove what was observed by an instrumented path. It does not by itself prove that the path was authorized, that a workflow reached durable terminal truth, or that a release claim is closed. Those claims need the owning authority, audit, and proof repos to link AITrace refs into their receipts.

Current Platform Role

The current package still supports the simple local tracing API shown below, but it now also carries stack-oriented evidence contracts:

Use ambient application config only for standalone tracing. Governed callers must pass explicit exporters and refs at the call site so export sinks are not silently selected by process configuration.

The current Extravaganza cutover returns complete route evidence through the product command path, including trace refs. Its headless receipts currently report trace_replay.status: not_emitted; that means the product proof has receipt and route evidence, but has not exported a replay bundle through AITrace for that command. When a product claims replay proof, Mezzanine or the owning product path must emit AITrace events/exports that AITrace can replay.

The Synapse governed-effect lift adds AITrace.GovernedEffectEvidence, a bounded evidence helper for effect refs, authority refs, dispatch refs, receipt refs, trace refs, lifecycle entries, and metadata hashes. It supports the staged-live diagnostic proof by making provider facts data in the receipt path, not control-flow branches in product code. AITrace remains evidence/replay infrastructure; authority and durable lifecycle truth remain owned by Citadel and Mezzanine.

The cross-stack proof command is:

cd /home/home/p/g/n/stack_lab

MIX_ENV=test mix stack_lab.synapse.staged_live.v1 --json

See Generalized Stack Boundary for the current repo boundary and cutover-proof posture.

Maintainers should also read Code Smell Remediation before changing context propagation, export profiles, collector ownership, file export, replay reduction, or runtime identity.

Trace Diagrams

flowchart TD

  Operation["Instrumented<br/>operation"] --> Trace["Trace"]

  Trace --> Span["Nested<br/>spans"]

  Span --> Event["Point<br/>events"]

  Event --> Attr["Bounded<br/>attributes"]

  Attr --> Export["Explicit<br/>exporter"]

  Export --> Receipt["Export<br/>receipt"]

  Receipt --> Owner["Evidence<br/>owner"]



flowchart LR

  Prompt["Prompt<br/>evidence"] --> Bounds["Trace<br/>bounds"]

  Guard["Guard<br/>evidence"] --> Bounds

  Replay["Replay<br/>evidence"] --> Bounds

  Eval["Eval<br/>evidence"] --> Bounds

  Cost["Cost<br/>evidence"] --> Bounds

  Provider["Provider<br/>identity"] --> Bounds

  Bounds --> Redaction["Redaction<br/>refs"]

  Redaction --> StackProof["StackLab<br/>proof joins"]

Developer Flow Diagrams

flowchart TD

  API["Trace<br/>API"] --> Context["Trace<br/>context"]

  Context --> Collector["Collector"]

  Collector --> Spans["Spans"]

  Collector --> Events["Events"]

  Spans --> Bounds["Export<br/>bounds"]

  Events --> Bounds

  Bounds --> Exporter["Explicit<br/>exporter"]

  Exporter --> Receipt["Export<br/>receipt"]



flowchart LR

  TraceRef["Trace<br/>refs"] --> Bundle["Replay<br/>bundle"]

  Bundle --> Engine["Replay<br/>engine"]

  Engine --> Redaction["Redaction<br/>manifest"]

  Engine --> Proof["Proof<br/>join"]

  Proof --> StackLab["StackLab"]

  Proof --> Authority["Authority<br/>owner"]

  Proof --> Audit["Audit<br/>owner"]

The Problem: Why Traditional Observability Fails

Debugging a simple web request is a solved problem. We have structured logs, metrics, and distributed tracing (like OpenTelemetry) that show the path of a request through a series of stateless services.

Debugging an AI agent is fundamentally different. It is like performing forensic analysis on a dream. The challenges are unique:

Logger.info/1 is inadequate. Traditional APM tools provide a high-level view but lack the granular, AI-specific context needed to answer the most important question: "Why did the agent do that?"

Core Concepts & Data Model

AITrace is built on a few simple but powerful concepts, heavily inspired by OpenTelemetry but adapted for AI workflows.

Installation

Add aitrace to your mix.exs dependencies:

def deps do

  [

    {:aitrace, "~> 0.2.0"}

  ]

end

Quick Start

defmodule MyApp.Agent do

  require AITrace  # Required to use the macros



  def handle_user_message(message, state) do

    # 1. Start a new trace for the entire transaction

    AITrace.trace "agent.handle_message" do

      # 2. Add point-in-time events with rich metadata

      AITrace.add_event("request_received", %{message_length: String.length(message)})



      # 3. Wrap discrete, timed operations in spans

      response = AITrace.span "reasoning_loop" do

        # Add attributes to the current span

        AITrace.with_attributes(%{model: "gpt-4", temperature: 0.7})



        # Perform reasoning

        think_about(message)

      end



      AITrace.add_event("reasoning_complete", %{token_usage: response.tokens})



      {:reply, response.answer, update_state(state)}

    end

  end

end

Core API

Starting a Trace

AITrace.trace "operation_name" do

  # Your code here - context is stored in process dictionary

end

Creating Spans

AITrace.span "span_name" do

  # Timed operation - duration is automatically measured

end

Adding Events

AITrace.add_event("event_name", %{key: "value"})

AITrace.add_event("simple_event")  # No attributes

Adding Attributes

AITrace.with_attributes(%{user_id: 42, region: "us-west"})

Accessing Context

ctx = AITrace.get_current_context()

IO.inspect(ctx.trace_id)

IO.inspect(ctx.span_id)

Direct Trace Export

trace =

  %AITrace.Trace{trace_id: "trace_123", created_at: 1_712_345_678_000_000, spans: [], metadata: %{}}



AITrace.export(trace)

This one-shot path is intended for integrations that already have a completed AITrace.Trace value and want to run it through the configured exporters without using the collector-backed macros.

Configuration

Configure exporters in your application config:

# config/config.exs

config :aitrace,

  exporters: [

    {AITrace.Exporter.Console, verbose: true, color: true},

    {AITrace.Exporter.File, directory: "./traces"}

  ]

Available Exporters

Creating Custom Exporters

Implement the AITrace.Exporter behavior:

defmodule MyApp.CustomExporter do

  @behaviour AITrace.Exporter



  @impl true

  def init(opts), do: {:ok, opts}



  @impl true

  def export(trace, state) do

    # Send trace to your backend

    IO.inspect(trace)

    {:ok, state}

  end



  @impl true

  def shutdown(_state), do: :ok

end

Examples

See examples/basic_usage.exs for a complete working example:

mix run examples/basic_usage.exs

Output:

Trace: b37b73325dbd626481e0ff3e89de02c8

▸ reasoning (10.84ms) ✓

  Attributes: %{model: "gpt-4", temperature: 0.7}

    • reasoning_complete

      %{thought_count: 3}

▸ tool_execution (5.95ms) ✓

  Attributes: %{tool: "web_search"}

▸ response_generation (8.98ms) ✓

  Attributes: %{tokens: 150}

Architecture

Data Model

Runtime

Future Integrations

AITrace is designed to integrate with other AI infrastructure:

Development Status

✅ Implemented (v0.2.0)

🚧 Planned

Testing

# Run all tests

mix test



# Run with coverage

mix test --cover



# Run example

mix run examples/basic_usage.exs

License

MIT - See LICENSE for details.

Contributing

AITrace is part of the AI Control Plane ecosystem. Contributions welcome!

Persistence Documentation

See docs/persistence.md for tiers, defaults, adapters, unsupported selections, config examples, restart claims, durability claims, debug sidecar behavior, redaction guarantees, migration or preflight behavior, and no-bypass scope when applicable.

Chassis Spans

Chassis emits bounded spans for deployment, provisioning, mesh, health, rollback, evolution, model materialization, hardware admission, and tensor reload events. Attributes must pass AITrace.ExportBounds.profile/0 and carry refs, digests, summaries, outcomes, and bounded counts instead of raw payloads.

Baseline Chassis span names include deployment accepted, adapter selected, provisioning started/completed, mesh joined, health checked, receipt emitted, and rollback triggered. Evolution and model spans are listed below.

Chassis Evolution / Model Event Names

Chassis Evolution span names include:

Model, hardware, and tensor span names include:

Redaction Posture For Chassis Events

Chassis events must not carry raw credentials, raw private transcript bodies, raw prompt payloads, raw diffs, raw provider payloads, raw model weight bytes, mutable filesystem state as authority, or unsafe atom values. Use refs, bounded summaries, digest summaries, receipt refs, trace refs, and explicit redaction posture fields. The detailed Chassis attribute filter rules are in ../j/jido_brainstorm/nshkrdotcom/docs/20260529/chassis_impl/0532_chassis_evolution_aitrace_and_observability.md.