Trace

CrucibleTrace

ElixirHex.pmDocumentationLicense

Structured causal reasoning chain logging for LLM code generation

CausalTrace enables transparency and debugging in LLM-based code generation by capturing the decision-making process. It logs causal reasoning chains with events, alternatives considered, confidence levels, and supporting rationale.

Features

Installation

Add causal_trace to your list of dependencies in mix.exs:

def deps do
  [
    {:crucible_trace, "~> 0.3.1"}
  ]
end

Or install from GitHub:

def deps do
  [
  ]
end

Quick Start

Creating Events Manually

# Create a new chain
chain = CrucibleTrace.new_chain("API Implementation")

# Create an event
event = CrucibleTrace.create_event(
  :hypothesis_formed,
  "Use Phoenix framework",
  "Well-established with great documentation and active community",
  alternatives: ["Plug alone", "Custom HTTP server"],
  confidence: 0.9
)

# Add event to chain
chain = CrucibleTrace.add_event(chain, event)

# View statistics
stats = CrucibleTrace.statistics(chain)
# => %{total_events: 1, avg_confidence: 0.9, ...}

Parsing LLM Output

llm_output = """
<event type="hypothesis_formed">
  <decision>Use GenServer for state management</decision>
  <alternatives>Agent, ETS table, Database</alternatives>
  <reasoning>GenServer provides good balance of simplicity and features</reasoning>
  <confidence>0.85</confidence>
  <code_section>StateManager</code_section>
</event>

<code>
defmodule StateManager do
  use GenServer
  # ... implementation
end
</code>
"""

# Parse into a chain
{:ok, chain} = CrucibleTrace.parse_llm_output(llm_output, "State Manager Implementation")

# Extract just the code
code = CrucibleTrace.extract_code(llm_output)

Building Prompts for LLMs

base_spec = """
Implement a caching layer for database queries with:
- TTL support for cache entries
- Cache invalidation on writes
- Thread-safe operations
"""

# Generate a prompt that instructs the LLM to emit causal trace events
prompt = CrucibleTrace.build_causal_prompt(base_spec)

# Send prompt to your LLM, it will include event tags in its response

Visualization

# Generate interactive HTML visualization
html = CrucibleTrace.visualize(chain, style: :light)

# Save to file
{:ok, path} = CrucibleTrace.save_visualization(chain, "trace.html")

# Or open directly in browser
{:ok, _path} = CrucibleTrace.open_visualization(chain)

Storage and Retrieval

# Save chain to disk
{:ok, path} = CrucibleTrace.save(chain)

# Load by ID
{:ok, loaded_chain} = CrucibleTrace.load(chain.id)

# List all chains
{:ok, chains} = CrucibleTrace.list_chains()

# Search with criteria
{:ok, results} = CrucibleTrace.search(
  name_contains: "API",
  min_events: 5,
  created_after: ~U[2024-01-01 00:00:00Z]
)

# Export to different formats
{:ok, markdown} = CrucibleTrace.export(chain, :markdown)
{:ok, csv} = CrucibleTrace.export(chain, :csv)

Analysis

# Find decision points with alternatives
decisions = CrucibleTrace.find_decision_points(chain)

# Find low confidence events
uncertain = CrucibleTrace.find_low_confidence(chain, 0.7)

# Filter by event type
hypotheses = CrucibleTrace.get_events_by_type(chain, :hypothesis_formed)

# Custom filtering
high_conf = CrucibleTrace.filter_events(chain, fn e ->
  e.confidence >= 0.9
end)

# Chain statistics
stats = CrucibleTrace.statistics(chain)
# => %{
#   total_events: 10,
#   event_type_counts: %{hypothesis_formed: 3, pattern_applied: 2, ...},
#   avg_confidence: 0.87,
#   duration_seconds: 45
# }

Chain Comparison (v0.2.0)

Compare two reasoning chains to analyze differences:

# Compare chains from different LLM runs
{:ok, diff} = CrucibleTrace.diff_chains(chain1, chain2)

# View summary
IO.puts(diff.summary)
# => "2 added, 1 removed, 3 modified"

# Check similarity
IO.puts("Similarity: #{diff.similarity_score * 100}%")

# Export diff reports
text_diff = CrucibleTrace.diff_to_text(diff)
html_diff = CrucibleTrace.diff_to_html(diff, chain1, chain2)

# Track confidence changes
diff.confidence_deltas
# => %{"event_id" => 0.15, ...}

Use cases:

Mermaid Diagram Export (v0.2.0)

Export chains as Mermaid diagrams for documentation:

# Export as flowchart
mermaid = CrucibleTrace.export_mermaid(chain, :flowchart,
  color_by_type: true,
  include_confidence: true
)

# Embed in markdown
File.write!("decisions.md", """
# Decision Log

#{mermaid}

""")

# Other formats
CrucibleTrace.export_mermaid(chain, :sequence)  # Sequence diagram
CrucibleTrace.export_mermaid(chain, :timeline)  # Timeline view
CrucibleTrace.export_mermaid(chain, :graph)     # Graph with relationships

# Via unified export API
{:ok, mermaid} = CrucibleTrace.export(chain, :mermaid_flowchart)

Generated Mermaid diagrams are compatible with:

Event Types

CausalTrace supports six event types:

Event Schema

Each event contains:

%CrucibleTrace.Event{
  id: "unique_event_id",
  timestamp: ~U[2024-01-15 10:30:00Z],
  type: :hypothesis_formed,
  decision: "What was decided",
  alternatives: ["Alternative 1", "Alternative 2"],
  reasoning: "Why this decision was made",
  confidence: 0.85,  # 0.0 to 1.0
  code_section: "ModuleName",  # optional
  spec_reference: "Section 3.2",  # optional
  metadata: %{}  # optional
}

LLM Integration

When using CausalTrace with LLMs, instruct them to emit events in this XML format:

<event type="hypothesis_formed">
  <decision>Your decision</decision>
  <alternatives>Alt1, Alt2, Alt3</alternatives>
  <reasoning>Your reasoning</reasoning>
  <confidence>0.85</confidence>
  <code_section>ModuleName</code_section>
  <spec_reference>Spec Section</spec_reference>
</event>

Use CrucibleTrace.build_causal_prompt/1 to automatically generate prompts with these instructions.

Architecture

CausalTrace is organized into six main modules:

Examples

The library includes comprehensive example files demonstrating various use cases:

Basic Usage (examples/basic_usage.exs)

Advanced Analysis (examples/advanced_analysis.exs)

LLM Integration (examples/llm_integration.exs)

Storage and Search (examples/storage_and_search.exs)

Chain Comparison (examples/chain_comparison.exs)

Mermaid Export (examples/mermaid_export.exs)

Run any example with:

mix run examples/basic_usage.exs
mix run examples/advanced_analysis.exs
mix run examples/llm_integration.exs
mix run examples/storage_and_search.exs
mix run examples/chain_comparison.exs
mix run examples/mermaid_export.exs

Run them all at once:

./examples/run_examples.sh

Example outputs (JSON/Markdown/HTML) are written to example_traces/ by default (gitignored). Override with EXAMPLES_OUTPUT_DIR=/your/path ./examples/run_examples.sh.

Testing

The library has comprehensive test coverage across all modules:

Total: 103+ tests with 100% pass rate

Run the test suite:

mix test

Run with coverage:

mix test --cover

Run with strict warnings:

mix test --warnings-as-errors

Configuration

CausalTrace can be configured in your config/config.exs:

config :causal_trace,
  storage_dir: "causal_traces",  # Default storage directory
  default_format: :json,          # Default storage format
  visualization_style: :light     # Default HTML theme (:light or :dark)

Use Cases

Debugging LLM Code Generation

Track why an LLM made specific implementation choices:

# Parse LLM output with reasoning
{:ok, chain} = CrucibleTrace.parse_llm_output(llm_response, "Feature Implementation")

# Find low confidence decisions that need review
uncertain = CrucibleTrace.find_low_confidence(chain, 0.7)

# Visualize to understand the reasoning flow
CrucibleTrace.open_visualization(chain)

Comparing Alternative Approaches

Analyze which alternatives were considered:

decisions = CrucibleTrace.find_decision_points(chain)

Enum.each(decisions, fn d ->
  IO.puts("Chose: #{d.decision}")
  IO.puts("Over: #{Enum.join(d.alternatives, ", ")}")
  IO.puts("Because: #{d.reasoning}\n")
end)

Building Training Data

Export reasoning chains for fine-tuning:

{:ok, chains} = CrucibleTrace.list_chains()

training_data =
  chains
  |> Enum.filter(&(&1.event_count > 5))
  |> Enum.map(fn metadata ->
    {:ok, chain} = CrucibleTrace.load(metadata.id)
    CrucibleTrace.export(chain, :json)
  end)

Auditing AI Decisions

Maintain transparent records of AI reasoning:

# Save all chains with metadata
CrucibleTrace.save(chain,
  metadata: %{
    model: "gpt-4",
    user: "john@example.com",
    project: "payment-system"
  }
)

# Search audit logs
{:ok, results} = CrucibleTrace.search(
  created_after: ~U[2024-01-01 00:00:00Z],
  name_contains: "payment"
)

Performance

Limitations

ML Training Integration (v0.3.0)

CrucibleTrace now provides first-class support for ML training workflows:

Training Events

# Start training
event = CrucibleTrace.training_started(
  "Begin ResNet training",
  "Transfer learning from ImageNet",
  model_name: "resnet50",
  experiment_id: "exp-001"
)

# Record epoch completion
event = CrucibleTrace.epoch_completed(5, %{
  train_loss: 0.234,
  val_loss: 0.289,
  accuracy: 0.876
})

# Record checkpoint
event = CrucibleTrace.checkpoint_saved(
  "/models/checkpoint_epoch_5.pt",
  metrics: %{val_accuracy: 0.876}
)

# Wrap training function with automatic events
{chain, result} = CrucibleTrace.trace_training(chain, fn ->
  train_model(data)
end)

Event Relationships

Events can now reference parent events and dependencies:

parent = CrucibleTrace.create_event(:training_started, "Start", "Reason")
child = CrucibleTrace.create_event(:epoch_completed, "Epoch 1", "Done",
  parent_id: parent.id,
  experiment_id: "exp-001"
)

# Query relationships
{:ok, children} = CrucibleTrace.get_children(chain, parent.id)
roots = CrucibleTrace.get_root_events(chain)
leaves = CrucibleTrace.get_leaf_events(chain)

# Validate no circular dependencies
{:ok, _} = CrucibleTrace.validate_relationships(chain)

Telemetry Integration

Automatically trace pipeline events:

# Attach handlers to capture crucible_framework events
CrucibleTrace.attach_telemetry()

# Events are automatically created for pipeline stage execution

# Detach when done
CrucibleTrace.detach_telemetry()

Advanced Querying

# Content search
events = CrucibleTrace.search_events(chain, "GenServer",
  type: [:hypothesis_formed, :pattern_applied],
  min_confidence: 0.8
)

# Regex search
events = CrucibleTrace.search_regex(chain, ~r/epoch \d+/i)

# Advanced boolean queries
events = CrucibleTrace.query_events(chain, %{
  or: [
    %{type: :training_started},
    %{confidence: {:gte, 0.9}}
  ],
  and: [
    %{stage_id: "training"}
  ]
})

# Aggregate by field
counts = CrucibleTrace.aggregate_by(chain, :type, &length/1)

Stage Tracing

Wrap pipeline stages with automatic tracing:

{chain, result} = CrucibleTrace.trace_stage(chain, "preprocessing", fn ->
  preprocess_data(data)
end, experiment_id: "exp-001")

Roadmap

Completed

Planned

Contributing

This is part of the Elixir AI Research project. Contributions welcome!

License

MIT License - see LICENSE file for details

Documentation

Full documentation can be generated with ExDoc:

mix docs

Then open doc/index.html in your browser.

Support

For questions or issues, please open an issue on the GitHub repository.