OpenTelemetryDecorator

A function decorator for OpenTelemetry traces.

Installation

Add open_telemetry_decorator to your list of dependencies in mix.exs. We include the opentelemetry_api package, but you'll need to add opentelemetry yourself in order to report spans and traces.

def deps do
  [
    {:opentelemetry, "~> 1.2"},
    {:opentelemetry_exporter, "~> 1.4"},
    {:open_telemetry_decorator, "~> 1.4"}
  ]
end

Then follow the directions for the exporter of your choice to send traces to to zipkin, honeycomb, etc. https://github.com/open-telemetry/opentelemetry-erlang/tree/main/apps/opentelemetry_zipkin

Honeycomb Example

config/runtime.exs

api_key = Map.fetch!(System.get_env(), "HONEYCOMB_KEY")

config :opentelemetry, :processors,
  otel_batch_processor: %{
    exporter:
      {:opentelemetry_exporter,
       %{
         protocol: :grpc,
         headers: [
           {'x-honeycomb-team', api_key},
           {'x-honeycomb-dataset', 'YOUR_APP_NAME'}
         ],
         endpoints: [{:https, 'api.honeycomb.io', 443, []}]
       }}
  }

Usage

Add use OpenTelemetryDecorator to the module, and decorate any methods you want to trace with @decorate with_span("span name").

The with_span decorator will automatically wrap the decorated function in an opentelemetry span with the provided name.

defmodule MyApp.Worker do
  use OpenTelemetryDecorator

  @decorate with_span("worker.do_work")
  def do_work(arg1, arg2) do
    ...doing work
  end
end

Span Attributes

The with_span decorator allows you to specify an include option which gives you more flexibility with what you can include in the span attributes. Omitting the include option with with_span means no attributes will be added to the span by the decorator.

defmodule MyApp.Worker do
  use OpenTelemetryDecorator

  @decorate with_span("worker.do_work", include: [:arg1, :arg2])
  def do_work(arg1, arg2) do
    # ...doing work
  end
end

The Attributes module includes a helper for setting additional attributes outside of the include option. Attributes added in either a set call or in the include that are not primitive OTLP values will be converted to strings with Kernel.inspect/1.

defmodule MyApp.Worker do
  use OpenTelemetryDecorator
  alias OpenTelemetryDecorator.Attributes

  @decorate with_span("worker.do_work")
  def do_work(arg1, arg2) do
    Attributes.set(arg1: arg1, arg2: arg2)
    # ...doing work
    Attributes.set(:output, "something")
  end
end

The decorator uses a macro to insert code into your function at compile time to wrap the body in a new span and link it to the currently active span. In the example above, the do_work method would become something like this:

defmodule MyApp.Worker do
  require OpenTelemetry.Tracer, as: Tracer

  def do_work(arg1, arg2) do
    Tracer.with_span "my_app.worker.do_work" do
      # ...doing work
      Tracer.set_attributes(arg1: arg1, arg2: arg2)
    end
  end
end

Prefixing Span Attributes

Honeycomb suggests that you namespace custom fields, specifically putting manual instrumentation under app.

In order to do this, you'll configure the attr_prefix option in config/config.exs

config :open_telemetry_decorator, attr_prefix: "app."

Changing the join character for nested attributes

By default, nested attributes are joined with an underscore. However, when you have an object with underscores and a property with underscores, this can be hard to visually parse. For example, my_struct.other_struct.field, would be exported as my_struct_other_struct_field.

To override this, you'll configure the attr_joiner option in config/config.exs. The default value will likely change from _ to . in a future version.

config :open_telemetry_decorator, attr_joiner: "."

Thanks to @benregn for the examples and inspiration for these two options!

Additional Examples

You can provide span attributes by specifying a list of variable names as atoms.

This list can include...

Any variables (in the top level closure) available when the function exits. Note that variables declared as part of a with block are in a separate scope so NOT available for include attributes

defmodule MyApp.Math do
  use OpenTelemetryDecorator

  @decorate with_span("my_app.math.add", include: [:a, :b, :sum])
  def add(a, b) do
    sum = a + b
    {:ok, sum}
  end
end

The result of the function by including the atom :result:

defmodule MyApp.Math do
  use OpenTelemetryDecorator

  @decorate with_span("my_app.math.add", include: [:result])
  def add(a, b) do
    {:ok, a + b}
  end
end

Map/struct properties using nested lists of atoms:

defmodule MyApp.Worker do
  use OpenTelemetryDecorator

  @decorate with_span("my_app.worker.do_work", include: [[:arg1, :count], [:arg2, :count], :total])
  def do_work(arg1, arg2) do
    total = some_calculation(arg1.count, arg2.count)
    {:ok, total}
  end
end

defmodule MyApp.Worker do
  use OpenTelemetryDecorator

  @decorate with_span("my_app.worker.do_work", include: [[:calc, "sum"], [:calc, "product"]])
  def do_work(obj) do
    calc = %{"sum" => 10, "product" => 25}
    {:ok, calc}
  end
end

The map/struct properties of the result of the function:

defmodule MyApp.Math do
  use OpenTelemetryDecorator

  @decorate with_span("my_app.math.add", include: [[:result, :sum]])
  def add(a, b) do
    %{sum: a + b}
  end
end

Development

make check before you commit! If you'd prefer to do it manually:

• mix do deps.get, deps.unlock --unused, deps.clean --unused if you change dependencies
• mix compile --warnings-as-errors for a stricter compile
• mix coveralls.html to check for test coverage
• mix credo to suggest more idiomatic style for your code
• mix dialyzer to find problems typing might reveal… albeit slowly
• mix docs to generate documentation