LokiLoggerModern

LokiLoggerModern is an Elixir :logger handler that ships your application logs to Grafana Loki.

Attribution: this project is based on LokiLogger by Ward Bekker (and includes improvements from the phanmn/LokiLogger fork). It was reworked to use Elixir's modern :logger handler API and is published under a new name, loki_logger_modern. Many thanks to the original authors and contributors.

Features

Implements the Erlang/Elixir :logger handler API (Elixir 1.18+), no legacy :gen_event backend
Attaches itself automatically when the application starts
Elixir Logger.Formatter formatting and metadata support
Snappy-compressed protobuf payload sent to the Loki push API
Buffered, asynchronous delivery (log calls never block on HTTP)
Periodic flushing of the buffer (every 5 seconds by default)
Pending logs are delivered when the application stops
Optional retry of failed pushes (network errors)
Pooled keep-alive connections (Finch)
HTTPS with certificate verification by default; custom CA or mTLS via ssl_options
X-Scope-OrgID header for Loki multi-tenancy
HTTP basic authentication

Installation

Add loki_logger_modern to your list of dependencies in mix.exs:

def deps do
  [
    {:loki_logger_modern, "~> 1.0"}
  ]
end

Requires Elixir 1.18 or later.

Configuration

LokiLoggerModern is configured through its application environment, e.g. in config/config.exs or config/runtime.exs:

import Config

config :loki_logger_modern,
  level: :info,
  format: "$time $metadata[$level] $message\n",
  metadata: [:request_id, :module],
  max_buffer: 300,
  flush_interval: 5_000,
  loki_labels: %{application: "my_app", elixir_node: node()},
  loki_host: "http://localhost:3100"

Options

Option	Default	Description
`enabled`	`true`	Set to `false` to not start anything nor attach the handler, e.g. in `config/test.exs`.
`loki_host`	`"http://localhost:3100"`	Base URL of the Loki server.
`loki_path`	`"/loki/api/v1/push"`	Path of the Loki push API.
`loki_labels`	`%{application: "loki_logger_modern"}`	Labels attached to the log stream (used to select it in e.g. Grafana).
`loki_scope_org_id`	`"fake"`	Tenant ID sent in the `X-Scope-OrgID` header (Loki multi-tenancy).
`basic_auth_user` / `basic_auth_password`	`nil`	Enables HTTP basic authentication when both are set.
`ssl_options`	`[]`	Extra `:ssl` client options for `https` hosts, e.g. `[cacertfile: "/path/to/ca.pem"]` for a private CA, or `certfile`/`keyfile` for mTLS. By default the server certificate is verified against the OS CA store (connections are made by Finch/Mint); on systems without one, add `:castore` to your dependencies.
`level`	`:info`	Minimum level of messages sent to Loki (`:debug`, `:info`, `:notice`, `:warning`, `:error`, ...).
`format`	`"$time $metadata[$level] $message\n"`	`Logger.Formatter` format string.
`metadata`	`:all`	Metadata keys included by `$metadata`, or `:all`.
`max_buffer`	`32`	Number of log entries buffered before they are pushed to Loki.
`flush_interval`	`5_000`	The buffer is also pushed every `flush_interval` milliseconds, so logs aren't held back in quiet periods. `0` disables periodic flushing.
`retry_count`	`0`	How many times a failed push is retried. `0` disables retries, a negative value retries forever.
`retry_delay`	`5_000`	Delay between retries in milliseconds.
`shutdown_timeout`	`5_000`	How long (ms) pending logs may take to be delivered when the application stops.

Note: pushes are sent one at a time; batches that fill up while a push is in flight (or being retried) are queued in memory. With a finite retry_count every push eventually completes or gives up, so the queue drains, but with retry_count < 0 (retry forever) the queue is unbounded: if Loki stays unreachable, memory usage grows for as long as the application keeps logging.

When the application stops, buffered entries are flushed, the in-flight push is given time to finish and queued batches are sent (without retries), all within shutdown_timeout milliseconds; anything still pending after that is dropped.

Delivery errors

Failed pushes (Loki unreachable, non-2xx responses, retries) are reported through Logger under the [:loki_logger_modern] domain, so they show up in your other handlers (e.g. the console) but are never sent to Loki themselves. To silence them, add a domain filter to the handler in question, e.g. for the default console handler:

:logger.add_handler_filter(
  :default,
  :no_loki_logger_modern,
  {&:logger_filters.domain/2, {:stop, :sub, [:elixir, :loki_logger_modern]}}
)

Runtime configuration

level, format and metadata can be changed at runtime:

:logger.update_handler_config(:loki_logger_modern, :level, :debug)
:logger.update_handler_config(:loki_logger_modern, :config, %{format: "$message\n", metadata: :all})

All other options are read once, when the application starts.

Development

Tests

mix test

The tests don't need a running Loki: they start a small fake Loki push endpoint (test/support/fake_loki.ex) and assert on the decoded protobuf payloads it receives.

Protobuf regeneration

Only needed when updating the Loki protobuf definitions. Requires protoc and the Elixir plugin (mix escript.install hex protobuf, with ~/.mix/escripts on your PATH). The plugin writes the file into a directory per package, so generate it elsewhere and move it into place:

mkdir -p /tmp/logproto
protoc --proto_path=lib/logproto --elixir_out=/tmp/logproto \
  --elixir_opt=package_prefix=loki_logger_modern lib/logproto/loki.proto
mv /tmp/logproto/loki_logger_modern/logproto/loki.pb.ex lib/logproto/loki.pb.ex

Publishing

Pushing a tag matching v*.*.* (e.g. v1.0.0) publishes the package to Hex via GitHub Actions. The HEX_API_KEY repository secret must be set.

License

The source code is released under the Apache v2.0 License. Check LICENSE for more information.