Bellwether — BEAM Adapter

Crash reporting for embedded Linux, from the runtime's side.

This captures supervised crashes, redacts them on the device, and hands them to the Bellwether Agent — a small native daemon that owns the spool and the uplink. The Adapter never talks to the network: it runs inside a process that may be in the middle of dying, so its job is to shape, fingerprint and hand over, then get out of the way.

def deps do
  [{:bellwether, "~> 0.1"}]
end

Wiring

One child. It starts the socket client and the Reporter and attaches the :logger handler, so there is one place to configure rather than four to keep in step.

children = [
  {Bellwether.Supervisor,
   socket: "/tmp/bellwether.sock",
   own_apps: [:my_app],
   backtrace_depth: 32}
]

No call sites are needed. The BEAM puts :crash_reason into Logger metadata for every crash a supervisor reports, so the handler catches the domain without your code mentioning it.

Options may equally live in application config, where a release can change them without a code change:

config :bellwether,
  socket: "/tmp/bellwether.sock",
  own_apps: [:my_app],
  backtrace_depth: 32,
  keep_messages_for: [RuntimeError],
  block: ["otp:MyApp.KnownNoisy*"]

The two settings that decide how good a report is

own_apps separates your code from the framework in a Fingerprint. Get it wrong and every crash groups on whichever OTP function happened to sit above yours.

backtrace_depth raises the VM's limit, which is eight frames. Eight is often short of the call that explains a fault, and it costs grouping too: a crash inside a dependency can carry none of your frames within eight, so unrelated faults collapse onto the framework frames they share. It is a global VM flag, which is why this sets it only when asked.

Redaction

Every payload that leaves is a Shape: code-derived parts intact, data-derived parts replaced by their type.

%Lock{id: 4, site: "acme-1", state: :unlocked}
#=> "%Lock{id: int, site: binary(6), state: :unlocked}"

Atoms, struct names and module.function/arity are written in source and survive. Binaries, integers, floats, pids and refs are data and become their type. On a fleet where process state holds site credentials, this is what makes a crash reporter deployable at all.

The exception to know about: a message-carrying exception reports almost nothing by default, because a message is where people interpolate an address or a serial.

%RuntimeError{message: binary(37)}      # by default
%KeyError{key: :missing, term: %{site: binary(6)}}   # structured, and readable

If you know your own messages are safe, name them with keep_messages_for. Only your application can judge that, which is why the default is the one that cannot leak.

Reporting something by hand

Supervised crashes arrive on their own. For anything else:

# A fault you caught rather than let crash.
Bellwether.capture_exception(e, __STACKTRACE__)

# Something the runtime has no opinion about.
Bellwether.capture_event(:coproc, :alarm, %{radio: :zwave, fault: :no_response},
  fingerprint: "radio:zwave:no_response")

Both Shape what you pass. Give anything recurring a :fingerprint — without one the server has nothing to group on, which is right for a one-off and wrong for everything else.

The Agent

The Adapter is half of it. The Agent is a static binary that explains each boot, harvests kernel panics and erl_crash.dump, holds the spool across reboots, and uploads. On Nerves:

mix bellwether.agent --endpoint https://... --key bwk_...

which installs it into rootfs_overlay/ with a matching config, so the socket path is stated once rather than in two files that must agree.

aarch64 is bundled with this package, which covers most Nerves boards. A Hex package cannot carry every target and two megabytes of the wrong architecture helps nobody, so other targets are pointed at with --from or built locally — mix help bellwether.agent lists where it looks.

Nothing is downloaded. A binary that arrived from the network with no provenance is not something to put in firmware.

What is in here

Apache-2.0.

Publishing

priv/bin/ is committed, so refresh it before a release or the package ships whichever Agent was built last:

mix bellwether.bundle
mix hex.publish

mix test fails if the binaries are missing or are not static aarch64. It cannot tell a one-commit-stale binary from a current one, which is why the first command is not optional.