Beam Bots Logo

bb_camera_avf

CI License: Apache 2.0 Hex version badge Hexdocs badge REUSE status Ask DeepWiki

A Beam Bots camera driver for macOS — built-in FaceTime cameras, USB webcams, Continuity Camera — capturing JPEG through ffmpeg's AVFoundation input in a supervised process.

It exists mostly so that a robot developed on a Mac can see through a real camera before it is deployed to Linux, where bb_camera_v4l takes over behind the same interface.

It is a BB.Sensor and a BB.Camera, declared like any other sensor:

topology do
  link :head do
    sensor :eye, {BB.Camera.AVF, device: "FaceTime HD Camera", width: 1280, height: 720}
  end
end

Consumers go through bb_camera, which shares one capture between all of them. Two browser tabs previewing the same camera produce one ffmpeg, not two.

Frames never travel on the robot's pubsub. That carries capture lifecycle only — BeginCapture and EndCapture. Frame bytes go over bb_camera's bounded media interface.

Installation

def deps do
  [
    {:bb, "~> 0.33"},
    {:bb_camera, "~> 0.1"},
    {:bb_camera_avf, "~> 0.1"}
  ]
end

Working on this driver from a bb_workspace checkout instead? BB_VERSION=local resolves bb, bb_camera and bb_camera_ffmpeg from the sibling checkouts rather than from Hex.

Requirements

Camera access

macOS grants camera access to an app, not to a process. An iex started in Terminal, iTerm or Ghostty captures with that terminal's permission, and the first capture from a terminal that has never asked raises the system prompt.

A VM started by something with no user interface — a launchd job, a daemon, an agent runner — may never be asked. It gets no prompt and no frames, and nothing says why: ffmpeg opens the camera and then waits for a first frame that never comes. In that wait it ignores both q and SIGTERM.

So capture gives up after :start_timeout_ms (10 seconds) without a frame, stops ffmpeg — with SIGKILL after :stop_timeout_ms, since it ignores SIGTERM — and fails with {:no_frames, 10000} as the reason. If the VM stops first, ffmpeg goes with it: it runs under MuonTrap, which kills it when the VM's end of the pipe closes. The warning on the console points here. The timeout cannot tell missing permission from a camera that sends nothing, so the error names what happened rather than a cause.

To check or grant access, open System Settings > Privacy & Security > Camera and look for the app you start the VM from.

Smoke test on a Mac

This walks from nothing to a frame from a real camera. The first half needs no camera at all, which is deliberate: if the synthetic source works and the real one does not, the problem is the camera or its permission, and you have narrowed it down without guessing.

1. Install ffmpeg

brew install ffmpeg

2. Prove the pipeline works without a camera

From the package directory:

BB_VERSION=local mix deps.get
BB_VERSION=local iex -S mix

BB_VERSION=local is needed on both commands. It is read each time mix runs, not stored in the lockfile.

Paste this robot into the session. It captures one of ffmpeg's own synthetic sources: real ffmpeg, real JPEG encoder, real pipe, real framing — and no hardware.

defmodule SynthCam do
  use BB

  settings do
    name :synth_cam
  end

  topology do
    link :head do
      sensor :eye, {BB.Camera.AVF, lavfi: "testsrc2", width: 320, height: 240, frame_rate: 10}
    end
  end
end

{:ok, _pid} = SynthCam.start_link([])

{:ok, session} = BB.Camera.Media.open(SynthCam, :eye, format: :jpeg)
{:ok, frame} = BB.Camera.Media.next(session, 10_000)

{frame.width, frame.height, byte_size(frame.data)}
#=> {320, 240, 8620}

File.write!("/tmp/frame.jpg", frame.data)
BB.Camera.Media.close(session)
open /tmp/frame.jpg

If that failed, nothing below will work, and the problem is ffmpeg rather than your camera.

3. Find the camera

BB_VERSION=local mix bb_camera_avf.scan
Cameras:

  [0] HD Pro Webcam C920
  [1] FaceTime HD Camera

Use the name, not the number - numbers move when cameras are plugged in:

    sensor :eye, {BB.Camera.AVF, device: "HD Pro Webcam C920"}

It asks ffmpeg, so the names are spelled the way ffmpeg will match them, and it opens no camera, so it needs no camera access.

Use the name. The number is a position in this list and moves when a camera is plugged in. Numbers past the last camera are screens, so a stale index can capture a display. Two identical webcams share one name, and this driver cannot tell them apart.

To find the sizes and rates a camera supports, ask for one it does not:

ffmpeg -hide_banner -f avfoundation -framerate 1 -video_size 640x480 \
  -i "HD Pro Webcam C920" -frames:v 1 -f null -

ffmpeg refuses, and prints every mode the camera lists:

Selected framerate (1.000000) is not supported by the device.
Supported modes:
  1280x720@[30.000030 30.000030]fps
  1920x1080@[30.000030 30.000030]fps
  ...

A rate written as 30.000030 is asked for as 30: ffmpeg matches rates to within 0.01.

4. Capture from the real camera

Run this from a terminal app, so that macOS has someone to ask. The first time, it shows a permission prompt; answer it within the 10-second timeout, or open the session again afterwards.

defmodule MyCam do
  use BB

  settings do
    name :my_cam
  end

  topology do
    link :head do
      sensor :eye, {BB.Camera.AVF, device: "FaceTime HD Camera", width: 1280, height: 720, frame_rate: 30}
    end
  end
end

{:ok, _pid} = MyCam.start_link([])

BB.subscribe(MyCam, [:sensor, :head, :eye])

{:ok, session} = BB.Camera.Media.open(MyCam, :eye, format: :jpeg)
{:ok, frame} = BB.Camera.Media.next(session, 15_000)
{frame.width, frame.height}
File.write!("/tmp/camera.jpg", frame.data)

If next/2 returns CaptureFailed with {:no_frames, 10000}, see Camera access.

frame.width and frame.height are read out of the JPEG that was actually encoded. If ffmpeg could not configure the camera as asked, it says Configuration of video device failed, falling back to default and captures at the camera's default size instead; this is where you find out.

5. Shut it down

BB.Camera.Media.close(session)
flush()
pgrep -lx ffmpeg

Nothing. Capture stops when the last consumer leaves, and the camera's light goes out.

A preview in a browser

bb_liveview mounts a dashboard that previews any camera a robot declares, this one included, as MJPEG. Declare the camera in your robot and point the dashboard at it; there is nothing driver-specific to wire up.

Options

Option Default
:device — Camera name, "default", or index. Required unless :lavfi is given
:lavfi — An ffmpeg synthetic source such as "testsrc2", instead of a camera
:width, :height 640, 480 Must be a size the camera lists
:frame_rate 30 A hertz unit, a number, or {numerator, denominator}
:input_format :nv12 Pixel format to ask AVFoundation for
:quality 5 ffmpeg's -q:v, 1 (best) to 31
:ffmpeg $PATH Path to the ffmpeg executable
:start_timeout_ms 10000 Fail capture when no frame has arrived by then
:max_frame_bytes 4 MiB Largest frame accepted before the stream is resynchronised
:drop_queue_len 8 Shed frames while more reads than this are waiting
:stdio_window 256 KiB Bytes of output ffmpeg may have in flight unacknowledged
:stop_timeout_ms 2000 How long ffmpeg gets after SIGTERM before SIGKILL

How it holds together

MuonTrap ──> ffmpeg ──pipe──> Capture ──frames──> BB.Camera.Hub ──> consumers
                                                       │
                                                       └─ lifecycle ──> BB.PubSub

Limitations

Testing

BB_VERSION=local mix deps.get
BB_VERSION=local mix test                   # no camera, no ffmpeg needed
BB_VERSION=local mix test --include ffmpeg  # also runs ffmpeg for real
BB_VERSION=local mix check --no-retry

The default suite drives the driver through bb_camera_ffmpeg's stub backend, so every lifecycle case — a capture that fails, a stop racing a start, a parameter change mid-stream — is a function call rather than something to wait for. No test sleeps. Framing, flow control and process cleanup are tested in bb_camera_ffmpeg.

The :ffmpeg tagged tests run this driver's own argument vector through real ffmpeg against a synthetic source. They need ffmpeg, but no camera. The one also tagged :macos needs AVFoundation and is skipped elsewhere.

Licence

Apache-2.0. See LICENSES/.