Membrane VideoMixer Plugin

Mixes multiple video inputs to a single output using ffmpeg filters. Allows dynamic input quality switches and source addition-removal while running.

This element is used in production.

Behavior and Policy

Core Mixing Policy

The mixer implements a strict synchronization policy: output frames are produced only when all required input pads have frames ready. This ensures deterministic, gap-free mixing without fallback frames.

Key Behaviors

Primary-driven output: The mixer produces output based on the primary input timing and PTS values.
Roles are required for dynamic pads: Each dynamic input must declare a unique :role atom.
Layouts driven by layout_builder: The builder function receives frame specs for all pads and returns either:
- {:layout, layout_name} - Use a predefined layout (:single_fit, :hstack, :vstack, :xstack, :primary_sidebar)
- {:raw, filter_graph} - Use a custom FFmpeg filter graph
Layout changes: When pads are added/removed, the mixer clears its filter graph cache. The layout is rebuilt lazily on the next mix operation.
Stream format requirements: All inputs must provide the same fixed framerate.
Pad activation: Pads become "active" when they send their first frame (not on stream_format). The layout is recalculated when the active pad set changes.

Synchronization and Draining

Mix only when ready: The mixer waits until all required pads (determined by the current layout) have frames queued before producing output.
No fallback frames: Unlike previous versions, the mixer does not generate black frames or reuse cached frames when inputs are missing.
Automatic draining: Non-required pad frames are automatically drained to prevent memory overflow while waiting for required pads.
Backpressure handling: When a required input stops producing frames, the mixer stops outputting until that input resumes.

Spec Changes

Spec tracking: Frame specifications are pushed to per-pad queues on stream_format and popped during mixing.
Layout builder access: The layout_builder function receives up-to-date specs for all pads that have sent stream formats.
Mixer rebuilding: The FFmpeg filter graph is rebuilt when frame specs change (resolution, pixel format, etc.).

Pad Options

Primary Pad (`:primary`)

role (atom, default: :primary) - The role identifier for this input
fit_mode (:crop | :fit, default: :crop) - How to fit the input into its layout region

Dynamic Input Pads (`:input`)

role (atom, required) - Unique role identifier for this input
fit_mode (:crop | :fit, default: :crop) - How to fit the input into its layout region

Removed Options

The following options were removed in favor of the simpler synchronization policy:

~~delay~~ - Primary delay has been removed. Mixing starts immediately when all required inputs have frames.
~~extra_queue_size~~ - Queue size limits have been removed. Non-required frames are drained automatically.

Benefits of the Current Policy

The strict synchronization policy provides several advantages:

Predictable behavior: Output is produced only when all required inputs are ready, eliminating timing-dependent edge cases.
No visual artifacts: No black frames or frozen frames from caching, ensuring clean output.
Simpler implementation: ~150 lines of complexity removed (fallback frames, black frame generation, last frame caching, delay logic).
Memory bounded: Automatic draining prevents unbounded queue growth.
Clear backpressure: When an input stops, mixing stops, making issues immediately visible.

Migration from Previous Versions

If you're upgrading from a version that supported delay or extra_queue_size:

Removed: Primary Delay

Old behavior: The :primary pad accepted a :delay option to buffer frames before mixing.

New behavior: Mixing starts immediately when all required pads have frames. If you need startup delay, implement it upstream of the mixer.

Removed: Black Frames and Frame Caching

Old behavior: When an input hadn't sent its first frame, a black frame was used. When an input paused, its last frame was reused.

New behavior: The mixer waits for all required inputs to have frames. No output is produced until all required pads are ready. If you need fallback behavior, handle it in your layout_builder by conditionally excluding pads from the layout.

Removed: Extra Queue Size

Old behavior: The :input pads accepted an :extra_queue_size option to limit buffering when the primary stalled.

New behavior: Non-required frames are automatically drained. No configuration needed.

Installation

def deps do
  [
    {:membrane_video_mixer_plugin, "~> 1.0.0"}
  ]
end