Membrane VideoInterop

The problem

Membrane is a multimedia framework. Its official elements exchange CPU-owned binary payloads in %Membrane.Buffer{} structs. Converting a GPU-bound frame into a CPU-owned binary copies the frame out of GPU storage and breaks a zero-copy pipeline.

A zero-copy GPU pipeline passes DMA-BUF descriptors between its elements. Each frame also carries synchronization state and an ownership contract so that its DMA-BUF remains valid until the consumer finishes using it. Rust elements produce and consume the GPU frames while the Membrane pipeline runs in Elixir.

VideoInterop defines the frame, DMA-BUF, synchronization, lease, and ownership primitives for that Rust and Elixir boundary. VideoInterop is framework-neutral and does not define Membrane elements.

The solution

Membrane VideoInterop builds on VideoInterop and exposes %VideoInterop.Frame{} through Membrane source and sink elements. Together, the two libraries provide the transport and ownership model for zero-copy GPU Membrane pipelines defined in Elixir with elements written in Rust.

The transport preserves the frame's storage representation:

Membrane.VideoInterop.Source accepts tagged frame messages, follows downstream demand, and retains at most one pending frame.
Membrane.VideoInterop.Sink hands each frame to a configured callback with an explicit consumption contract.
Membrane.VideoInterop.RawVideo performs explicit conversion between compatible CPU-owned RGB or RGBA frames and raw-video buffers.

The transport elements place the frame directly in %Membrane.Buffer{payload: frame}. DMA-BUF descriptors, synchronization metadata, and leases remain attached to the frame.

Installation

def deps do
  [
    {:membrane_video_interop, "~> 0.1.0"}
  ]
end

Version 0.1 supports Elixir 1.17/OTP 27 and later.

Transport frames

Add the source and sink to a pipeline:

child(:frames, %Membrane.VideoInterop.Source{
  notify: producer_pid,
  message_tag: :video_interop_frame
})
|> child(:display, %Membrane.VideoInterop.Sink{
  submit: {MyFrameConsumer, :submit, []},
  target: :camera
})

:video_interop_frame is the default message tag. When playback starts, the source sends {:video_interop_source_ready, source_pid} to producer_pid. The producer then transfers a frame to the source:

send(source_pid, {:video_interop_frame, frame})

The sink invokes the configured callback as MyFrameConsumer.submit(frame, :camera). Extra values in the MFA tuple are appended after the target.

Invalid source frames and sink failures are reported to the pipeline parent as {:video_interop_source_error, reason} and {:video_interop_sink_error, reason} respectively.

Ownership rules

Sending a matching {message_tag, frame} transfers ownership to the source. The producer must not release or reuse that frame afterward.
Messages with another tag are ignored and do not transfer ownership.
The source releases invalid frames, replaced pending frames, and pending frames during shutdown.
Once the sink callback is entered, the callback must consume the frame on every normal return, including error returns.
The sink releases the frame when validation fails or the callback raises, exits, or throws. A callback must not transfer ownership and then raise.

Owned binary frames have no lease. Borrowed DMA-BUF frames retain their exact VideoInterop.Lease throughout transport.

Convert RawVideo

For compatible owned frames, conversion is explicit:

alias Membrane.VideoInterop.RawVideo

{:ok, buffer, raw_format} = RawVideo.frame_to_buffer(frame)
{:ok, frame} = RawVideo.frame_from_buffer(buffer, raw_format)

Conversion supports aligned, progressive RGB and straight-alpha RGBA frames with square pixels and a full visible rectangle. Membrane.RawVideo does not carry VideoInterop colorimetry. DMA-BUF frames are not copied by these helpers.

See the changelog for release notes.

License

Apache-2.0. See the license.

Next Page → Changelog