View Source Membrane.VideoCompositor (Membrane VideoCompositor Plugin v0.8.0)

Membrane SDK for VideoCompositor, that makes advanced, real-time video composition possible.

Input streams

Inputs are simply linked as Membrane Pads, no additional requests are required. Input registration happens automatically. After registering and linking input stream VideoCompositor will notify parent with input_registered_msg/0. After receiving this message, input can be used in composition.

Output streams

Outputs have to registered before linking. To register output parent sends register_output_msg/0. After registering output, VideoCompositor will notify parent with output_registered_msg/0. Using output in composition is only available after registration. Once VideoCompositor starts producing output stream, it will notify parent with new_output_stream_msg/0. Linking outputs is only available after receiving that message.

Composition specification - Scene

To specify what VideoCompositor should render parent should send vc_request/0. Scene is a top level specification of what VideoCompositor should render.

As an example, if two inputs with IDs "input_0" and "input_1" and single output with ID "output" are registered, sending such update_scene request would result in receiving inputs merged in layout on output:

scene_update_request = %{
  type: "update_scene",
  nodes: [
    %{
      type: "built-in",
      node_id: "layout",
      transformation: "tiled_layout",
      input_pads: ["input_0", "input_1"]
    }
  ],
  outputs: [
    %{
      output_id: "output",
      input_pad: "layout"
    }
  ]
}

{[notify_child: {:video_compositor, {:vc_request, scene_update_request}}]}

VideoCompositor will notify parent with vc_request_response/0.

You can use renderers/nodes to process input streams into outputs. VideoCompositor has builtin renders for most common use cases, but you can also register your own shaders, images and websites to tune VideoCompositor for specific business requirements.

Pads unlinking

Before unlinking pads make remove them from used scene, otherwise VC will crash on pad unlinking. Inputs/outputs are unregistered automatically on pad unlinking.

API reference

You can find more detailed API reference here. Only update_scene and register_renderer request are available (inputs/outputs registration, start and init is done by SDK).

General concepts

General concepts of scene are explained here.

Examples

Examples can be found in examples directory of Membrane VideoCompositor Plugin. Scene API usage examples can be found in VideoCompositor repo.

Bin options

Passed via struct Membrane.VideoCompositor.t/0

  • framerate

    Membrane.RawVideo.framerate_t()

    Required
    Framerate of VideoCompositor outputs.

  • port_range

    port_range()

    Default value: {6000, 10000}
    Port range in which input and output streams would try to be registered. If all ports in range will be used, VideoCompositor will crash on input/output registration.

  • init_web_renderer?

    boolean()

    Default value: true
    Enables web rendering for VideoCompositor. If set to false, attempts to register and use web renderers will fail.

  • stream_fallback_timeout

    Membrane.Time.t()

    Default value: 10000000000
    Timeout that defines when the VideoCompositor should switch to fallback on the input stream that stopped sending frames.

  • start_composing_strategy

    :on_init | :on_message

    Default value: :on_init
    Specifies when VideoCompositor starts composing frames. In :on_message strategy, :start_composing message has to be sent to start composing.

  • vc_server_config

    :start_on_random_port
    | {:start_on_port, :inet.port_number()}
    | {:already_started, :inet.port_number()}

    Default value: :start_on_random_port
    Defines how VideoCompositor bin should start-up VideoCompositor server.

    There are three available options:

    • :start_on_random_port - VC server is automatically started on port randomly chosen from port_range
    • :start_on_port - VC server is automatically started on specified port
    • :already_started - VideoCompositor bin assumes, that VC server is already started, initialized and should be available at specified port. Useful for sharing VC server between multiple pipelines or running custom version of VC server.

Pads

:input

Accepted formats:

%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction::input
Availability::on_request

Pad options:

  • input_id

    input_id()

    Required

:output

Accepted formats:

%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction::output
Availability::on_request

Pad options:

  • output_id

    output_id()

    Required

Summary

Types

Preset of VideoCompositor output video encoder. See FFmpeg docs to learn more.

Input stream id, used in scene after adding input stream.

Options for pad :input

Notification sent to parent after VideoCompositor receives the first frame from the input stream (registered on input pad link).

Notification sent to parent after VideoCompositor starts producing streams and in ready to link output pad.

Output stream id, used in scene after adding output stream.

Options for pad :output

Notification sent to parent after output registration.

Range in which VideoCompositor search available ports.

Notification sent to VideoCompositor to register output stream.

Elixir translated body of VideoCompositor requests.

t()

Struct containing options for Membrane.VideoCompositor

Request send to VideoCompositor.

VideoCompositor request response.

Functions

Returns description of options available for this module

Types

@type encoder_preset() ::
  :ultrafast
  | :superfast
  | :veryfast
  | :faster
  | :fast
  | :medium
  | :slow
  | :slower
  | :veryslow
  | :placebo

Preset of VideoCompositor output video encoder. See FFmpeg docs to learn more.

@type input_id() :: String.t()

Input stream id, used in scene after adding input stream.

@type input_pad_opts() :: [{:input_id, input_id()}]

Options for pad :input

Link to this type

input_registered_msg()

View Source
@type input_registered_msg() ::
  {:input_registered, input_id(), Membrane.VideoCompositor.Context.t()}

Notification sent to parent after VideoCompositor receives the first frame from the input stream (registered on input pad link).

Input can be used in scene only after registration.

Link to this type

new_output_stream_msg()

View Source
@type new_output_stream_msg() ::
  {:new_output_stream, output_id(), Membrane.VideoCompositor.Context.t()}

Notification sent to parent after VideoCompositor starts producing streams and in ready to link output pad.

See "Output streams" section in doc for more information.

@type output_id() :: String.t()

Output stream id, used in scene after adding output stream.

@type output_pad_opts() :: [{:output_id, output_id()}]

Options for pad :output

Link to this type

output_registered_msg()

View Source
@type output_registered_msg() ::
  {:output_registered, output_id(), Membrane.VideoCompositor.Context.t()}

Notification sent to parent after output registration.

Output can be used in scene only after registration.

@type port_range() ::
  {lower_bound :: :inet.port_number(), upper_bound :: :inet.port_number()}

Range in which VideoCompositor search available ports.

This range should be at least a few times wider then expected sum of inputs and outputs.

@type register_output_msg() ::
  {:register_output, Membrane.VideoCompositor.OutputOptions.t()}

Notification sent to VideoCompositor to register output stream.

See "Output streams" section in the documentation for more information.

@type request_body() :: map()

Elixir translated body of VideoCompositor requests.

Elixir types are mapped into JSON types:

  • map -> object
  • atom -> string

This request body:

%{
  type: "update_scene",
  nodes: [
    %{
      type: "built-in",
      node_id: "layout",
      transformation: "tiled_layout",
      input_pads: ["input_0", "input_1"]
    }
  ],
  outputs: [
    %{
      output_id: "output",
      input_pad: "layout"
    }
  ]
}

will translate into the following JSON:

{
  "type": "update_scene",
  "nodes": [
    {
      "type": "built-in",
      "node_id": "layout",
      "transformation": "tiled_layout",
      "input_pads": ["input_0", "input_1"]
    }
  ],
  outputs: [
    {
      "output_id": "output",
      "input_pad": "layout"
    }
  ]
}
@type t() :: %Membrane.VideoCompositor{
  framerate: Membrane.RawVideo.framerate_t(),
  init_web_renderer?: boolean(),
  port_range: port_range(),
  start_composing_strategy: :on_init | :on_message,
  stream_fallback_timeout: Membrane.Time.t(),
  vc_server_config:
    :start_on_random_port
    | {:start_on_port, :inet.port_number()}
    | {:already_started, :inet.port_number()}
}

Struct containing options for Membrane.VideoCompositor

@type vc_request() :: {:vc_request, request_body()}

Request send to VideoCompositor.

User of SDK should only send update_scene or register_renderer requests. API reference can be found here.

@type vc_request_response() ::
  {:vc_request_response, request_body(), Req.Response.t(),
   Membrane.VideoCompositor.Context.t()}

VideoCompositor request response.

Functions

@spec options() :: keyword()

Returns description of options available for this module