View Source Membrane.VideoCompositor (Membrane VideoCompositor Plugin v0.8.0)
Membrane SDK for VideoCompositor, that makes advanced, real-time video composition possible.
Input streams
Inputs are simply linked as Membrane Pads, no additional requests are required.
Input registration happens automatically.
After registering and linking input stream VideoCompositor will notify parent with input_registered_msg/0
.
After receiving this message, input can be used in composition.
Output streams
Outputs have to registered before linking.
To register output parent sends register_output_msg/0
.
After registering output, VideoCompositor will notify parent with output_registered_msg/0
.
Using output in composition is only available after registration.
Once VideoCompositor starts producing output stream, it will notify parent with new_output_stream_msg/0
.
Linking outputs is only available after receiving that message.
Composition specification - Scene
To specify what VideoCompositor should render parent should send vc_request/0
.
Scene
is a top level specification of what VideoCompositor should render.
As an example, if two inputs with IDs "input_0"
and "input_1"
and
single output with ID "output"
are registered, sending such update_scene
request would result in receiving inputs merged in layout on output:
scene_update_request = %{
type: "update_scene",
nodes: [
%{
type: "built-in",
node_id: "layout",
transformation: "tiled_layout",
input_pads: ["input_0", "input_1"]
}
],
outputs: [
%{
output_id: "output",
input_pad: "layout"
}
]
}
{[notify_child: {:video_compositor, {:vc_request, scene_update_request}}]}
VideoCompositor will notify parent with vc_request_response/0
.
You can use renderers/nodes to process input streams into outputs. VideoCompositor has builtin renders for most common use cases, but you can also register your own shaders, images and websites to tune VideoCompositor for specific business requirements.
Pads unlinking
Before unlinking pads make remove them from used scene, otherwise VC will crash on pad unlinking. Inputs/outputs are unregistered automatically on pad unlinking.
API reference
You can find more detailed API reference here.
Only update_scene
and register_renderer
request are available (inputs
/outputs
registration, start
and init
is done by SDK).
General concepts
General concepts of scene are explained here.
Examples
Examples can be found in examples
directory of Membrane VideoCompositor Plugin.
Scene
API usage examples can be found in VideoCompositor repo.
Bin options
Passed via struct Membrane.VideoCompositor.t/0
framerate
Membrane.RawVideo.framerate_t()
Required
Framerate of VideoCompositor outputs.port_range
port_range()
Default value:
{6000, 10000}
Port range in which input and output streams would try to be registered. If all ports in range will be used, VideoCompositor will crash on input/output registration.init_web_renderer?
boolean()
Default value:
true
Enables web rendering for VideoCompositor. If set to false, attempts to register and use web renderers will fail.stream_fallback_timeout
Membrane.Time.t()
Default value:
10000000000
Timeout that defines when the VideoCompositor should switch to fallback on the input stream that stopped sending frames.start_composing_strategy
:on_init | :on_message
Default value:
:on_init
Specifies when VideoCompositor starts composing frames. In:on_message
strategy,:start_composing
message has to be sent to start composing.vc_server_config
:start_on_random_port | {:start_on_port, :inet.port_number()} | {:already_started, :inet.port_number()}
Default value:
:start_on_random_port
Defines how VideoCompositor bin should start-up VideoCompositor server.There are three available options:
- :start_on_random_port - VC server is automatically started on port randomly chosen from port_range
- :start_on_port - VC server is automatically started on specified port
- :already_started - VideoCompositor bin assumes, that VC server is already started, initialized and should be available at specified port. Useful for sharing VC server between multiple pipelines or running custom version of VC server.
Pads
:input
Accepted formats:
%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction: | :input |
Availability: | :on_request |
Pad options:
input_id
input_id()
Required
:output
Accepted formats:
%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
Direction: | :output |
Availability: | :on_request |
Pad options:
output_id
output_id()
Required
Summary
Types
Preset of VideoCompositor output video encoder. See FFmpeg docs to learn more.
Input stream id, used in scene after adding input stream.
Options for pad :input
Notification sent to parent after VideoCompositor receives the first frame from the input stream (registered on input pad link).
Notification sent to parent after VideoCompositor starts producing streams and in ready to link output pad.
Output stream id, used in scene after adding output stream.
Options for pad :output
Notification sent to parent after output registration.
Range in which VideoCompositor search available ports.
Notification sent to VideoCompositor to register output stream.
Elixir translated body of VideoCompositor requests.
Struct containing options for Membrane.VideoCompositor
Request send to VideoCompositor.
VideoCompositor request response.
Types
@type encoder_preset() ::
:ultrafast
| :superfast
| :veryfast
| :faster
| :fast
| :medium
| :slow
| :slower
| :veryslow
| :placebo
Preset of VideoCompositor output video encoder. See FFmpeg docs to learn more.
@type input_id() :: String.t()
Input stream id, used in scene after adding input stream.
@type input_pad_opts() :: [{:input_id, input_id()}]
Options for pad :input
@type input_registered_msg() :: {:input_registered, input_id(), Membrane.VideoCompositor.Context.t()}
Notification sent to parent after VideoCompositor receives the first frame from the input stream (registered on input pad link).
Input can be used in scene
only after registration.
@type new_output_stream_msg() :: {:new_output_stream, output_id(), Membrane.VideoCompositor.Context.t()}
Notification sent to parent after VideoCompositor starts producing streams and in ready to link output pad.
See "Output streams" section in doc for more information.
@type output_id() :: String.t()
Output stream id, used in scene after adding output stream.
@type output_pad_opts() :: [{:output_id, output_id()}]
Options for pad :output
@type output_registered_msg() :: {:output_registered, output_id(), Membrane.VideoCompositor.Context.t()}
Notification sent to parent after output registration.
Output can be used in scene
only after registration.
@type port_range() :: {lower_bound :: :inet.port_number(), upper_bound :: :inet.port_number()}
Range in which VideoCompositor search available ports.
This range should be at least a few times wider then expected sum of inputs and outputs.
@type register_output_msg() :: {:register_output, Membrane.VideoCompositor.OutputOptions.t()}
Notification sent to VideoCompositor to register output stream.
See "Output streams" section in the documentation for more information.
@type request_body() :: map()
Elixir translated body of VideoCompositor requests.
Elixir types are mapped into JSON types:
- map -> object
- atom -> string
This request body:
%{
type: "update_scene",
nodes: [
%{
type: "built-in",
node_id: "layout",
transformation: "tiled_layout",
input_pads: ["input_0", "input_1"]
}
],
outputs: [
%{
output_id: "output",
input_pad: "layout"
}
]
}
will translate into the following JSON:
{
"type": "update_scene",
"nodes": [
{
"type": "built-in",
"node_id": "layout",
"transformation": "tiled_layout",
"input_pads": ["input_0", "input_1"]
}
],
outputs: [
{
"output_id": "output",
"input_pad": "layout"
}
]
}
@type t() :: %Membrane.VideoCompositor{ framerate: Membrane.RawVideo.framerate_t(), init_web_renderer?: boolean(), port_range: port_range(), start_composing_strategy: :on_init | :on_message, stream_fallback_timeout: Membrane.Time.t(), vc_server_config: :start_on_random_port | {:start_on_port, :inet.port_number()} | {:already_started, :inet.port_number()} }
Struct containing options for Membrane.VideoCompositor
@type vc_request() :: {:vc_request, request_body()}
Request send to VideoCompositor.
User of SDK should only send update_scene
or register_renderer
requests.
API reference can be found here.
@type vc_request_response() :: {:vc_request_response, request_body(), Req.Response.t(), Membrane.VideoCompositor.Context.t()}
VideoCompositor request response.
Functions
@spec options() :: keyword()
Returns description of options available for this module