View Source Membrane.RTC.Engine (Membrane RTC Engine v0.3.2)
RTC Engine implementation.
RTC Engine is an abstraction layer responsible for linking together different types of Endpoints
.
From the implementation point of view, RTC Engine is a Membrane.Pipeline
.
messages
Messages
The RTC Engine works by sending messages which notify user logic about important events like
"There is a new peer, would you like to to accept it?".
To receive RTC Engine messages you have to register your process so that RTC Engine will
know where to send them.
All messages RTC Engine can emit are described in Membrane.RTC.Engine.Message
docs.
registering-for-messages
Registering for messages
Registration can be done using register/2
e.g.
Engine.register(rtc_engine, self())
This will register your process to receive RTC Engine messages.
If your process implements GenServer
behavior then all messages can be handled
by GenServer.handle_info/2
, e.g.
@impl true
def handle_info(%Message.NewPeer{rtc_engine: rtc_engine, peer: peer}, state) do
Engine.accept_peer(rtc_engine, peer.id)
{:noreply, state}
end
You can register multiple processes to receive messages from an RTC Engine instance. In such a case each message will be sent to each registered process.
client-libraries
Client Libraries
RTC Engine allows creating Client Libraries that can send and receive media tracks from it.
The current version of RTC Engine ships with WebRTC Client Library which connects to the RTC Engine
via WebRTC standard.
Communication with Client Libraries is done using Media Events
.
Media Events are control messages which notify about e.g. new peer joining to the RTC Engine.
When Client Library receives Media Event it can invoke some callbacks.
In the case of WebRTC Client Library, these are e.g. onPeerJoined
or onTrackAdded
.
When RTC Engine receives Media Event it can emit some messages e.g. Membrane.RTC.Engine.Message.NewPeer.t/0
.
More about Media Events can be read in subsequent sections.
Below there is a figure showing the architecture of the RTC Engine working in conjunction with some Client Library.
+--------------------------------- media events -----------------------------+
| (signaling layer) |
| |
| |
+--------+ +---------+ +--------+ +---------+
| user | <- media -> | Client | | RTC | <- media -> | user |
| client | events | Library | <- media -> | Engine | events | backend |
| logic | <- callbacks - | | | | - messages -> | logic |
+--------+ +---------+ +--------+ +---------+
media-events
Media Events
Media Events are blackbox messages that carry data important for the RTC Engine and its Client Library, but not for the user. There are two types of Media Events:
- Internal Media Events - generic, protocol-agnostic Media Events sent by RTC Engine itself.
Example Internal Media Events are
peerJoined
,peerLeft
,tracksAdded
ortracksRemoved
. - Custom Media Events - they can be used to send custom data from Client Library to some Endpoint inside RTC Engine
and vice versa. In the case of WebRTC Client Library, these are
sdpOffer
,sdpAnswer
, oriceCandidate
.
An application is obligated to transport Media Events from an RTC Engine instance to its Client Library, and vice versa.
When the RTC Engine needs to send a Media Event to a specific client, registered processes will
receive Membrane.RTC.Engine.Message.MediaEvent.t/0
message with to
field indicating where this Media Event
should be sent to.
This can be either :broadcast
, when the event should be sent to all peers, or peer_id
when the messages should be sent to the specified peer. The event
is encoded in binary format,
so it is ready to send without modification.
Feeding an RTC Engine instance with Media Events from a Client Library can be done using receive_media_event/2
.
Assuming the user process is a GenServer, the Media Event can be received by GenServer.handle_info/2
and
conveyed to the RTC Engine in the following way:
@impl true
def handle_info({:media_event, from, event} = msg, state) do
Engine.receive_media_event(state.rtc_engine, from, event)
{:noreply, state}
end
What is important, Membrane RTC Engine doesn't impose usage of any specific transport layer for carrying Media Events through the network. You can e.g. use Phoenix and its channels. This can look like this:
@impl true
def handle_in("mediaEvent", %{"data" => event}, socket) do
Engine.receive_media_event(socket.assigns.room, socket.assigns.peer_id, event)
{:noreply, socket}
end
peers
Peers
Each peer represents some user that can possess some metadata. A Peer can be added in two ways:
- by sending proper Media Event from a Client Library
- using
add_peer/3
Adding a peer will cause RTC Engine to emit Media Event which will notify connected clients about new peer.
peer-id
Peer id
Peer ids must be assigned by application code. This is not done by the RTC Engine or its client library. Ids can be assigned when a peer initializes its signaling layer.
endpoints
Endpoints
Endpoints
are Membrane.Bin
s able to publish their own tracks and subscribe for tracks from other Endpoints.
One can think about Endpoint as an entity responsible for handling some specific task.
An Endpoint can be added and removed using add_endpoint/3
and remove_endpoint/2
respectively.
There are two types of Endpoints:
- Standalone Endpoints - they are in most cases spawned only once per RTC Engine instance and they are not associated with any peer.
- Peer Endpoints - they are associated with some peer. Associating Endpoint with Peer will cause RTC Engine to send some Media Events to the Enpoint's Client Library e.g. one which indicates which tracks belong to which peer.
Currently RTC Engine ships with the implementation of two Endpoints:
Membrane.RTC.Engine.Endpoint.WebRTC
which is responsible for establishing a connection with some WebRTC peer (mainly browser) and exchanging media with it. WebRTC Endpoint is a Peer Endpoint.Membrane.RTC.Engine.Endpoint.HLS
which is responsible for receiving media tracks from all other Endpoints and saving them to files by creating HLS playlists. HLS Endpoint is a Standalone Endpoint.
User can also implement custom Endpoints.
implementing-custom-rtc-engine-endpoint
Implementing custom RTC Engine Endpoint
Each RTC Engine Endpoint has to:
- implement
Membrane.Bin
behavior - specify input, output, or both input and output pads depending on what it is intended to do. For example, if Endpoint will not publish any tracks but only subscribe for tracks from other Endpoints it can specify only input pads. Pads should have the following form
def_input_pad :input,
demand_unit: :buffers,
caps: <caps>,
availability: :on_request
def_output_pad :output,
demand_unit: :buffers,
caps: <caps>,
availability: :on_request
Where caps
are Membrane.Caps.t/0
or :any
.
- publish for some tracks using actions
publish_action_t/0
and subscribe for some tracks using functionMembrane.RTC.Engine.subscribe/5
. The first will cause RTC Engine to send a message in form of{:new_tracks, tracks}
wheretracks
is a list ofMembrane.RTC.Engine.Track.t/0
to all other Endpoints. When an Endpoint receives such a message it can subscribe for new tracks by usingMembrane.RTC.Engine.subscribe/5
function. An Endpoint will be notified about track readiness it subscribed for inMembrane.Bin.handle_pad_added/3
callback. An example implementation ofhandle_pad_added
callback can look like this
@impl true
def handle_pad_added(Pad.ref(:input, _track_id) = pad, _ctx, state) do
links = [
link_bin_input(pad)
|> via_in(pad)
|> to(:my_element)
]
{{:ok, spec: %ParentSpec{links: links}}, state}
end
Where :my_element
is a custom Membrane element responsible for processing track.
Endpoint will be also notified when some tracks it subscribed for are removed with
{:removed_tracks, tracks}
message where tracks
is a list of Membrane.RTC.Engine.Track.t/0
.
Link to this section Summary
Types
Membrane action that will generate Custom Media Event.
Endpoint configuration options.
RTC Engine configuration options.
Membrane action that will cause RTC Engine to publish some message to all other endpoints.
Types of messages that can be published to other Endpoints.
Subscription options.
Membrane action that will inform RTC Engine about track readiness.
Functions
Allows peer for joining to the RTC Engine
Adds endpoint to the RTC Engine
Adds peer to the RTC Engine
Deny peer from joining to the RTC Engine.
The same as deny_peer/2
but allows for passing any data that will be returned to the client.
Changes playback state of pipeline to :playing
.
Changes playback state to :prepared
.
Sends Media Event to RTC Engine.
Registers process with pid who
for receiving messages from RTC Engine
Removes endpoint from the RTC Engine
Removes peer from RTC Engine.
Changes playback state to :stopped
.
Changes pipeline's playback state to :stopped
and terminates its process.
Subscribes endpoint for tracks.
Changes pipeline's playback state to :stopped
and terminates its process.
Unregisters process with pid who
from receiving messages from RTC Engine
Link to this section Types
@type custom_media_event_action_t() :: {:notify, {:custom_media_event, data :: binary()}}
Membrane action that will generate Custom Media Event.
Endpoint configuration options.
peer_id
- associate endpoint with exisiting peerendpoint_id
- assign endpoint id. If not provided it will be generated by RTC Engine. This option cannot be used together withpeer_id
. Endpoints associated with peers have the idpeer_id
.node
- node on which endpoint should be spawned. If not provided, current node is used.
RTC Engine configuration options.
id
is used by logger. If not provided it will be generated.trace_ctx
is used by OpenTelemetry. All traces from this engine will be attached to this context. Example function from which you can get Otel Context isget_current/0
fromOpenTelemetry.Ctx
.display_manager?
- set totrue
if you want to limit number of tracks sent fromMembrane.RTC.Engine.Endpoint.WebRTC
to a browser.
@type publish_action_t() :: {:notify, {:publish, publish_message_t()}}
Membrane action that will cause RTC Engine to publish some message to all other endpoints.
@type publish_message_t() :: {:new_tracks, [Membrane.RTC.Engine.Track.t()]} | {:removed_tracks, [Membrane.RTC.Engine.Track.t()]}
Types of messages that can be published to other Endpoints.
@type subscription_opts_t() :: [{:default_simulcast_encoding, String.t()}]
Subscription options.
default_simulcast_encoding
- initial encoding that endpoint making subscription wants to receive. This option has no effect for audio tracks and video tracks that are not simulcast.
@type track_ready_action_t() :: {:notify, {:track_ready, Membrane.RTC.Engine.Track.id(), Membrane.RTC.Engine.Track.encoding(), depayloading_filter :: Membrane.ParentSpec.child_spec_t()}}
Membrane action that will inform RTC Engine about track readiness.
Link to this section Functions
Allows peer for joining to the RTC Engine
@spec add_endpoint( pid :: pid(), endpoint :: Membrane.ParentSpec.child_spec_t(), opts :: endpoint_options_t() ) :: :ok | :error
Adds endpoint to the RTC Engine
Returns :error
when there are both peer_id
and endpoint_id
specified in opts
.
For more information refer to endpoint_options_t/0
.
@spec add_peer(pid :: pid(), peer :: Membrane.RTC.Engine.Peer.t()) :: :ok
Adds peer to the RTC Engine
Deny peer from joining to the RTC Engine.
The same as deny_peer/2
but allows for passing any data that will be returned to the client.
This can be used for passing reason of peer refusal.
@spec get_registry_name() :: atom()
@spec play(pid()) :: :ok
Changes playback state of pipeline to :playing
.
@spec prepare(pid()) :: :ok
Changes playback state to :prepared
.
@spec receive_media_event( rtc_engine :: pid(), media_event :: {:media_event, pid(), any()} ) :: :ok
Sends Media Event to RTC Engine.
Registers process with pid who
for receiving messages from RTC Engine
Removes endpoint from the RTC Engine
Removes peer from RTC Engine.
If reason is other than nil
, RTC Engine will inform client library about peer removal with passed reason.
@spec start(options :: options_t(), process_options :: GenServer.options()) :: GenServer.on_start()
@spec start_link(options :: options_t(), process_options :: GenServer.options()) :: GenServer.on_start()
@spec stop(pid()) :: :ok
Changes playback state to :stopped
.
Changes pipeline's playback state to :stopped
and terminates its process.
@spec subscribe( rtc_engine :: pid(), endpoint_id :: String.t(), track_id :: Membrane.RTC.Engine.Track.id(), format :: atom(), opts :: subscription_opts_t() ) :: :ok | {:error, :timeout | :invalid_track_id | :invalid_track_format | :invalid_default_simulcast_encoding}
Subscribes endpoint for tracks.
Endpoint will be notified about track readiness in Membrane.Bin.handle_pad_added/3
callback.
tracks
is a list in form of pairs {track_id, track_format}
, where track_id
is id of track this endpoint subscribes for
and track_format
is the format of track that this endpoint is willing to receive.
If track_format
is :raw
Endpoint will receive track in Membrane.RTC.Engine.Track.encoding/0
format.
Endpoint_id is a an id of endpoint, which want to subscribe on tracks.
Changes pipeline's playback state to :stopped
and terminates its process.
Unregisters process with pid who
from receiving messages from RTC Engine