Dala.Media.Gpu.Processor (dala v0.5.0)

Copy Markdown View Source

GPU-accelerated media processing via EXCubeCL.

Provides real-time video and image processing pipelines using GPU compute kernels. Designed for:

  • Camera frame processing (blur, beauty, filters)
  • Video effects (transitions, overlays, color grading)
  • Livestream effects (virtual background, AR)
  • Image preprocessing for ML inference
  • Video encoding/transcoding via ExCubecl.Transcode

EXCubeCL Compatibility

This module is compatible with EXCubeCL 0.3.0+ and 0.4.0+ APIs:

EXCubeCL 0.4.0 Media Modules

Architecture

Camera Frame → GPU Buffer → CubeCL Kernels → GPU Buffer → Display/Encoder

Example: Real-time blur filter

# Create processing context
{:ok, ctx} = Dala.Media.Gpu.Processor.start_pipeline(640, 480)

# Process frames in a loop
receive do
  {:camera_frame, rgba_data} ->
    {:ok, output} = Dala.Media.Gpu.Processor.process_frame(ctx, rgba_data, [
      {:blur, %{radius: 3, sigma: 1.5}},
      {:sharpen, %{amount: 0.3}}
    ])
    # Display or encode output
end

Example: Virtual background

{:ok, ctx} = Dala.Media.Gpu.Processor.start_pipeline(1280, 720)

{:ok, output} = Dala.Media.Gpu.Processor.process_frame(ctx, camera_frame, [
  {:segmentation, %{model: :deeplab}},
  {:background_replace, %{background: bg_image}},
  {:blend, %{alpha: 0.9}}
])

Example: Media source → filter → encode

{:ok, source} = Dala.Media.Gpu.Processor.open_source("video.mp4")
{:ok, streams} = Dala.Media.Gpu.Processor.streams(source)
{:ok, encoder} = Dala.Media.Gpu.Processor.start_encoder("output.mp4", %{codec: :h264})

# Read, filter, encode loop
{:ok, frame} = Dala.Media.Gpu.Processor.read_video_frame(source)
{:ok, filtered} = Dala.Media.Gpu.Processor.apply_filter(frame, :blur, %{radius: 2})
:ok = Dala.Media.Gpu.Processor.write_video_frame(encoder, filtered)

:ok = Dala.Media.Gpu.Processor.finish_encoder(encoder)

Available Filters

FilterDescriptionParams
:blurGaussian blur%{radius: 3, sigma: 1.5}
:sharpenUnsharp mask%{amount: 0.5}
:denoiseBilateral filter%{strength: 0.5}
:beautySkin smoothing%{strength: 0.3}
:grayscaleRGB to grayscale%{}
:sepiaSepia tone%{intensity: 0.8}
:vignetteVignette effect%{intensity: 0.5}
:lutColor LUT transform%{lut: lut_data}
:brightnessBrightness adjustment%{value: 0.1}
:contrastContrast adjustment%{value: 0.2}
:saturationSaturation adjustment%{value: 0.3}
:white_balanceWhite balance correction%{temperature: 6500}
:hdrHDR tone mapping%{exposure: 1.0}
:segmentationSemantic segmentation%{model: :deeplab}
:background_replaceBackground replacement%{background: buf}
:blendAlpha blending%{alpha: 0.5}

Summary

Types

context()
encoder()
filter_spec()
source()

Functions

apply_filter(frame, filter, params \\ %{})

Apply a named GPU-accelerated filter to a video frame.

apply_filter(rgba_data, width, height, filter, params \\ %{})

Apply a single filter to an RGBA binary.

blur(data, w, h, opts \\ [])

Blur filter.

brightness(data, w, h, value)

Brightness adjustment.

close_source(source)

Close a media source and release all associated resources.

contrast(data, w, h, value)

Contrast adjustment.

convert(frame, to_format, opts \\ %{})

Convert a video frame to a different pixel format via ExCubecl.Video.

convert_audio_channels(samples, target_layout, opts \\ %{})

Convert audio channel layout via ExCubecl.Audio.

crop(frame, opts)

Crop a video frame via ExCubecl.Video.

filter_chain(frame, filters, global_params \\ %{})

Apply a chain of GPU-accelerated filters to a video frame.

finish_encoder(encoder)

Finalize encoding and close the output file.

grayscale(data, w, h)

Grayscale filter.

mix(a, b, opts \\ %{})

Mix two video frames with equal weight via ExCubecl.Video.

mix_audio(a, b, opts \\ %{})

Mix two audio sample buffers via ExCubecl.Audio.

open_source(path)

Open a media source via ExCubecl.Media.

overlay(base, overlay, opts \\ %{})

Overlay one video frame onto another via ExCubecl.Video.

overlay_audio(base, overlay, opts \\ %{})

Overlay audio samples onto a base via ExCubecl.Audio.

process_frame(ctx, rgba_data, filters)

Process a frame through a chain of GPU filters.

process_frame_async(ctx, rgba_data, filters)

Process a frame asynchronously. Returns {:ok, cmd_id}.

read_audio_samples(source)

Read audio samples from a media source.

read_video_frame(source)

Read a video frame from a media source.

resample_audio(samples, target_rate, opts \\ %{})

Resample audio to a different sample rate via ExCubecl.Audio.

saturation(data, w, h, value)

Saturation adjustment.

scale(frame, opts)

Scale a video frame to new dimensions via ExCubecl.Video.

sharpen(data, w, h, opts \\ [])

Sharpen filter.

start_encoder(output_path, opts \\ %{})

Start a transcoder/encoder via ExCubecl.Transcode.

start_pipeline(width, height)

Start a new GPU processing pipeline for the given dimensions.

stop_pipeline(processor)

Stop a processing pipeline and free all GPU resources.

streams(source)

Get stream information from an opened media source.

transcode_file(input_path, output_path, opts \\ [])

Transcode a media file to another format via ExCubecl.Transcode.

write_audio_samples(encoder, samples)

Write audio samples to an encoder.

write_video_frame(encoder, frame)

Write a video frame to an encoder.

Types

context()

@type context() :: %Dala.Media.Gpu.Processor{
  height: non_neg_integer(),
  input_buf: Dala.Gpu.Compute.Buffer.t() | nil,
  output_buf: Dala.Gpu.Compute.Buffer.t() | nil,
  pipeline: reference() | nil,
  temp_buf: Dala.Gpu.Compute.Buffer.t() | nil,
  width: non_neg_integer()
}

encoder()

@type encoder() :: reference()

filter_spec()

@type filter_spec() :: {atom(), map()}

source()

@type source() :: reference()

Functions

apply_filter(frame, filter, params \\ %{})

@spec apply_filter(ExCubecl.VideoFrame.t(), atom(), map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Apply a named GPU-accelerated filter to a video frame.

Parameters

  • frame — %ExCubecl.VideoFrame{} struct
  • filter — filter atom (e.g. :blur, :sharpen, :denoise, :beauty)
  • params — map of filter-specific parameters

Returns {:ok, %ExCubecl.VideoFrame{}}.

apply_filter(rgba_data, width, height, filter, params \\ %{})

@spec apply_filter(binary(), non_neg_integer(), non_neg_integer(), atom(), map()) ::
  {:ok, binary()} | {:error, term()}

Apply a single filter to an RGBA binary.

blur(data, w, h, opts \\ [])

@spec blur(binary(), non_neg_integer(), non_neg_integer(), keyword()) ::
  {:ok, binary()} | {:error, term()}

Blur filter.

brightness(data, w, h, value)

@spec brightness(binary(), non_neg_integer(), non_neg_integer(), float()) ::
  {:ok, binary()} | {:error, term()}

Brightness adjustment.

close_source(source)

@spec close_source(source()) :: :ok | {:error, term()}

Close a media source and release all associated resources.

contrast(data, w, h, value)

@spec contrast(binary(), non_neg_integer(), non_neg_integer(), float()) ::
  {:ok, binary()} | {:error, term()}

Contrast adjustment.

convert(frame, to_format, opts \\ %{})

@spec convert(ExCubecl.VideoFrame.t(), atom(), map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Convert a video frame to a different pixel format via ExCubecl.Video.

Parameters

  • frame — %ExCubecl.VideoFrame{}
  • format — target format atom (e.g. :rgba, :bgra, :yuv420p, :nv12)
  • opts — additional options map

Returns {:ok, %ExCubecl.VideoFrame{}}.

convert_audio_channels(samples, target_layout, opts \\ %{})

@spec convert_audio_channels(ExCubecl.AudioSamples.t(), atom(), map()) ::
  {:ok, ExCubecl.AudioSamples.t()} | {:error, term()}

Convert audio channel layout via ExCubecl.Audio.

Returns {:ok, %ExCubecl.AudioSamples{}}.

crop(frame, opts)

@spec crop(ExCubecl.VideoFrame.t(), map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Crop a video frame via ExCubecl.Video.

Parameters

  • frame — %ExCubecl.VideoFrame{}
  • opts — options map:
    • :x — left edge
    • :y — top edge
    • :width — crop width
    • :height — crop height

Returns {:ok, %ExCubecl.VideoFrame{}}.

filter_chain(frame, filters, global_params \\ %{})

@spec filter_chain(ExCubecl.VideoFrame.t(), [filter_spec()], map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Apply a chain of GPU-accelerated filters to a video frame.

filters is a list of {filter_name, params} tuples, applied in order.

Returns {:ok, %ExCubecl.VideoFrame{}}.

finish_encoder(encoder)

@spec finish_encoder(encoder()) :: :ok | {:error, term()}

Finalize encoding and close the output file.

Returns :ok or {:error, term()}.

grayscale(data, w, h)

@spec grayscale(binary(), non_neg_integer(), non_neg_integer()) ::
  {:ok, binary()} | {:error, term()}

Grayscale filter.

mix(a, b, opts \\ %{})

@spec mix(ExCubecl.VideoFrame.t(), ExCubecl.VideoFrame.t(), map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Mix two video frames with equal weight via ExCubecl.Video.

Returns {:ok, %ExCubecl.VideoFrame{}}.

mix_audio(a, b, opts \\ %{})

@spec mix_audio(ExCubecl.AudioSamples.t(), ExCubecl.AudioSamples.t(), map()) ::
  {:ok, ExCubecl.AudioSamples.t()} | {:error, term()}

Mix two audio sample buffers via ExCubecl.Audio.

Returns {:ok, %ExCubecl.AudioSamples{}}.

open_source(path)

@spec open_source(String.t()) :: {:ok, source()} | {:error, term()}

Open a media source via ExCubecl.Media.

path can be a file path, URL, or device identifier.

Returns {:ok, source} where source is a reference to the opened media.

overlay(base, overlay, opts \\ %{})

@spec overlay(ExCubecl.VideoFrame.t(), ExCubecl.VideoFrame.t(), map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Overlay one video frame onto another via ExCubecl.Video.

Parameters

  • base — base %ExCubecl.VideoFrame{}
  • overlay — overlay %ExCubecl.VideoFrame{}
  • opts — options map:
    • :x — horizontal position (default: 0)
    • :y — vertical position (default: 0)
    • :alpha — blend alpha 0.0..1.0 (default: 1.0)

Returns {:ok, %ExCubecl.VideoFrame{}}.

overlay_audio(base, overlay, opts \\ %{})

@spec overlay_audio(ExCubecl.AudioSamples.t(), ExCubecl.AudioSamples.t(), map()) ::
  {:ok, ExCubecl.AudioSamples.t()} | {:error, term()}

Overlay audio samples onto a base via ExCubecl.Audio.

Returns {:ok, %ExCubecl.AudioSamples{}}.

process_frame(ctx, rgba_data, filters)

@spec process_frame(context(), binary(), [filter_spec()]) ::
  {:ok, binary()} | {:error, term()}

Process a frame through a chain of GPU filters.

rgba_data is a raw RGBA8888 binary of size width * height * 4. filters is a list of {filter_name, params} tuples.

Returns {:ok, binary()} with the processed RGBA8888 data.

process_frame_async(ctx, rgba_data, filters)

@spec process_frame_async(context(), binary(), [filter_spec()]) ::
  {:ok, non_neg_integer()} | {:error, term()}

Process a frame asynchronously. Returns {:ok, cmd_id}.

read_audio_samples(source)

@spec read_audio_samples(source()) ::
  {:ok, ExCubecl.AudioSamples.t()} | {:error, term()}

Read audio samples from a media source.

Returns {:ok, %ExCubecl.AudioSamples{}} or {:error, :eof} when done.

read_video_frame(source)

@spec read_video_frame(source()) :: {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Read a video frame from a media source.

Returns {:ok, %ExCubecl.VideoFrame{}} or {:error, :eof} when done.

resample_audio(samples, target_rate, opts \\ %{})

@spec resample_audio(ExCubecl.AudioSamples.t(), non_neg_integer(), map()) ::
  {:ok, ExCubecl.AudioSamples.t()} | {:error, term()}

Resample audio to a different sample rate via ExCubecl.Audio.

Returns {:ok, %ExCubecl.AudioSamples{}}.

saturation(data, w, h, value)

@spec saturation(binary(), non_neg_integer(), non_neg_integer(), float()) ::
  {:ok, binary()} | {:error, term()}

Saturation adjustment.

scale(frame, opts)

@spec scale(ExCubecl.VideoFrame.t(), map()) ::
  {:ok, ExCubecl.VideoFrame.t()} | {:error, term()}

Scale a video frame to new dimensions via ExCubecl.Video.

Parameters

  • frame — %ExCubecl.VideoFrame{}
  • opts — options map:
    • :width — target width
    • :height — target height

    • :interpolation — :nearest | :bilinear | :bicubic (default: :bilinear)

Returns {:ok, %ExCubecl.VideoFrame{}}.

sharpen(data, w, h, opts \\ [])

@spec sharpen(binary(), non_neg_integer(), non_neg_integer(), keyword()) ::
  {:ok, binary()} | {:error, term()}

Sharpen filter.

start_encoder(output_path, opts \\ %{})

@spec start_encoder(String.t(), map()) :: {:ok, encoder()} | {:error, term()}

Start a transcoder/encoder via ExCubecl.Transcode.

Parameters

  • output_path — destination file path
  • opts — encoder options map, e.g.:

    • :codec — :h264 | :h265 | :vp9 | :av1

    • :bitrate — target bitrate in bits/sec
    • :width / :height — output dimensions
    • :fps — output frame rate
    • :sample_rate — audio sample rate

    • :audio_codec — :aac | :opus | :mp3

Returns {:ok, encoder}.

start_pipeline(width, height)

@spec start_pipeline(non_neg_integer(), non_neg_integer()) ::
  {:ok, context()} | {:error, term()}

Start a new GPU processing pipeline for the given dimensions.

stop_pipeline(processor)

@spec stop_pipeline(context()) :: :ok

Stop a processing pipeline and free all GPU resources.

streams(source)

@spec streams(source()) :: {:ok, [map()]} | {:error, term()}

Get stream information from an opened media source.

Returns {:ok, streams} where streams is a list of stream info maps. Each map contains keys like :type (:video | :audio), :codec, :width, :height, :sample_rate, etc.

transcode_file(input_path, output_path, opts \\ [])

@spec transcode_file(String.t(), String.t(), keyword()) :: :ok | {:error, term()}

Transcode a media file to another format via ExCubecl.Transcode.

Convenience wrapper for file-to-file transcode.

Options

  • :video — keyword list with :codec, :bitrate, :fps, :width, :height
  • :audio — keyword list with :codec, :bitrate, :sample_rate

Example

Dala.Media.Gpu.Processor.transcode_file("input.mp4", "output.mkv",
  video: [codec: :h265, bitrate: "4M"],
  audio: [codec: :opus, bitrate: "128k"]
)

write_audio_samples(encoder, samples)

@spec write_audio_samples(encoder(), ExCubecl.AudioSamples.t()) ::
  :ok | {:error, term()}

Write audio samples to an encoder.

samples is an %ExCubecl.AudioSamples{} struct. encoder is a reference returned by start_encoder/2.

Returns :ok or {:error, term()}.

write_video_frame(encoder, frame)

@spec write_video_frame(encoder(), ExCubecl.VideoFrame.t()) :: :ok | {:error, term()}

Write a video frame to an encoder.

frame is an %ExCubecl.VideoFrame{} struct. encoder is a reference returned by start_encoder/2.

Returns :ok or {:error, term()}.