Element that wraps a Bumblebee.Audio.speech_to_text_whisper/4 serving, producing transcripts of the input audio.

The serving must be provided by the user. For details on the configuration of the serving, see the description of the serving option of this element.

Element options

Passed via struct Membrane.Whisper.TranscriberFilter.t/0

• serving

  Nx.Serving.t()

  Required
  The result of a call to Bumblebee.Audio.speech_to_text_whisper/4, with the following options set:

  • :stream: Must be true. Enables output streaming, e.g. it makes calls to Nx.Serving.run/2 return an Elixir Stream that will return outputs from Whisper.
  • :chunk_num_seconds: Must be set. Enables long-form transcription by splitting the input into chunks of the given length. This means that calls to the serving with Nx.Serving.run/2 will only accept enumerable input when :chunk_num_seconds is set.

  Example of creating a compatible serving:

    hf_repo = "openai/whisper-tiny"
  
    {:ok, whisper} = Bumblebee.load_model({:hf, hf_repo})
    {:ok, featurizer} = Bumblebee.load_featurizer({:hf, hf_repo})
    {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, hf_repo})
    {:ok, generation_config} = Bumblebee.load_generation_config({:hf, hf_repo})
  
    serving = Bumblebee.Audio.speech_to_text_whisper(
      whisper,
      featurizer,
      tokenizer,
      generation_config,
      stream: true,
      chunk_num_seconds: 10
    )

  One could also enable timestamp prediction by the model, though this will impact the result by producing transcripts of segments of varying length, e.g. a transcript of 7 seconds of audio followed by a transcript of 13 seconds of audio, etc.

    hf_repo = "openai/whisper-tiny"
  
    {:ok, whisper} = Bumblebee.load_model({:hf, hf_repo})
    {:ok, featurizer} = Bumblebee.load_featurizer({:hf, hf_repo})
    {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, hf_repo})
    {:ok, generation_config} = Bumblebee.load_generation_config({:hf, hf_repo})
  
    serving_with_timestamps = Bumblebee.Audio.speech_to_text_whisper(
      whisper,
      featurizer,
      tokenizer,
      generation_config,
      stream: true,
      chunk_num_seconds: 10,
      timestamps: :segments
    )

  For other useful options, see Bumblebee.Audio.speech_to_text_whisper/4.

Pads

:input

Accepted formats:

%RawAudio{sample_format: :f32le, channels: 1, sample_rate: 16000}

:output

Accepted formats:

%RawAudio{sample_format: :f32le, channels: 1, sample_rate: 16000}