View Source Ollama.Streaming (Ollama v0.5.0)

When an Ollama endpoint is called with the :stream option set to true, a Ollama.Streaming.t/0 struct is returned providing a unqiue reference/0 for the streaming request, and a lazy enumerable that generates messages as they are received from the streaming request.

Ollama.Streaming implements the Enumerable protocol, so can be used directly with any Stream functions. Most of the time, you'll just want to asynchronously call send_to/2, which will run the stream and send each message to a process of your chosing.

Messages are sent in the following format, allowing the receiving process to pattern match against the reference/0 of the streaming request:

{request_ref, {:data,  :data}}

Example

A typical example is to make a streaming request as part of a LiveView event, and send each of the streaming messages back to the same LiveView process.

defmodule MyApp.ChatLive do
  use Phoenix.LiveView
  alias Ollama.Streaming

  # When the client invokes the "prompt" event, create a streaming request and
  # asynchronously send messages back to self.
  def handle_event("prompt", %{"message" => prompt}, socket) do
    {:ok, streamer} = Ollama.completion(Ollama.init(), [
      model: "llama2",
      prompt: prompt,
      stream: true,
    ])

    pid = self()
    {:noreply,
      socket
      |> assign(current_request: streamer.ref)
      |> start_async(:streaming, fn -> Streaming.send_to(streaming, pid) end)
    }
  end

  # The streaming request sends messages back to the LiveView process.
  def handle_info({_request_ref, {:data, _data}} = message, socket) do
    ref = socket.assigns.current_request
    case message do
      {^ref, {:data, %{"done" => false} = data}} ->
        # handle each streaming chunk

      {^ref, {:data, %{"done" => true} = data}} ->
        # handle the final streaming chunk

      {_ref, _data} ->
        # this message was not expected!
    end
  end

  # When the streaming request is finished, remove the current reference.
  def handle_async(:streaming, :ok, socket) do
    {:noreply, assign(socket, current_request: nil)}
  end
end

Summary

Types

t()

Streaming request struct

Functions

Runs the stream and sends each message from the streaming request to the process associated with the given pid.

Types

@type t() :: %Ollama.Streaming{enum: Enumerable.t(), ref: reference()}

Streaming request struct

Functions

@spec send_to(t(), pid()) :: :ok

Runs the stream and sends each message from the streaming request to the process associated with the given pid.

Messages are sent in the following format, allowing the receiving process to pattern match against the reference/0 of the streaming request:

{request_ref, {:data,  :data}}