View Source Ollama.Streaming (Ollama v0.5.0)
When an Ollama endpoint is called with the :stream
option set to true
, a
Ollama.Streaming.t/0
struct is returned providing a unqiue reference/0
for the streaming request, and a lazy enumerable that generates messages as
they are received from the streaming request.
Ollama.Streaming
implements the Enumerable
protocol, so can be used
directly with any Stream
functions. Most of the time, you'll just want to
asynchronously call send_to/2
, which will run the stream and send each
message to a process of your chosing.
Messages are sent in the following format, allowing the receiving process
to pattern match against the reference/0
of the streaming request:
{request_ref, {:data, :data}}
Example
A typical example is to make a streaming request as part of a LiveView event, and send each of the streaming messages back to the same LiveView process.
defmodule MyApp.ChatLive do
use Phoenix.LiveView
alias Ollama.Streaming
# When the client invokes the "prompt" event, create a streaming request and
# asynchronously send messages back to self.
def handle_event("prompt", %{"message" => prompt}, socket) do
{:ok, streamer} = Ollama.completion(Ollama.init(), [
model: "llama2",
prompt: prompt,
stream: true,
])
pid = self()
{:noreply,
socket
|> assign(current_request: streamer.ref)
|> start_async(:streaming, fn -> Streaming.send_to(streaming, pid) end)
}
end
# The streaming request sends messages back to the LiveView process.
def handle_info({_request_ref, {:data, _data}} = message, socket) do
ref = socket.assigns.current_request
case message do
{^ref, {:data, %{"done" => false} = data}} ->
# handle each streaming chunk
{^ref, {:data, %{"done" => true} = data}} ->
# handle the final streaming chunk
{_ref, _data} ->
# this message was not expected!
end
end
# When the streaming request is finished, remove the current reference.
def handle_async(:streaming, :ok, socket) do
{:noreply, assign(socket, current_request: nil)}
end
end
Summary
Functions
Runs the stream and sends each message from the streaming request to the process associated with the given pid.
Types
@type t() :: %Ollama.Streaming{enum: Enumerable.t(), ref: reference()}
Streaming request struct
Functions
Runs the stream and sends each message from the streaming request to the process associated with the given pid.
Messages are sent in the following format, allowing the receiving process
to pattern match against the reference/0
of the streaming request:
{request_ref, {:data, :data}}