View Source README

fluvio-ex

💧Elixir client for Fluvio streaming platform

unit tests status test coverage hex.pm version hex.pm downloads

Docs

installation

Installation

You can add fluvio to your list of dependencies in mix.exs:

def deps do
  [
    {:fluvio, "~> 0.2.0"}
  ]
end

features

Features

The following Fluvio Rust abstractions are supported on Elixir:

RustElixir
fluvio::PartitionConsumerFluvio.Consumer
fluvio::TopicProducerFluvio.Producer
fluvio::FluvioAdminFluvio.Admin

Each Elixir abstraction tend to provide equivalent functionalities that fluvio crate exposes, although more Elixir-oriented to be well integrated with Elixir ecosystem and OTP. fluvio::FluvioAdmin is minimally supported since provisioning cluster resources is typically done upfront and not at the application level, but for experimentation it's useful to have Fluvio.Admin.

examples

Examples

consumer

Consumer

This snippet illustrates a Consumer connected to a "lobby" topic and using an optional SmarModule filter. Initially, this consumer calls Consumer.stream_unfold/2 to lazily take 4 reords and chunk every 2 records, printing them. After that, keeps executing Consumer.stream_each/2 consuming the stream. Consumer.stream_each is a higher-level function that recursively calls Consumer.stream_next/2.

alias Fluvio.Consumer

{:ok, pid} =
  Consumer.start_link(%{
    topic: "lobby",
    offset: [from_beginning: 0],
    smartmodule_path: "./examples/wasm/map_reverse.wasm"
  })

Consumer.stream_unfold(pid)
|> Stream.take(4)
|> Stream.chunk_every(2)
|> Enum.to_list()
|> IO.inspect()

Consumer.stream_each(pid, fn result ->
  case result do
    {:ok, record} -> IO.inspect(record)
    {:error, msg} -> IO.inspect("Error: #{msg}")
  end
end)

producer

Producer

This snippet illustrates a Producer connected to a "lobby" topic. Initially, a string "hello" is sent and flushed, after that, more three strigs "are", "you", "there?" are sent and asserted. Finally, integers from 1 to 20 mapped as strings are sent asynchronously in chunk of 10 performing a flush between each iteration. If you're going to send and await records asynchronusly, make sure that it's suitable and benefitial to your application:

alias Fluvio.Producer

{:ok, pid} = Producer.start_link(%{topic: "lobby"})

{:ok, _} = Producer.send(pid, "hello")
{:ok, _} = Producer.flush(pid)

Producer.send(pid, ["are", "you", "there?"])
|> Enum.each(&({:ok, _} = &1))

{:ok, _} = Producer.flush(pid)

[] =
  1..20
  |> Stream.chunk_every(10)
  |> Stream.flat_map(fn chunk ->
    [
      chunk
      |> Enum.map(fn value ->
        Task.async(fn -> {Producer.send(pid, to_string(value)), value} end)
      end)
      |> Task.await_many()
      |> Enum.filter(&match?({{:error, _msg}, _value}, &1)),
      [{Producer.flush(pid), :flush}]
      |> Enum.filter(&match?({{:error, _msg}, _value}, &1))
    ]
  end)
  |> Stream.concat()
  |> Enum.to_list()