VowpalFleet
Vowpal Fleet - manage Vowpal Wabbit instances usint Swarm
Info
- create one cluster per model using
VowpalFleet.start_worker/2
VowpalFleet.train/3
goes to all living instanceVowpalFleet.predict/2
picks random instance to do the prediction- configure auto save interval, root directory and vw options with mix config
:vowpal_fleet
inconfig.exs
Installation
- Make sure you have Vowpal Wabbit installed and it is findable in
$PATH
- add the dependency to your mix.exs
def deps do
[
{:vowpal_fleet, "~> 0.1.0"}
]
end
ef application do
[
extra_applications: [:vowpal_fleet]
]
end
- configure the parameters, edit
config/config.exs
config :vowpal_fleet,
root: "/tmp/vw",
some_cluster_id: %{:autosave => 300_000, :args => ["--random_seed", "123"]}
Work In Progress
More testing is needed to ensure that the failure scenarios are covered, at the moment the code just works but.. well take it with grain of salt
Examples
iex> VowpalFleet.start_worker(:some_cluster_id, :instance_1)
...
:ok
iex> VowpalFleet.start_worker(:some_cluster_id, :instance_2)
...
:ok
iex> VowpalFleet.train(:some_cluster_id, 1, [{"features", [1, 2, 3]}])
:ok
iex> VowpalFleet.predict(:some_cluster_id, [{"features", [1, 2, 3]}])
1.0
Handoff
When the process has to be moved to a different node, the working model is saved, and then handed off to the starting process
Links
credit
Icons made by Freepik from Flaticon is licensed by CC 3.0 BY