MLServe.ModelInstance (MLServe v0.1.0)

Copy Markdown View Source

The supervision subtree for one loaded model version.

MLServe.ModelInstance                (:rest_for_one)
├── MLServe.ModelServer              lifecycle, status, drain
├── MLServe.WorkerSupervisor         only when concurrency: :exclusive
│     └── MLServe.Worker × N
└── MLServe.Batcher                  only when batching is configured

:rest_for_one is the correct strategy and the ordering is load-bearing. MLServe.ModelServer owns the loaded backend state that workers are handed; if it dies, the state its workers hold is stale, so the workers and batcher must restart behind it. The reverse is not true — a crashed worker has no bearing on the model server, so restarting only what follows is exactly right.

The worker supervisor and batcher are started by MLServe.ModelServer once loading succeeds, rather than declared here, because neither can exist before there is state to give them.