shigoto_resilience (shigoto v1.9.10)
View SourceSeki-based resilience integration for Shigoto workers.
Provides per-worker rate limiting, circuit breaking, bulkhead (concurrency limiting), and system-level load shedding. All primitives are lazily initialized on first use based on worker callbacks.
Rate Limiting
Workers that export rate_limit/0 get per-worker rate limiting via seki.
Jobs that hit the rate limit are snoozed (rescheduled for later) rather than failed.
Circuit Breaking
When a worker fails repeatedly, its circuit breaker opens and subsequent jobs are snoozed until the breaker transitions to half-open, keeping work off a known-broken dependency.
Bulkhead
Workers that export concurrency/0 get a global concurrency limit across
all queues. This is separate from per-queue concurrency.
Load Shedding
System-level CoDel-based load shedding can be enabled via config. Low-priority jobs are shed first when the system is overloaded.
Summary
Functions
Check per-worker bulkhead. Returns ok or {snooze, 5}.
Check circuit breaker for a worker. Returns ok or {snooze, 10}.
Check the per-worker global concurrency limit across all nodes via PostgreSQL.
Execute through circuit breaker, returning the result.
Check per-worker rate limit. Returns ok or {snooze, Seconds}.
Report load shedder completion.
Ensure seki primitives exist for a worker. Lazy and idempotent.
Release bulkhead slot for a worker.
Set up system-level load shedder if configured.
Functions
-spec check_bulkhead(module(), map()) -> ok | {snooze, pos_integer()}.
Check per-worker bulkhead. Returns ok or {snooze, 5}.
-spec check_circuit(module(), map()) -> ok | {snooze, pos_integer()}.
Check circuit breaker for a worker. Returns ok or {snooze, 10}.
-spec check_global_concurrency(module(), map()) -> ok | {snooze, pos_integer()}.
Check the per-worker global concurrency limit across all nodes via PostgreSQL.
The check-and-admit is serialised per worker with a PostgreSQL transaction-scoped
advisory lock, so two nodes can never both admit a job past the cap. It counts
other jobs of the worker already executing (excluding the job being checked,
which claiming has already marked executing), so a job at exactly the limit is
not spuriously snoozed. When the limit is reached the job's slot is freed back to
available inside the same locked transaction, so the next serialised checker
observes the reduced count. That makes a burst of concurrently claimed jobs snooze
one at a time until exactly global_concurrency/0 remain running, instead of all
snoozing together and livelocking.
Returns ok or {snooze, Seconds}. Fails open (ok) on any database error, so a
transient DB problem never turns the check into a job failure.
-spec check_load(map()) -> ok | {snooze, pos_integer()}.
Execute through circuit breaker, returning the result.
-spec check_rate_limit(module(), map()) -> ok | {snooze, pos_integer()}.
Check per-worker rate limit. Returns ok or {snooze, Seconds}.
-spec complete_load(map(), non_neg_integer()) -> ok.
Report load shedder completion.
-spec ensure_worker_primitives(module()) -> ok.
Ensure seki primitives exist for a worker. Lazy and idempotent.
-spec release_bulkhead(module()) -> ok.
Release bulkhead slot for a worker.
-spec setup_load_shedder() -> ok.
Set up system-level load shedder if configured.