Hourglass.Worker.WorkflowPollLoop (hourglass v0.4.0)

Copy Markdown View Source

Per-Worker Task that drives BridgeHolder.poll_workflow_activation/1 in a tight loop. On each {:ok, bytes} result it dispatches the activation directly to the shared Hourglass.WorkflowEvaluator.DynamicSupervisor, spawning a fresh ephemeral evaluator Task per activation. The evaluator runs once and exits :normal; Core handles redelivery on crashes.

Bridge access

This loop does NOT hold a raw bridge handle. Every iteration calls BridgeHolder.poll_workflow_activation(task_queue) and the Application-level holder mediates the NIF call (dispatching the long-poll to its own child Task and replying when Core delivers).

Shutdown

When BridgeHolder.poll_workflow_activation/1 returns {:error, %Bridge.Error{kind: :shutdown}} the loop disambiguates via BridgeHolder.registered?/1:

  • registered? → false: the Worker is being torn down (graceful Worker.terminate/2 already called unregister_worker). Exit :normal and let the inner Supervisor reap the loop.
  • registered? → true: the holder recycled the bridge handle mid-flight (poll-caller DOWN). Sleep briefly + retry against the new handle.

If the call returns {:error, :worker_not_registered} the loop treats it as a transient retry condition (sleep + retry) — registration may be in flight, e.g. the holder restarted moments ago and Worker.Supervisor.start_worker/1's register call hasn't completed yet.

Other bridge errors log + sleep + retry. The Supervisor's :transient restart for this child means :normal exit is final (no flapping during graceful shutdown).

Sticky-cache-miss recovery (do NOT crash the loop)

WorkflowTypeResolver.resolve/2 has two tiers: the activation's initialize_workflow job (present on full-history / non-sticky deliveries) and the worker's WorkflowStateCache (populated by prior activations, mirrors Core's sticky cache). An incremental (sticky) activation for a run this worker never cached — or evicted — exhausts both tiers. That is a recoverable Core/worker cache desync, not a bug in the activation; it shows up under load when workflows sit idle for long stretches or the live-workflow count approaches max_cached_workflows.

Historically resolve/2 raised here, the raise propagated through dispatch/2loop/1, and the whole poll-loop Task crashed. It self-healed only because Core eventually timed out the sticky task and re-delivered non-sticky with full history — but every occurrence was a crash + :transient restart + re-register + replay, spamming errors.

The correct Temporal behavior is to fail the current (sticky) workflow task: ship a WorkflowActivationCompletion{status: {:failed, ...}} for the run. SDK-Core responds by evicting the run from its sticky cache and reporting the workflow-task failure to the server; the server reschedules the task, and because Core no longer has the run cached it fetches full history and re-delivers non-sticky — that activation carries initialize_workflow, so tier 1 resolves it. The poll loop never crashes: dispatch/2 ships the failure inline and keeps polling. Workflow tasks retry indefinitely on the server (they do not fail the workflow), so this is safe and self-correcting.

Summary

Types

Options accepted by start_link/1.

Functions

Loop entry. Iterates BridgeHolder.poll_workflow_activation/1 until shutdown.

Spawn a linked Task that runs the poll loop until the bridge reports :shutdown or the parent supervisor terminates it.

Types

opts()

@type opts() :: [
  task_queue: String.t(),
  complete_fn: (String.t(), binary() -> :ok | {:error, term()})
]

Options accepted by start_link/1.

Functions

run(opts)

@spec run(opts()) :: :ok

Loop entry. Iterates BridgeHolder.poll_workflow_activation/1 until shutdown.

start_link(opts)

@spec start_link(opts()) :: {:ok, pid()}

Spawn a linked Task that runs the poll loop until the bridge reports :shutdown or the parent supervisor terminates it.