Assembly Line v0.5.0 AssemblyLine.JobQueue.Handler
Responsible for passing jobs from the queue to whatever the backend responsible for processing them.
The Handler consumes work from the specified queue and hands the job records
to the appropriate workers for processing. In the case that the
AssemblyLine.Job.t
struct doesn’t specify a worker
then the worker will
be pulled from the Application Envrionment.
Processing a Queue
Starting the processing of a queue is simple:
alias AssemblyLine.JobQueue.Handler
Handler.start_all "the doc queue"
start_all/1
will execute until all jobs finish or a job exits abnormally.
It is important to note that the Handler
functions will not timeout. They
will continue to wait on the worker until some form of response is received.
If you wish to implement a timeout your worker’s perform
function should
track that and exit
after the time limit has been reached.
When a stage in the job queue has multiple jobs which can be executed in
parallel the Handler
will keep track of the jobs which succeeded and the
jobs which failed. All successful jobs are removed from the set in the
AssemblyLine.JobQueue.Server
. This means it is always safe to re-attempt
processing in the case of a partial completion.
Configuration
There are two configuration values which govern the behavior of the Handler
:
check_interval
job_executor
The check_interval
determines how many miliseconds the Handler
waits for
results from the workers. If this value is not set then a default of 1000
(1 second) will be used.
If this interval elapses before all the workers have replied the Handler
will proceed to update the job queue state for those jobs that have either
succeeded or failed to taht point. It will then resume waiting, this loop
will continue until all jobs have succeeded or failed.
The job_executor
provides an option for setting a default Module that should
process jobs. The worker can be set on a per job basis as well so this value
is entirely optional.
Summary
Functions
Processes the specified queue
Processes a set of n jobs
Starts job processing for the queue
Functions
Specs
process(String.t, [AssemblyLine.Job.t]) ::
{:incomplete, [AssemblyLine.Job.t]} |
:finished
Processes the specified queue
Returns :finished
if the entire pipeline completes successfuly. Otherwise
it returns {:incomplete, [%AssemblyLine.Job{}]}
where the second part of
the tuple is a list of AssemblyLine.Job
structs which failed to process.
Specs
process_set(String.t, [AssemblyLine.Job.t]) ::
{:incomplete, [AssemblyLine.Job.t]} |
{:incomplete, []}
Processes a set of n jobs.
Returns {:incomplete, list}
where list
is a list of jobs that failed, the
list
can be empty.
Specs
start_all(String.t) :: :finished | :incomplete
Starts job processing for the queue
.
Returns :finished
or :incomplete
.