Parallel map implementation over iterator.
It starts workers immediately, not on-demand. It kills workers when there is no more work or
as close
cleanup.
The ordered
option controls if the order of elements in the resulting iterator should match
the order of input iterator.
- If ordered
is true
, the order is guaranteed, but pool utilization might be not optimal if
the time to process single element is not uniform (head-of-line blocking).
- If ordered
is false
, the order is not guaranteed, but pool utilization is optimal. Keep in
mind however that single worker can stay busy far beyond the recv_timeout
.
It prioritizes that all the workers are busy over returning the result immediately (so it does not return a result untill all workers are busy or inner iterator is depleted).
Worker processes are linked to the caller process. So it relies on link mechanism to kill the workers in case of errors in map function. It is not recommended to catch the error / trap exit and continue, because it may leave the workers hanging alive forever.tag() = any()
flush/1 | If your pmap has crashed and you had to catch the error, you can use this function to flush the results from the workers. |
pmap/2 | |
pmap/3 | Parallel map over iterator. |
flush(Tag::tag()) -> [pid()]
If your pmap has crashed and you had to catch the error, you can use this function to flush the results from the workers. But it is recommended to not catch the error and crash. If error is catched, there is a risk that workers are not killed.
pmap(F, I) -> any()
pmap(F::fun((InType) -> OutType), I::iterator:iterator(InType), Opts::#{concurrency => pos_integer(), recv_timeout => timeout(), ordered => boolean(), tag => tag()}) -> iterator:iterator(OutType)
InType = any()
OutType = any()
F
: function to apply to each element of the input iterator (executed inside worker process)
I
: input iterator.
Opts
: options:
concurrency
(default: 10) - number of workers; also, this number of items will be
read-ahead from the input iteratorrecv_timeout
(default: infinity) - timeout for receiving result from worker, if
reached, the pool will be shut down and timeout
error is generatedordered
(default: true) - if true
, the order of elements in the resulting iterator
will match the order of input iterator; otherwise order is not guaranteed, but worker
pool utilization will be better if task size is not uniformtag
(default: erlang:make_ref()
) - unique tag to identify the
pool (see flush/1
). Make sure it is small, because it is sent between processes
a lotParallel map over iterator.
Generated by EDoc