Module iterator_pmap

Parallel map implementation over iterator.

Description

Parallel map implementation over iterator.

It starts workers immediately, not on-demand. It kills workers when there is no more work or as close cleanup.

The ordered option controls if the order of elements in the resulting iterator should match the order of input iterator. - If ordered is true, the order is guaranteed, but pool utilization might be not optimal if the time to process single element is not uniform (head-of-line blocking). - If ordered is false, the order is not guaranteed, but pool utilization is optimal. Keep in mind however that single worker can stay busy far beyond the recv_timeout.

It prioritizes that all the workers are busy over returning the result immediately (so it does not return a result untill all workers are busy or inner iterator is depleted).

Worker processes are linked to the caller process. So it relies on link mechanism to kill the workers in case of errors in map function. It is not recommended to catch the error / trap exit and continue, because it may leave the workers hanging alive forever.

Data Types

tag()

tag() = any()

Function Index

flush/1If your pmap has crashed and you had to catch the error, you can use this function to flush the results from the workers.
pmap/2
pmap/3Parallel map over iterator.

Function Details

flush/1

flush(Tag::tag()) -> [pid()]

If your pmap has crashed and you had to catch the error, you can use this function to flush the results from the workers. But it is recommended to not catch the error and crash. If error is catched, there is a risk that workers are not killed.

pmap/2

pmap(F, I) -> any()

pmap/3

pmap(F::fun((InType) -> OutType), I::iterator:iterator(InType), Opts::#{concurrency => pos_integer(), recv_timeout => timeout(), ordered => boolean(), tag => tag()}) -> iterator:iterator(OutType)

F: function to apply to each element of the input iterator (executed inside worker process)
I: input iterator.
Opts: options:


Parallel map over iterator.


Generated by EDoc