Crawly v0.3.0 Crawly.Utils View Source
Utility functions for Crawly
Link to this section Summary
Functions
Pipeline/Middleware helper
A helper function which returns a Request structure for the given URL
A helper function which converts a list of URLS into a requests list.
Link to this section Functions
pipe(arg1, item, state)
View Source
pipe(pipelines, item, state) :: result
when pipelines: [Crawly.Pipeline.t()],
item: map(),
state: map(),
result: {new_item | false, new_state},
new_item: map(),
new_state: map()
pipe(pipelines, item, state) :: result when pipelines: [Crawly.Pipeline.t()], item: map(), state: map(), result: {new_item | false, new_state}, new_item: map(), new_state: map()
Pipeline/Middleware helper
Executes a given list of pipelines on the given item, mimics filtermap behavior (but probably in a more complex way). Takes an item and state and passes it through a list of modules which implements a pipeline behavior, executing the pipeline's run.
The pipe function must return boolean (false) or updated item. In case if false is returned the item is not being processed by all descendant pipelines, and dropped.
In case if a given pipeline crashes for the given item, it's result are being ignored, and the item is being processed by all other descendant pipelines.
The state variable is used to persist the information accross multiple items.
request_from_url(url)
View Source
request_from_url(binary()) :: Crawly.Request.t()
request_from_url(binary()) :: Crawly.Request.t()
A helper function which returns a Request structure for the given URL
requests_from_urls(urls)
View Source
requests_from_urls([binary()]) :: [Crawly.Request.t()]
requests_from_urls([binary()]) :: [Crawly.Request.t()]
A helper function which converts a list of URLS into a requests list.