Crawler v1.1.1 Crawler.Fetcher.Modifier
Modifies request options and headers before dispatch.
Link to this section Summary
Functions
Allows modifing headers prior to making a crawl request.
Allows passing opts to httpPoison prior to making the crawl request
Link to this section Functions
Link to this function
headers(opts)
Allows modifing headers prior to making a crawl request.
Example implementation
def headers(opts) do
if opts[:url] == "http://modifier" do
[{"Referer", "http://fetcher"}]
end
[]
end
Link to this function
opts(opts)
Allows passing opts to httpPoison prior to making the crawl request
Example implementation
def opts(opts) do
if opts[:url] == "http://modifier" do
# add a new pool to hackney
[hackney: [pool: :modifier]]
end
[]
end