Crawler v1.1.1 Crawler.Fetcher.Modifier

Modifies request options and headers before dispatch.

Link to this section Summary

Functions

Allows modifing headers prior to making a crawl request.

Allows passing opts to httpPoison prior to making the crawl request

Link to this section Functions

Allows modifing headers prior to making a crawl request.

Example implementation

def headers(opts) do
  if opts[:url] == "http://modifier" do
    [{"Referer", "http://fetcher"}]
  end
  []
end

Allows passing opts to httpPoison prior to making the crawl request

Example implementation

def opts(opts) do
  if opts[:url] == "http://modifier" do
    # add a new pool to hackney
    [hackney: [pool: :modifier]]
  end
  []
end