crawlie v0.3.1 Crawlie.Stage.UrlManager.State

Summary

Types

t()
t() :: %Crawlie.Stage.UrlManager.State{discovered: Crawlie.PqueueWrapper.t, in_flight: MapSet.t, initial: Enum.t, options: Keyword.t, pending_demand: integer, visited: MapSet.t}

Functions

add_page(state, page)
finished_crawling?(state)
finished_crawling?(Crawlie.Stage.UrlManager.State.t) :: boolean
finished_processing(state, url)
new(initial_pages, options)
started_processing(state, url)
take_pages(state, count)

Marks the url as “already visited” in the state

visited?(state, url)

Checks if the url was already visited by the crawler