crawlie v0.5.1 Crawlie.Stage.UrlManager.State

Summary

Types

t()
t() :: %Crawlie.Stage.UrlManager.State{discovered: Crawlie.PqueueWrapper.t, in_flight: MapSet.t, options: Keyword.t, pending_demand: integer, visited: MapSet.t}

Functions

add_page(state, page)
finished_crawling?(state)
finished_crawling?(Crawlie.Stage.UrlManager.State.t) :: boolean
finished_processing(state, uri)
new(initial_pages, options)
started_processing(state, uri)
take_pages(state, count)

Marks the uri as “already visited” in the state

visited?(state, uri)
visited?(Crawlie.Stage.UrlManager.State.t, URI.t) :: boolean

Checks if the uri was already visited by the crawler