crawlie v1.0.0 Crawlie.Stage.UrlManager.State
Summary
Functions
Marks the uri as “already visited” in the state
Checks if the uri was already visited by the crawler
Types
t()
t() :: %Crawlie.Stage.UrlManager.State{discovered: Crawlie.PqueueWrapper.t, in_flight: MapSet.t, options: Keyword.t, pending_demand: integer, visited: MapSet.t}
Functions
finished_processing(state, uri)
finished_processing(Crawlie.Stage.UrlManager.State.t, String.t) :: Crawlie.Stage.UrlManager.State.t
started_processing(state, uri)
started_processing(Crawlie.Stage.UrlManager.State.t, URI.t) :: Crawlie.Stage.UrlManager.State.t
take_pages(state, count)
take_pages(Crawlie.Stage.UrlManager.State.t, integer) :: {Crawlie.Stage.UrlManager.State.t, [Crawlie.Page.t]}
Checks if the uri was already visited by the crawler