crawlie v0.3.1 Crawlie.Stage.UrlManager.State
Summary
Functions
Marks the url as “already visited” in the state
Checks if the url was already visited by the crawler
Types
t()
t() :: %Crawlie.Stage.UrlManager.State{discovered: Crawlie.PqueueWrapper.t, in_flight: MapSet.t, initial: Enum.t, options: Keyword.t, pending_demand: integer, visited: MapSet.t}
Functions
finished_processing(state, url)
finished_processing(Crawlie.Stage.UrlManager.State.t, String.t) :: Crawlie.Stage.UrlManager.State.t
started_processing(state, url)
started_processing(Crawlie.Stage.UrlManager.State.t, String.t) :: Crawlie.Stage.UrlManager.State.t
take_pages(state, count)
take_pages(Crawlie.Stage.UrlManager.State.t, integer) :: {Crawlie.Stage.UrlManager.State.t, [Crawlie.Page.t]}
Checks if the url was already visited by the crawler