SitemapXml.SitemapUrls (sitemap_xml v0.1.1)

A module to fetch and parse URLs form sitemap XML concurrently.

Summary

Functions

Issues a DELETE request to the given url.

Issues a DELETE request to the given url, raising an exception in case of failure.

Fetches the raw sitemap XML from the given URL.

Fetches and parses the sitemap from the provided URL.

Issues a GET request to the given url.

Issues a GET request to the given url, raising an exception in case of failure.

Issues a HEAD request to the given url.

Issues a HEAD request to the given url, raising an exception in case of failure.

Issues an OPTIONS request to the given url.

Issues a OPTIONS request to the given url, raising an exception in case of failure.

Parses the sitemap XML to extract URLs or process nested sitemaps.

Issues a PATCH request to the given url.

Issues a PATCH request to the given url, raising an exception in case of failure.

Issues a POST request to the given url.

Issues a POST request to the given url, raising an exception in case of failure.

Callback implementation for HTTPoison.Base.process_headers/1.

process_url(url) deprecated

Callback implementation for HTTPoison.Base.process_url/1.

Issues a PUT request to the given url.

Issues a PUT request to the given url, raising an exception in case of failure.

Issues an HTTP request using a Request struct.

Issues an HTTP request with the given method to the given url.

Issues an HTTP request with the given method to the given url, raising an exception in case of failure.

Starts HTTPoison and its dependencies.

Requests the next message to be streamed for a given HTTPoison.AsyncResponse.

Types

@type body() :: HTTPoison.Base.body()
@type headers() :: HTTPoison.Base.headers()
@type method() :: HTTPoison.Base.method()
@type options() :: HTTPoison.Base.options()
@type params() :: HTTPoison.Base.params()
@type request() :: HTTPoison.Base.request()
@type url() :: HTTPoison.Base.url()

Functions

Link to this function

delete(url, headers \\ [], options \\ [])

Issues a DELETE request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

delete!(url, headers \\ [], options \\ [])

Issues a DELETE request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

fetch_sitemap(url)

Fetches the raw sitemap XML from the given URL.

Examples

iex> SitemapXml.SitemapUrls.fetch_sitemap("https://web.site/sitemap.xml")
{:ok, "<urlset>...</urlset>"}

iex> SitemapXml.SitemapUrls.fetch_sitemap("https://web.site/404.xml")
{:error, "HTTP error with status 404"}
Link to this function

fetch_urls(url)

Fetches and parses the sitemap from the provided URL.

Examples

iex> SitemapXml.SitemapUrls.fetch_urls("https://web.site/sitemap.xml")
{:ok, ["https://web.site/page1", "https://web.site/page2", ...]}
Link to this function

get(url, headers \\ [], options \\ [])

@spec get(binary(), headers(), Keyword.t()) ::
  {:ok, HTTPoison.Response.t() | HTTPoison.AsyncResponse.t()}
  | {:error, HTTPoison.Error.t()}

Issues a GET request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

get!(url, headers \\ [], options \\ [])

Issues a GET request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

head(url, headers \\ [], options \\ [])

@spec head(binary(), headers(), Keyword.t()) ::
  {:ok, HTTPoison.Response.t() | HTTPoison.AsyncResponse.t()}
  | {:error, HTTPoison.Error.t()}

Issues a HEAD request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

head!(url, headers \\ [], options \\ [])

Issues a HEAD request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

options(url, headers \\ [], options \\ [])

Issues an OPTIONS request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

options!(url, headers \\ [], options \\ [])

Issues a OPTIONS request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

parse_sitemap(body)

Parses the sitemap XML to extract URLs or process nested sitemaps.

Examples

iex> SitemapXml.SitemapUrls.parse_sitemap("<urlset>...</urlset>")
{:ok, ["https://web.site/page1", "https://web.site/page2"]}

iex> SitemapXml.SitemapUrls.parse_sitemap("<sitemapindex>...</sitemapindex>")
...
Link to this function

patch(url, body, headers \\ [], options \\ [])

Issues a PATCH request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

patch!(url, body, headers \\ [], options \\ [])

Issues a PATCH request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

post(url, body, headers \\ [], options \\ [])

Issues a POST request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

post!(url, body, headers \\ [], options \\ [])

Issues a POST request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

process_headers(headers)

This function is deprecated. Use process_response_headers/1 instead.
@spec process_headers(list()) :: any()

Callback implementation for HTTPoison.Base.process_headers/1.

Link to this function

process_request_body(body)

@spec process_request_body(body()) :: body()

Callback implementation for HTTPoison.Base.process_request_body/1.

Link to this function

process_request_headers(headers)

@spec process_request_headers(headers()) :: headers()

Callback implementation for HTTPoison.Base.process_request_headers/1.

Link to this function

process_request_options(options)

@spec process_request_options(options()) :: options()

Callback implementation for HTTPoison.Base.process_request_options/1.

Link to this function

process_request_params(params)

@spec process_request_params(params()) :: params()

Callback implementation for HTTPoison.Base.process_request_params/1.

Link to this function

process_request_url(url)

@spec process_request_url(url()) :: url()

Callback implementation for HTTPoison.Base.process_request_url/1.

Link to this function

process_response(response)

@spec process_response(HTTPoison.Base.response()) :: any()

Callback implementation for HTTPoison.Base.process_response/1.

Link to this function

process_response_body(body)

@spec process_response_body(binary()) :: any()

Callback implementation for HTTPoison.Base.process_response_body/1.

Link to this function

process_response_chunk(chunk)

@spec process_response_chunk(binary()) :: any()

Callback implementation for HTTPoison.Base.process_response_chunk/1.

Link to this function

process_response_headers(headers)

@spec process_response_headers(list()) :: any()

Callback implementation for HTTPoison.Base.process_response_headers/1.

Link to this function

process_response_status_code(status_code)

@spec process_response_status_code(integer()) :: any()

Callback implementation for HTTPoison.Base.process_response_status_code/1.

Link to this function

process_status_code(status_code)

This function is deprecated. Use process_response_status_code/1 instead.
@spec process_status_code(integer()) :: any()

Callback implementation for HTTPoison.Base.process_status_code/1.

Link to this function

process_url(url)

This function is deprecated. Use process_request_url/1 instead.
@spec process_url(url()) :: url()

Callback implementation for HTTPoison.Base.process_url/1.

Link to this function

put(url, body \\ "", headers \\ [], options \\ [])

Issues a PUT request to the given url.

Returns {:ok, response} if the request is successful, {:error, reason} otherwise.

See request/5 for more detailed information.

Link to this function

put!(url, body \\ "", headers \\ [], options \\ [])

Issues a PUT request to the given url, raising an exception in case of failure.

If the request does not fail, the response is returned.

See request!/5 for more detailed information.

Link to this function

request(request)

Issues an HTTP request using a Request struct.

This function returns {:ok, response}, {:ok, async_response}, or {:ok, maybe_redirect} if the request is successful, {:error, reason} otherwise.

Redirect handling

If the option :follow_redirect is given, HTTP redirects are automatically follow if the method is set to :get or :head and the response's status_code is 301, 302 or 307.

If the method is set to :post, then the only status_code that get's automatically followed is 303.

If any other method or status_code is returned, then this function returns a returns a {:ok, %HTTPoison.MaybeRedirect{}} containing the redirect_url for you to re-request with the method set to :get.

Examples

request = %HTTPoison.Request{
  method: :post,
  url: "https://my.website.com",
  body: "{\"foo\": 3}",
  headers: [{"Accept", "application/json"}]
}

request(request)
Link to this function

request(method, url, body \\ "", headers \\ [], options \\ [])

Issues an HTTP request with the given method to the given url.

This function is usually used indirectly by get/3, post/4, put/4, etc

Args:

  • method - HTTP method as an atom (:get, :head, :post, :put, :delete, etc.)
  • url - target url as a binary string or char list
  • body - request body. See more below
  • headers - HTTP headers as an orddict (e.g., [{"Accept", "application/json"}])
  • options - Keyword list of options

Body: see type HTTPoison.Request

Options: see type HTTPoison.Request

This function returns {:ok, response}, {:ok, async_response}, or {:ok, maybe_redirect} if the request is successful, {:error, reason} otherwise.

Redirect handling

If the option :follow_redirect is given, HTTP redirects are automatically follow if the method is set to :get or :head and the response's status_code is 301, 302 or 307.

If the method is set to :post, then the only status_code that get's automatically followed is 303.

If any other method or status_code is returned, then this function returns a returns a {:ok, %HTTPoison.MaybeRedirect{}} containing the redirect_url for you to re-request with the method set to :get.

Examples

request(:post, "https://my.website.com", "{\"foo\": 3}", [{"Accept", "application/json"}])
Link to this function

request!(method, url, body \\ "", headers \\ [], options \\ [])

Issues an HTTP request with the given method to the given url, raising an exception in case of failure.

request!/5 works exactly like request/5 but it returns just the response in case of a successful request, raising an exception in case the request fails.

Starts HTTPoison and its dependencies.

Link to this function

stream_next(resp)

@spec stream_next(HTTPoison.AsyncResponse.t()) ::
  {:ok, HTTPoison.AsyncResponse.t()} | {:error, HTTPoison.Error.t()}

Requests the next message to be streamed for a given HTTPoison.AsyncResponse.

See request!/5 for more detailed information.