Funkspector.Utils (funkspector v1.2.2)

Common functionality for the scrapers.

Summary

Functions

Given a collection of URLs and a base URL, absolutifies the relative links.

Returns a boolean telling if the URL seems valid according to the regexp.

Functions

Link to this function

absolutify(links, base_url)

Given a collection of URLs and a base URL, absolutifies the relative links.

Examples

iex> Funkspector.Utils.absolutify(["javascript:alert(hi)", "/faqs?section=legal", "/about", "http://example.com/faqs"], "http://example.com")
["javascript:alert(hi)", "http://example.com/faqs?section=legal", "http://example.com/about", "http://example.com/faqs"]
Link to this function

valid_url?(url)

Returns a boolean telling if the URL seems valid according to the regexp.

Examples

iex> Funkspector.Utils.valid_url?("https://example.com") true

iex> Funkspector.Utils.valid_url?("joe@example.com") false

iex> Funkspector.Utils.valid_url?(nil) false

iex> Funkspector.Utils.valid_url?(" ") false