Spidey v0.1.0 Spidey View Source

Spidey is a basic web crawler which runs through all the links of a same domain and outputs them in a simple text sitemap format.

Link to this section Summary

Functions

Crawls a website for all the same-domain urls, returning a list.

Crawls a website for all the sam-domain urls and Saves the list of urls to file

Link to this section Functions

Crawls a website for all the same-domain urls, returning a list.

Link to this function

crawl_to_file(url, path)

View Source

Crawls a website for all the sam-domain urls and Saves the list of urls to file