A production-grade recursive web crawler and scraper for legacy phpBB2 forums supporting both public (anonymous) and authenticated scraping built with Elixir, Finch, and Floki.
It automatically walks the board hierarchy—from index categories to individual forums, paginated topic threads, posts, and user profiles—utilizing concurrent streams and built-in rate limiting to safely migrate data without overwhelming legacy server infrastructure.
Examples
# 1. Initialize an anonymous (public) scraper session
crawler = Phpbb2Crawler.new("http://legacy-forum.local/phpBB2")
# 2. Alternatively, authenticate to access protected boards
{:ok, crawler} = Phpbb2Crawler.login("http://legacy-forum.local/phpBB2", "admin", "secret_pass")
# 3. Execute the full recursive crawl
scraped_data = Phpbb2Crawler.crawl_all(crawler)
IO.inspect(length(scraped_data.forums), label: "Total Forums")
IO.inspect(length(scraped_data.topics), label: "Total Topics")
IO.inspect(length(scraped_data.posts), label: "Total Posts")Summary
Functions
Starts a full crawl (works whether the struct has a cookie or is nil).
Logs into the phpBB2 forum and returns an authenticated crawler struct.
Initializes a public (unauthenticated) crawler session.
Lazily streams topics across all pages of a forum.
Lazily streams posts across all pages of a topic.
Functions
Starts a full crawl (works whether the struct has a cookie or is nil).
Logs into the phpBB2 forum and returns an authenticated crawler struct.
Initializes a public (unauthenticated) crawler session.
Lazily streams topics across all pages of a forum.
Lazily streams posts across all pages of a topic.