PROJECT 07 / 11
Crawler
Modular Python crawler for Norwegian classifieds — polite, diff-based, drives the search and scoring tools.
The data layer underneath Turboscore and Spotscore. A modular crawler for the public listing portals across verticals (cars, real estate, jobs) with category-specific parsers, respectful rate limits, full HTML snapshotting, and diff-based change detection so we can show price-history trails without re-fetching every listing every day.
Built to be polite — sustained throughput well under what the source sees from a human browser session, request fingerprints that don’t pretend to be something they aren’t, and a per-listing TTL that respects the underlying market dynamics.