People build scrapers that can find every Applebee’s on the planet or collect congressional legislation and votes or track fancy watches for sale on fan websites. Businesses use scrapers to manage their online retail inventory and monitor competitors’ prices. Lots of well-known sites use scrapers to do things like track airline ticket prices and job listings. Google is essentially a giant, crawling web scraper.
Scrapers are also the tools of watchdogs and journalists, which is why The Markup filed an amicus brief in a case before the U.S. Supreme Court this week that threatens to make scraping illegal.
I’ve become quite a fan of The Markup’s work over the past year or so. They are fighting for important causes in the modern tech world, especially user and data privacy. They created the Blacklight, a really cool service that inspects web pages and reports on trackers.
This piece on scraping really hits home for me too. A few years ago, I co-founded and then sold a startup almost entirely based on scraping tech that I created. It was really fun and would have never been possible without web scrapers like Google.