Web scraping is the automated extraction of data from websites. Every business now uses it: sales teams to build lead lists, marketers to monitor competitor pricing, product teams to gather training data for AI models, researchers to compile public-record datasets. This guide explains what it is, when to use it vs an API, and how to do it legally.
Scraping vs API: which to use
APIs are official data interfaces (Google Maps Places API, LinkedIn Recruiter API). They're usually rate-limited, paid, and restrict what you can store. Scraping pulls the same data from public web pages — slower but unlimited and yours to keep. Use the API when it covers your need and the cost works; scrape when the API doesn't exist, costs too much, or restricts data usage.
Is web scraping legal?
In most jurisdictions (US, EU, UK): yes, for publicly accessible data not behind a login. The hiQ Labs v. LinkedIn case (2022, US 9th Circuit) confirmed that scraping public LinkedIn profiles doesn't violate the Computer Fraud and Abuse Act. EU rules under GDPR require lawful basis when scraping personal data; for B2B (business email, name, role) the "legitimate interest" basis usually applies. Consult a lawyer for high-volume commercial use.
What tools do people use?
Three tiers: (1) DIY — Python + BeautifulSoup or Playwright. Cheap but you maintain it. (2) Self-serve cloud — Kavex, Outscraper, Apify. Pay per result, no maintenance. (3) Enterprise — Bright Data, Apify Enterprise. Customised + dedicated proxies for high volume.
Every number on this page comes from that run, not from a price list.
Questions people ask
Can I scrape anything?
No. Pages behind a login (e.g. private LinkedIn profiles, paid databases) need the account holder's permission. Sites with explicit robots.txt or ToS prohibitions are grey-area — consult a lawyer for commercial use.
What's a proxy?
A relay server that routes your request through a different IP address. Necessary because many sites rate-limit per IP. Residential proxies (real consumer IPs) are most reliable; datacenter proxies are cheaper but get blocked faster.