# What is Web Scraping?

Also called screen scraping, data scraping.

Web scraping is the automated extraction of data from web pages, turning content meant for human reading into structured records. A scraper fetches pages, parses the markup, selects the relevant elements, and stores the result. It is a long standing practice whose legality and acceptability depend heavily on what is collected, how it is collected, and what is done with it.

A basic scraper issues HTTP requests and parses the returned markup. Pages that assemble themselves in the browser require a headless browser instead. Beyond fetching, real scrapers need politeness controls, retry and backoff logic, deduplication, and a plan for the inevitable day the site's structure changes and every selector stops matching at once.

Language models changed the parsing half of the problem. Instead of brittle selectors, a model can read messy page text and return structured fields, which survives layout changes far better than hand written rules. The fetching half is unchanged, and that is where the cost, the blocking, and the ethics live. Models can also be quietly wrong, so sampled verification matters.

Norms and law differ. Site terms of service, the robots exclusion file, copyright, database rights, and data protection law can each apply, and the rules vary by jurisdiction and by case. Scraping personal data is treated far more strictly than scraping public prices. This area is genuinely contested, so a significant project deserves legal review rather than a rule of thumb.

Independent of law, good citizenship is cheap. Identify your crawler honestly, respect the exclusion file, limit concurrency, crawl during quiet hours, cache aggressively, and prefer an official API or bulk export wherever one exists. Aggressive scraping is also self defeating, since it is the fastest route to being blocked entirely.

## Key points

- Automated extraction of structured data from web pages.
- JavaScript heavy sites need a headless browser, not a plain fetch.
- Models parse messy pages well, but fetching remains the hard part.
- Legality is contested and depends on data, method, and jurisdiction.
- Prefer an official API or bulk export where one exists.

## In practice

A researcher wants opening hours for two hundred public libraries. A script fetches each library's contact page at one request every few seconds, passes the page text to a model, and asks for the hours as structured JSON with a null value where they are not stated. Results go into a spreadsheet, and twenty of them are checked by hand against the original pages before the data set is used.

## Related terms

- [Headless Browser](/en/glossary/headless-browser)
- [Browser Automation](/en/glossary/browser-automation)
- [robots.txt for AI Agents](/en/glossary/robots-txt-for-ai-agents)
- [Rate Limiting](/en/glossary/rate-limiting)
- [Structured Output](/en/glossary/structured-output)

[Back to the AI Glossary](/en/glossary)