# Scrape Any Website

Your employees can read the full content of any public webpage. Just give them a URL and they'll extract clean, readable text.

## TL;DR

Every employee can read the full content of any public webpage. Paste a URL and they extract clean, readable text with no ads or clutter. No setup required.

## How It Works

The employee fetches the page, strips boilerplate (headers, footers, ads, navigation), and returns clean text you can work with.

## What It Can Do

| Capability | Example |
|-----------|---------|
| **Read articles** | "Read this blog post and summarize the key points" |
| **Extract data** | "Pull the pricing from this page" |
| **Research competitors** | "Read their features page and compare to ours" |
| **Process documentation** | "Read this API doc and write me a quick-start guide" |
| **Digest long content** | "Read this 5,000-word report and give me the top 3 takeaways" |

## How to Set It Up

**Nothing to do.** Web Scraper is enabled for every employee at hire.

You can ask the employee to manage their own tools, or do it manually:

1. Select the employee
2. Click the **Tools** tab
3. Find "Web Scraper" in the Actions section

| Action | What it does | How |
|--------|-------------|-----|
| **Enable / Disable** | Controls whether the employee can use this tool | Toggle the switch |
| **Tool Rules** | Custom instructions that guide how the employee uses this specific tool, e.g. "always extract tables as markdown" or "skip navigation content" | Expand the tool, then write your rules in the text field |
| **Delete** | Permanently removes the tool from the employee | Click the delete button |

## Tips & Tricks

- **Give the direct URL.** Not a search results page or redirect link
- **Ask for specific extraction.** "Read this page and pull out only the pricing table" works better than "read this"
- **Combine with search.** "Search for X, then read the top 3 results" lets the employee use both tools together
- **Say when a site is a JavaScript-heavy app.** If you know the page is a single-page app or dashboard, mention that and the employee will render it with a headless browser right away instead of trying the fast path first

## Behind the Scene

| | |
|---|---|
| **How it works** | A staged extraction pipeline. Each stage only runs if the one before it comes up empty or thin |

| Stage | Method | Handles |
|-------|--------|---------|
| **1. Fast text extraction** | Lightweight fetch, no browser | Most pages: articles, docs, blogs, news |
| **2. Browser User-Agent retry** | Same fetch, browser-like headers | Sites that block bots but don't require JavaScript |
| **3. Headless browser rendering** | A real headless browser we run ourselves | JavaScript-heavy single-page apps and dashboards. No Desktop companion app needed |
| **4. Desktop companion browser** | Your own browser, via the Desktop companion app | The rare page that defeats stages 1-3, only if you have the companion app connected |

Most pages resolve on stage 1 in under a second. Stage 3 handles the majority of JavaScript-heavy sites automatically, no extra setup required.

### Web Scraper vs Web Search vs Browser Controller

| | **Web Scraper** | **Web Search** | **Browser Controller** |
|---|---|---|---|
| **Purpose** | Read full content from a specific URL | Find information via search engine | Control a real browser. Navigate, click, fill forms |
| **Input** | A URL you already have | A question or topic | Instructions like "go to this site and..." |
| **Output** | Full page text, clean, no ads or nav | Titles, snippets, and source links | Screenshots, extracted data, completed actions |
| **Best for** | "Read this article for me" | "What's happening with X?" | "Log into this site and download the report" |
| **When to use** | You know the exact page | You don't know where to look | You need to interact with a page |
| **Requires** | Nothing, built in | Nothing, built in | Desktop companion app |

The employee often combines these automatically, searching first to find URLs, then reading the best results in full.

## What It Costs

| | |
|---|---|
| **Cost** | Free. Reading a page costs no credits |
| **Exception** | If every other stage fails and the employee falls back to your Desktop companion browser, that fallback is billed as Browser Controller usage, not as Web Scraper |
| **Rate limits** | None from our side, but the target website may block rapid consecutive requests |
| **Truncation** | Extremely long pages are truncated with a notice rather than cut silently. In practice, almost no real page hits the limit |

## Is It Safe

- **Public pages only.** The scraper can only access publicly available content. No login-protected or paywalled content is accessible
- **Results in your chat.** Scraped content is summarized in the employee's response, which is saved in your conversation history like any other message

## Good to Know

- **Clean text only.** The scraper strips all HTML, scripts, styles, and navigation. You get readable text, not raw markup
- **No login-protected content.** The scraper can only access publicly available pages. For authenticated content, use Browser Controller with the Desktop companion app
- **JavaScript-heavy sites work without the Desktop app.** Single-page apps and dashboards are rendered by a headless browser we run ourselves. The Desktop companion app is only needed for the rare page that defeats every other stage, or for content behind a login

## Frequently Asked Questions

**Q: Can the employee read PDFs from a URL?**
A: The scraper is optimized for HTML web pages. For PDFs, upload the file directly to the employee's chat. They can read uploaded documents natively.

**Q: What happens with pages behind a login?**
A: The scraper can only access public pages. For authenticated content, use Browser Controller. It uses your actual browser session, so any site you're logged into is accessible.

**Q: Does the employee cache scraped pages?**
A: No. Every scrape fetches the live page, so you always get current content.

**Q: Can I scrape multiple pages at once?**
A: Yes. Give the employee a list of URLs and they'll read each one. They may do this automatically when combining Web Search and Web Scraper.

**Q: Why is the content truncated?**
A: Only extremely long pages get truncated, and it happens with a clear notice rather than silently. If you need the full page, ask the employee to focus on a specific section or split the request.