# Turn web pages into data your agents can use.

> Clean Markdown and structured data from public pages, each with a receipt. It’s also open source!

The page: /

## What it does

- **Scrape a page.** (Hosted · Your computer): One public URL in; Markdown, links, metadata and the fields you ask for out. A page that needs scripts opens in a browser, which on hosted Octocrawl takes an account key.
- **Map a site.** (Hosted · Your computer): The URLs a site lists in its sitemaps and links to from its start page, without reading each page. Near-duplicates fold into one, and what was left out is counted.
- **Crawl a site.** (Hosted with an account · Your computer): Follow links across a site or one section, within the page, depth and path limits you set, and read each page on the way. Hosted crawls go up to 500 pages; larger ones run on your computer.
- **Scrape a batch.** (Hosted with an account · Your computer): Hand over a list of URLs and get one result for each, failures included, so no page goes missing quietly. Hosted batches take up to 1,000 URLs.
- **Act in a browser.** (Your computer): Click and scroll before reading, so what sits behind a button or further down a long list comes back too. It runs in Chromium on your computer.
- **Monitor a page.** (Your computer): Read a page again on a schedule and send each change to your HTTPS webhook, signed when you give it a secret. Timed runs come from the repository’s local service.

## Get started

**CLI.** Runs on your computer with Node.js 22.13 or later: no server, no key, no daily limit. [Advanced reference](/docs/reference/)

```bash
# Read one page as Markdown: nothing to install, no key
npx octocrawl scrape https://example.com --markdown

# List the pages a site links to
npx octocrawl map https://example.com

# Read a list of pages and keep every result with its evidence
npx octocrawl batch https://example.com https://example.org --out results
ls results
```

**MCP.** Cursor, OpenCode and Codex take the same URL. [Connect MCP](/docs/connect-mcp/)

```bash
# Add hosted Octocrawl to Claude Code: one URL, no key to start
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

# Then ask your agent
Use Octocrawl to read https://example.com and give me its Markdown.
```

**TypeScript.** Hosted, keyless within the daily allowance; or `http://127.0.0.1:8787` after `npx octocrawl serve`. [Advanced reference](/docs/reference/)

```ts
// npm install @octocrawl/sdk
import { W2L } from '@octocrawl/sdk'

const octo = new W2L({ baseUrl: 'https://api.octocrawl.dev' })
const page = await octo.scrape('https://example.com')
console.log(page.markdown)
```

**Python.** Hosted, keyless within the daily allowance; or `http://127.0.0.1:8787` after `npx octocrawl serve`. [Advanced reference](/docs/reference/)

```python
# pip install octocrawl-client
from octocrawl_client import W2L

octo = W2L(base_url="https://api.octocrawl.dev")
page = octo.scrape("https://example.com")
print(page["markdown"])
```

**REST.** Add `-H 'authorization: Bearer <key>'` to spend your account's credits instead. [Limits and result states](/docs/limits/)

```bash
# Hosted: no key within the daily allowance
curl -sS -X POST https://api.octocrawl.dev/v1/scrape \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}'
```

- **In your agent**: Add the MCP server to Claude Code, Cursor or Codex, and your agent reads pages as Markdown itself. [Connect MCP](/docs/connect-mcp/)
- **In the Playground**: Scrape a page or map a site in your browser, on your account’s credits, with a browser for pages that need one. [Open the Playground](/dashboard/scrape/)
- **From your code**: Call the REST API or the Node and Python SDKs, or run the same scrape on your own computer with the CLI. [First call](/docs/scrape/)
- **Need more, or a team**: Higher limits, a team, or something Octocrawl does not do yet: tell us what you would collect. [hello@octocrawl.dev](mailto:hello@octocrawl.dev)

## Questions

### Does Octocrawl respect robots.txt? Is this legal?

We can’t give legal advice; here is what the preview does. It reads a site’s robots.txt before it fetches a page. If the page is disallowed, or robots.txt can’t be reached (a server error, no answer or a timeout), it stops and reports the page as blocked; a robots.txt that answers with a 4xx status counts as no rules, as RFC 9309 provides. It never signs in, solves a CAPTCHA or gets past a verification page, and it refuses private network addresses. What you do with a page is up to you and the site’s terms.

### Which sites work?

Public pages anyone can open without signing in. The preview reads them over HTTP without running JavaScript, so a page that only appears in a browser may come back incomplete. It reads pages up to 2 MiB and files such as PDFs up to 5 MiB, and stops after 40 seconds. Amazon.sg product pages (`/dp/ASIN`) are in Beta. X and Reddit posts often don’t come through: robots rules, sign-in walls or verification pages can stop the preview, and a hosted X or Reddit result hasn’t been verified yet. See [Limits and result states](/docs/limits/).

### Why only five previews a day?

The preview is a limited public trial: five previews per visitor and 150 for the whole site each UTC day. A request turned down before a preview starts, such as a malformed URL, or localhost or a private IP address typed into it, doesn’t count. Once a preview starts it counts, whatever the result, including a host name that turns out to point to a private network or a page stopped by robots.txt. Octocrawl on your own computer has no daily limit.

### Do you store the URLs I submit, or the results?

Results aren’t saved: your recent runs live in this page and are gone when you leave it. Each preview logs its state and the host of the page, never its path. While you type, the page asks the service for a short hint about the address. That request sends the address in its body; the hosting request log records the request path, not its body. Visit and outcome logs are kept for 30 days. The details are on the [Privacy](/docs/privacy/) page.

### How is Octocrawl different from Firecrawl or Crawl4AI?

Octocrawl reports what it actually read: a blocked, incomplete or timed-out page is a result with a reason, and checked fields carry their source. The [benchmark notes](https://github.com/77777R7/Octocrawl/blob/main/docs/benchmark-gate.md) compare the three tools on the same test suite, with the limits of that comparison. For moving off Firecrawl, see the [local v1 migration guide](/blog/firecrawl-v1-migration/): a partial shim, not a v2 drop-in replacement.

### Can I call it from a script or an agent?

Not this preview page: it is for trying Octocrawl in a browser. Call [hosted Octocrawl](/docs/connect-mcp/) instead: `https://api.octocrawl.dev/v1/scrape` over REST, or `https://mcp.octocrawl.dev/mcp` from Claude Code, Cursor, OpenCode or Codex, keyless within a daily allowance and with a key for more. Or run Octocrawl yourself with no limit.

### Is it free?

Yes, to start. This preview (five a day) and hosted Octocrawl without a key (20 pages a day per address) need no account. [Signing in](/signin/) is free and needs no card: it gives 500 credits a month for the hosted API and MCP, and API keys you make yourself in the dashboard. A page read over HTTP costs 1 credit and one read in a browser 5; a blocked page costs nothing. Credit packs for more are planned but not sold yet. Octocrawl is open source under the AGPL-3.0, and running it yourself costs nothing but your own machine.

Free tools: [URL to Markdown](/tools/url-to-markdown/), [Link Extractor](/tools/link-extractor/), [Website to Text](/tools/website-to-text/). Documentation: /docs/.
