Turn web pages into
data your agents can use.
Clean Markdown and structured data from public pages, each with a receipt. It’s also open source!
Clean Markdown and structured data from public pages, each with a receipt. It’s also open source!
YOUR RESULTS
This visit only · cleared when you leave the page
Use it in your agent: Connect MCP · Run it yourself
SELECTED RUN
Hosted · Your computer
One public URL in; Markdown, links, metadata and the fields you ask for out. A page that needs scripts opens in a browser, which on hosted Octocrawl takes an account key.
Hosted · Your computer
The URLs a site lists in its sitemaps and links to from its start page, without reading each page. Near-duplicates fold into one, and what was left out is counted.
Hosted with an account · Your computer
Follow links across a site or one section, within the page, depth and path limits you set, and read each page on the way. Hosted crawls go up to 500 pages; larger ones run on your computer.
Hosted with an account · Your computer
Hand over a list of URLs and get one result for each, failures included, so no page goes missing quietly. Hosted batches take up to 1,000 URLs.
Your computer
Click and scroll before reading, so what sits behind a button or further down a long list comes back too. It runs in Chromium on your computer.
Your computer
Read a page again on a schedule and send each change to your HTTPS webhook, signed when you give it a secret. Timed runs come from the repository’s local service.
The start of the Markdown Octocrawl returned for the Python docs datetime page, its content selected: # `datetime` — Basic date and time types¶ **Source code:** [Lib/datetime.py](https://github.com/python/cpython/tree/3.15/Lib/datetime.py) The `datetime` module supplies classes for manipulating dates and times.
An example of what a map returns: the URLs of a docs site: docs.example.com/ ├─ /guide/ │ ├─ /guide/install/ │ └─ /guide/config/ ├─ /api/ └─ /changelog/ from links, sitemap.xml · left out 2 on other hosts
An example of a crawl: each page it read and how: ✓ /guide/ 200 Markdown ✓ /guide/install/ 200 Markdown ✓ /guide/config/ 200 Markdown × /guide/old/ 404 not found ✓ /guide/faq/ 200 Markdown completed · 4 read · 1 failed
An example of a batch: one result per URL, and the reports: ✓ 0001-example.com.md ✓ 0002-example.org.md × 0003-shop.example blocked by robots.txt report.json results.csv results.jsonl
An example of browser actions run before a read: → click button "Load more" → scroll to the end, 3 times ✓ read 48 items · Markdown and a list
An example of a monitor: two runs, and the change sent to a webhook:
09:00 unchanged
15:00 changed
→ POST https://you.example/hook signed
The first window is a recorded read of the Python docs (17 Oct 2026). The others show what each one returns, on example addresses.
GET STARTED
Try a single page at octocrawl.dev. Connect through MCP, use the REST API, or work with the TypeScript and Python clients.
# Read one page as Markdown: nothing to install, no key npx octocrawl scrape https://example.com --markdown This domain is for use in documentation examples without needing permission. … # List the pages a site links to npx octocrawl map https://example.com # Read a list of pages and keep every result with its evidence npx octocrawl batch https://example.com https://example.org --out results ls results 0001-example.com.md 0002-example.org.md report.json results.csv results.jsonl
Runs on your computer with Node.js 22.13 or later: no server, no key, no daily limit. Advanced reference
# Add hosted Octocrawl to Claude Code: one URL, no key to start claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp # Then ask your agent Use Octocrawl to read https://example.com and give me its Markdown.
Cursor, OpenCode and Codex take the same URL. Connect MCP
// npm install @octocrawl/sdk import { W2L } from '@octocrawl/sdk' const octo = new W2L({ baseUrl: 'https://api.octocrawl.dev' }) const page = await octo.scrape('https://example.com') console.log(page.markdown)
Hosted, keyless within the daily allowance; or http://127.0.0.1:8787 after npx octocrawl serve. Advanced reference
# pip install octocrawl-client from octocrawl_client import W2L octo = W2L(base_url="https://api.octocrawl.dev") page = octo.scrape("https://example.com") print(page["markdown"])
Hosted, keyless within the daily allowance; or http://127.0.0.1:8787 after npx octocrawl serve. Advanced reference
# Hosted: no key within the daily allowance curl -sS -X POST https://api.octocrawl.dev/v1/scrape \ -H 'content-type: application/json' \ -d '{"url":"https://example.com","formats":["markdown"]}'
Add -H 'authorization: Bearer <key>' to spend your account's credits instead. Limits and result states
FAQ
We can’t give legal advice; here is what the preview does. It reads a site’s robots.txt before it fetches a page. If the page is disallowed, or robots.txt can’t be reached (a server error, no answer or a timeout), it stops and reports the page as blocked; a robots.txt that answers with a 4xx status counts as no rules, as RFC 9309 provides. It never signs in, solves a CAPTCHA or gets past a verification page, and it refuses private network addresses. What you do with a page is up to you and the site’s terms.
Public pages anyone can open without signing in. The preview reads them over HTTP without running JavaScript, so a page that only appears in a browser may come back incomplete. It reads pages up to 2 MiB and files such as PDFs up to 5 MiB, and stops after 40 seconds. Amazon.sg product pages (/dp/ASIN) are in Beta. X and Reddit posts often don’t come through: robots rules, sign-in walls or verification pages can stop the preview, and a hosted X or Reddit result hasn’t been verified yet. See Limits and result states.
The preview is a limited public trial: five previews per visitor and 150 for the whole site each UTC day. A request turned down before a preview starts, such as a malformed URL, or localhost or a private IP address typed into it, doesn’t count. Once a preview starts it counts, whatever the result, including a host name that turns out to point to a private network or a page stopped by robots.txt. Octocrawl on your own computer has no daily limit.
Results aren’t saved: your recent runs live in this page and are gone when you leave it. Each preview logs its state and the host of the page, never its path. While you type, the page asks the service for a short hint about the address. That request sends the address in its body; the hosting request log records the request path, not its body. Visit and outcome logs are kept for 30 days. The details are on the Privacy page.
Octocrawl reports what it actually read: a blocked, incomplete or timed-out page is a result with a reason, and checked fields carry their source. The benchmark notes compare the three tools on the same test suite, with the limits of that comparison. For moving off Firecrawl, see the local v1 migration guide: a partial shim, not a v2 drop-in replacement.
Not this preview page: it is for trying Octocrawl in a browser. Call hosted Octocrawl instead: https://api.octocrawl.dev/v1/scrape over REST, or https://mcp.octocrawl.dev/mcp from Claude Code, Cursor, OpenCode or Codex, keyless within a daily allowance and with a key for more. Or run Octocrawl yourself with no limit.
Yes, to start. This preview (five a day) and hosted Octocrawl without a key (20 pages a day per address) need no account. Signing in is free and needs no card: it gives 500 credits a month for the hosted API and MCP, and API keys you make yourself in the dashboard. A page read over HTTP costs 1 credit and one read in a browser 5; a blocked page costs nothing. Credit packs for more are planned but not sold yet. Octocrawl is open source under the AGPL-3.0, and running it yourself costs nothing but your own machine.