Turn web pages into
data your agents can use.

Clean Markdown and structured data from public pages, each with a receipt. It’s also open source!

Start for freeConnect your agent

5 free previews a day · · Use it from your agent

Get code

Use it in your code or agent

The same page, format and options, through hosted Octocrawl (api.octocrawl.dev and mcp.octocrawl.dev: no key within a daily allowance, a key for more and the browser lane) or on your own computer with no limit (npx octocrawl serve, npx -y @octocrawl/mcp). A browser-lane run may differ from this HTTP preview.

    See what it does

    What it does

    From one page to a whole site. One tool.

    Scrape a page, map a site, crawl a section or watch it for changes, all through one API. No crawler to build, no HTML to clean.

    • Scrape
    • Map
    • Crawl
    • Batch
    • Browser actions
    • Monitor
    1. Hosted · Your computer

      One public URL in; Markdown, links, metadata and the fields you ask for out. A page that needs scripts opens in a browser, which on hosted Octocrawl takes an account key.

    2. Hosted · Your computer

      The URLs a site lists in its sitemaps and links to from its start page, without reading each page. Near-duplicates fold into one, and what was left out is counted.

    3. Hosted with an account · Your computer

      Follow links across a site or one section, within the page, depth and path limits you set, and read each page on the way. Hosted crawls go up to 500 pages; larger ones run on your computer.

    4. Hosted with an account · Your computer

      Hand over a list of URLs and get one result for each, failures included, so no page goes missing quietly. Hosted batches take up to 1,000 URLs.

    5. Your computer

      Click and scroll before reading, so what sits behind a button or further down a long list comes back too. It runs in Chromium on your computer.

    6. Your computer

      Read a page again on a schedule and send each change to your HTTPS webhook, signed when you give it a secret. Timed runs come from the repository’s local service.

    The start of the Markdown Octocrawl returned for the Python docs datetime page, its content selected:
    # `datetime` — Basic date and time types¶
    
    **Source code:** [Lib/datetime.py](https://github.com/python/cpython/tree/3.15/Lib/datetime.py)
    
    The `datetime` module supplies classes for manipulating dates and times.
    An example of what a map returns: the URLs of a docs site:
    docs.example.com/
    ├─ /guide/
    │  ├─ /guide/install/
    │  └─ /guide/config/
    ├─ /api/
    └─ /changelog/
    
    from links, sitemap.xml · left out 2 on other hosts
    An example of a crawl: each page it read and how:
    ✓ /guide/          200  Markdown
    ✓ /guide/install/  200  Markdown
    ✓ /guide/config/   200  Markdown
    × /guide/old/      404  not found
    ✓ /guide/faq/      200  Markdown
    
    completed · 4 read · 1 failed
    An example of a batch: one result per URL, and the reports:
    ✓ 0001-example.com.md
    ✓ 0002-example.org.md
    × 0003-shop.example  blocked by robots.txt
    
    report.json  results.csv  results.jsonl
    An example of browser actions run before a read:
    → click   button "Load more"
    → scroll  to the end, 3 times
    ✓ read    48 items · Markdown and a list
    An example of a monitor: two runs, and the change sent to a webhook:
    09:00  unchanged
    15:00  changed
           → POST https://you.example/hook  signed

    The first window is a recorded read of the Python docs (17 Oct 2026). The others show what each one returns, on example addresses.

    GET STARTED

    Start in your browser.
    Build with your agent or code.

    Try a single page at octocrawl.dev. Connect through MCP, use the REST API, or work with the TypeScript and Python clients.

    terminal
    # Read one page as Markdown: nothing to install, no key
    npx octocrawl scrape https://example.com --markdown
    This domain is for use in documentation examples without needing permission. …
    
    # List the pages a site links to
    npx octocrawl map https://example.com
    
    # Read a list of pages and keep every result with its evidence
    npx octocrawl batch https://example.com https://example.org --out results
    ls results
    0001-example.com.md  0002-example.org.md  report.json  results.csv  results.jsonl

    Runs on your computer with Node.js 22.13 or later: no server, no key, no daily limit. Advanced reference

    # Add hosted Octocrawl to Claude Code: one URL, no key to start
    claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
    
    # Then ask your agent
    Use Octocrawl to read https://example.com and give me its Markdown.

    Cursor, OpenCode and Codex take the same URL. Connect MCP

    // npm install @octocrawl/sdk
    import { W2L } from '@octocrawl/sdk'
    
    const octo = new W2L({ baseUrl: 'https://api.octocrawl.dev' })
    const page = await octo.scrape('https://example.com')
    console.log(page.markdown)

    Hosted, keyless within the daily allowance; or http://127.0.0.1:8787 after npx octocrawl serve. Advanced reference

    # pip install octocrawl-client
    from octocrawl_client import W2L
    
    octo = W2L(base_url="https://api.octocrawl.dev")
    page = octo.scrape("https://example.com")
    print(page["markdown"])

    Hosted, keyless within the daily allowance; or http://127.0.0.1:8787 after npx octocrawl serve. Advanced reference

    # Hosted: no key within the daily allowance
    curl -sS -X POST https://api.octocrawl.dev/v1/scrape \
      -H 'content-type: application/json' \
      -d '{"url":"https://example.com","formats":["markdown"]}'

    Add -H 'authorization: Bearer <key>' to spend your account's credits instead. Limits and result states

    FAQ

    Questions,
    answered plainly.

    Does Octocrawl respect robots.txt? Is this legal?

    We can’t give legal advice; here is what the preview does. It reads a site’s robots.txt before it fetches a page. If the page is disallowed, or robots.txt can’t be reached (a server error, no answer or a timeout), it stops and reports the page as blocked; a robots.txt that answers with a 4xx status counts as no rules, as RFC 9309 provides. It never signs in, solves a CAPTCHA or gets past a verification page, and it refuses private network addresses. What you do with a page is up to you and the site’s terms.

    Which sites work?

    Public pages anyone can open without signing in. The preview reads them over HTTP without running JavaScript, so a page that only appears in a browser may come back incomplete. It reads pages up to 2 MiB and files such as PDFs up to 5 MiB, and stops after 40 seconds. Amazon.sg product pages (/dp/ASIN) are in Beta. X and Reddit posts often don’t come through: robots rules, sign-in walls or verification pages can stop the preview, and a hosted X or Reddit result hasn’t been verified yet. See Limits and result states.

    Why only five previews a day?

    The preview is a limited public trial: five previews per visitor and 150 for the whole site each UTC day. A request turned down before a preview starts, such as a malformed URL, or localhost or a private IP address typed into it, doesn’t count. Once a preview starts it counts, whatever the result, including a host name that turns out to point to a private network or a page stopped by robots.txt. Octocrawl on your own computer has no daily limit.

    Do you store the URLs I submit, or the results?

    Results aren’t saved: your recent runs live in this page and are gone when you leave it. Each preview logs its state and the host of the page, never its path. While you type, the page asks the service for a short hint about the address. That request sends the address in its body; the hosting request log records the request path, not its body. Visit and outcome logs are kept for 30 days. The details are on the Privacy page.

    How is Octocrawl different from Firecrawl or Crawl4AI?

    Octocrawl reports what it actually read: a blocked, incomplete or timed-out page is a result with a reason, and checked fields carry their source. The benchmark notes compare the three tools on the same test suite, with the limits of that comparison. For moving off Firecrawl, see the local v1 migration guide: a partial shim, not a v2 drop-in replacement.

    Can I call it from a script or an agent?

    Not this preview page: it is for trying Octocrawl in a browser. Call hosted Octocrawl instead: https://api.octocrawl.dev/v1/scrape over REST, or https://mcp.octocrawl.dev/mcp from Claude Code, Cursor, OpenCode or Codex, keyless within a daily allowance and with a key for more. Or run Octocrawl yourself with no limit.

    Is it free?

    Yes, to start. This preview (five a day) and hosted Octocrawl without a key (20 pages a day per address) need no account. Signing in is free and needs no card: it gives 500 credits a month for the hosted API and MCP, and API keys you make yourself in the dashboard. A page read over HTTP costs 1 credit and one read in a browser 5; a blocked page costs nothing. Credit packs for more are planned but not sold yet. Octocrawl is open source under the AGPL-3.0, and running it yourself costs nothing but your own machine.