Browse docs: Connect MCP

Octocrawl / Get started

Connect Octocrawl MCP

Connect your AI agent to Octocrawl at https://mcp.octocrawl.dev/mcp. Hosted MCP supports scrape for page content and map for site URLs. An account key enables browser-backed scraping and the Amazon.sg scrape_product tool, subject to service limits. With an account (a key or sign-in), it also runs crawl and batch jobs. Without a key, hosted access reads over HTTP within a daily allowance.

For Monitor workflows, and for crawls and batches larger than hosted takes, start the local API with npx octocrawl serve and connect it through the @octocrawl/mcp package. Scheduled monitoring also requires local workers. Successful scrape results include an Evidence Record describing the capture; inspect any separate verification result before relying on extracted fields.

Octocrawl’s official website is octocrawl.dev, and its official source repository is github.com/77777R7/Octocrawl. See the introduction for REST, SDK and MCP entry points.

To see what map finds before connecting a client, try the free online Link Extractor. The guide to finding all pages on a website explains which URLs a sitemap, crawler or web archive can find and how to check the list.

Connect to hosted Octocrawl

One URL, no account: https://mcp.octocrawl.dev/mcp. Scrape and map over HTTP, a daily allowance per address; a key for more.

Then send a first task

Add hosted Octocrawl to Codex as a Streamable HTTP server.

codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp

Run codex mcp list, open a new Codex task, then use /mcp to check that the scrape tool is available.

Verified 2026-10-13Codex setup docs ↗

Using another MCP client?

https://mcp.octocrawl.dev/mcp
Its Streamable HTTP URL is https://mcp.octocrawl.dev/mcp; a key goes in an Authorization: Bearer header, never in the URL.

Add a key for more

A key spends your account’s credits instead of the keyless daily allowance and opens the browser lane (pages that only appear in a browser, and scrape_product for Amazon.sg). Sign in to make one: an account gets 500 credits a month, a page read over HTTP costs 1 and a page read in a browser 5. The key goes in an Authorization: Bearer header, never in the URL:

bash
# Claude Code
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp --header "Authorization: Bearer oc_…"
# Codex: the token is read from an environment variable
export OCTOCRAWL_API_KEY=oc_…
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp --bearer-token-env-var OCTOCRAWL_API_KEY
json
{ "mcpServers": { "octocrawl": { "url": "https://mcp.octocrawl.dev/mcp", "headers": { "Authorization": "Bearer oc_…" } } } }

Cursor takes that block in .cursor/mcp.json ("Bearer ${env:OCTOCRAWL_API_KEY}" reads it from the environment); OpenCode takes the same headers object inside its remote entry. Over REST, curl -H 'authorization: Bearer oc_…' https://api.octocrawl.dev/v1/scrape.

Clients can also sign in instead of sending a key.

Sign in instead of a key

https://mcp.octocrawl.dev/mcp-oauth is the same server for clients that sign in with OAuth instead of sending a key. The client opens an Octocrawl page where you sign in and choose Authorize. Its calls then spend your account’s credits, as a key’s do, and show under the client’s name in Activity. Each connection is listed under MCP connections on API keys, where you can disconnect it; it also ends after 90 days unused. An account holds up to ten connections, besides its five keys.

Use one URL per client: /mcp for keyless use or a key, /mcp-oauth for sign-in. The setups below were checked on 2026-10-16 and 2026-10-17: each client signed in, ran a scrape on the account’s credits, and was refused after the connection was disconnected (record). Another client that follows the MCP authorization spec should sign in the same way, but has not been checked here.

Claude Code

bash
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp-oauth

Then run /mcp in Claude Code, select octocrawl and sign in in the browser it opens (claude mcp login octocrawl does the same from a terminal). The Octocrawl plugin sets up the same server: /plugin install octocrawl --marketplace 77777R7/Octocrawl, then sign in from /mcp the same way. Clear authentication in /mcp signs it out. The key setup above keeps working on /mcp. (Claude Code’s guide)

If you added the keyless server first (the home page’s claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp), it has the same name: run claude mcp remove octocrawl before the command above. Codex: codex mcp remove octocrawl; Cursor, OpenCode and Cline: replace the octocrawl entry in the JSON.

Checked with Claude Code 2.1.301 on 2026-10-16.

Cursor

json
{ "mcpServers": { "octocrawl": { "url": "https://mcp.octocrawl.dev/mcp-oauth" } } }

Merge this into .cursor/mcp.json (or ~/.cursor/mcp.json for every project) with no headers, and sign in when Cursor asks for it. The key setup above keeps working on /mcp. (Cursor’s guide)

Checked with Cursor 3.25.6 on 2026-10-16.

Codex

bash
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp-oauth

Codex sees that the server signs in with OAuth and opens a browser to sign in as part of add. codex mcp login octocrawl signs in again later. codex mcp logout octocrawl removes the sign-in from that machine; Disconnect on API keys ends the connection itself. The key setup above keeps working on /mcp. (Codex’s guide)

Checked with Codex CLI 0.162.1 on 2026-10-17.

OpenCode

json
{ "mcp": { "octocrawl": { "type": "remote", "url": "https://mcp.octocrawl.dev/mcp-oauth", "enabled": true } } }

Merge this into the mcp object of opencode.json with no headers, then run opencode mcp auth octocrawl and sign in in the browser it opens. opencode mcp list shows whether it is connected. opencode mcp logout octocrawl removes the sign-in from that machine; Disconnect on API keys ends the connection itself. The key setup above keeps working on /mcp. (OpenCode’s guide)

Checked with OpenCode 1.18.26 on 2026-10-17.

Send your first task

text
Use Octocrawl's scrape tool on https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview with formats ["markdown"]. Show the final URL, the HTTP status and the robots.txt decision from its evidenceRecord, then the first heading of the Markdown.

The answer carries the page’s Markdown and an Evidence Record: final URL, fetch time, HTTP status, the robots.txt decision and hashes of what was read. A page that could not be read comes back as a result whose status says so (blocked with a blockReason, failed with a failureReason such as timeout, or empty_verified), not as an invented page. Over the daily allowance the tool answers with the time until it resets (00:00 UTC) and how to get a key. For a walk-through of a real task in Claude Code, from finding the page with map to citing its Evidence Record, see Claude Code web scraping.

Check the first result before continuing

A connected server and a listed tool establish the connection. To assess the task, inspect the actual tool result:

  • Compare the returned Markdown with the heading or content you asked for. HTTP 200 and nonempty text can also describe a navigation shell or an error page.
  • Keep the fetch status separate from verification.status. not_requested means no task contract was checked. passed means the configured checks passed; it does not establish independent field accuracy.
  • Inspect readiness when present. not_loaded, not_extracted and not_served need different next steps. A missing readiness field is unknown.
  • Keep incomplete results and failures visible. Do not convert missing content to a zero value or retry paid access automatically.

The tool descriptions keep each option to a line or two. Every option’s full behaviour, with examples, is in the scrape options reference, and crawl and batch in Batch, crawl and map.

If the client lists no tools, check the selected server’s connection error. If the tool returns a quota or access refusal, follow that reason. Changing the MCP client does not establish that the target content can be read. See result states and the Evidence Record reference.

Crawl and batch jobs

A connection with an account (a key on /mcp, or sign-in on /mcp-oauth) also lists the job tools:

  • crawl and batch_scrape start a job;
  • get_crawl and get_batch report its status;
  • get_crawl_pages, get_batch_items, get_crawl_errors and get_batch_errors page through what it read;
  • wait_crawl and wait_batch wait for it;
  • cancel_crawl and cancel_batch stop it.

Without an account the tools are not listed, and a call says it needs one. A job runs on Octocrawl’s servers in slices, so it keeps going after the client disconnects:

  • A crawl reads at most 500 pages, and a batch at most 1,000 URLs.
  • Each page costs credits as a scrape does: 1 over HTTP and 5 in a browser.
  • An account runs two jobs at a time, with up to ten more waiting. A job reads 8 pages at once, 2 of them in a browser, across up to 100 sites a slice.
  • Results are kept 7 days after the job ends. Once it has ended, get_crawl, get_batch and the waits carry export: a link that downloads every page as one gzip JSONL file, good for 24 hours, and asking again gives a fresh one, so a thousand pages need not pass through the chat. A job that runs out of credits is not lost: it waits, its status says so and where to ask for more, and it goes on by itself when credits come back.
  • A wait returns after at most 55 seconds (timeoutMs, default 30 seconds; a wait on a server on your computer takes up to 120 seconds, but Claude Code ends an HTTP MCP request that has sent nothing for 60 seconds) with the status so far. Call it again, or check later with get_crawl or get_batch.
text
Use Octocrawl's batch_scrape tool on https://example.com/ and https://books.toscrape.com/ with fastMode true. Wait for it, then list its items with each URL, status and first heading.

Claude Code, Codex and OpenCode each ran a job end to end through these tools on 2026-10-20 (record). Codex asks before it runs batch_scrape, since starting a job is not read-only. In codex exec, which cannot ask, set default_tools_approval_mode = "approve" on the server’s entry.

Over REST the same jobs are POST https://api.octocrawl.dev/v1/batches and POST /v1/crawl with a key. GET /v1/jobs lists your jobs, newest first. Signed in on octocrawl.dev, Jobs in the dashboard shows each job’s progress while it runs, what it cost, and an export of its results once it ends. maxCredits caps what a job spends, on the tools and over REST. webhook is REST only: it posts the job’s end to an https address, signed with a secret the start returns. The hosted API reference has the signature and a verification example.

Every signed-in account can run hosted jobs. If they are ever paused, a start answers 503 jobs_paused with Retry-After, and the same job runs on your computer meanwhile.

What hosted Octocrawl does not do

  • The Monitor tools, and crawls over 500 pages or batches over 1,000 URLs: on your computer (below), where they run without these limits. The hosted answer names this.
  • Proxies, CAPTCHA solving and stealth: not offered on the hosted service. A page that blocks a plain request comes back as blocked with the reason; the access grant for those routes works on your own server.
  • Private network addresses, robots.txt overrides and saved logins: refused in hosted mode; robots.txt is obeyed for every URL.

The allowances and what is recorded are on Limits and Privacy.

Run it on your computer

With Node.js 22.13 or later, start the API in any terminal and keep it running:

bash
npx octocrawl serve

It listens on http://127.0.0.1:8787, on this computer only, with every tool: scrape, map, crawl, batch, saved logins, handoff to your own Chrome, the Amazon.sg product tools and the Monitor and delivery tools, and no daily limit. Pages are read over HTTP; for pages that only appear in a browser, run npx playwright install chromium once and the browser lane is used too.

A page behind your own sign-in can often be read here too, with no new login for a site you are already signed in to in Chrome. The agent is told to ask you first, then reads the page in that Chrome, or with a login you save with npx octocrawl login import <site> and save again when it expires. A site can still refuse. Saved logins stay on this computer. Scrape a website with login shows each step.

Add the server to your client

The server runs on this computer as npx -y @octocrawl/mcp and talks to the API from the step above.

Start the API first

Add the stdio server to Codex.

codex mcp add octocrawl -- npx -y @octocrawl/mcp

Run codex mcp list, open a new Codex task, then use /mcp to check that the scrape tool is available.

Follows the Codex docs; not run here yetCodex setup docs ↗

Using another MCP client?

npx -y @octocrawl/mcp
Register the stdio command npx -y @octocrawl/mcp. It calls the API at http://127.0.0.1:8787; pass --base-url to use another.

If the client reports no tools, check that npx octocrawl serve is still running; if a call fails to connect, the API is not on 127.0.0.1:8787 (pass --base-url to the server command). The repository also has a managed local service for macOS (npm run first-use:local) that adds the Monitor scheduler and the delivery worker behind http://127.0.0.1:8791/mcp; the Monitor guides use it. Then continue with Monitor → HTTPS Webhook or Amazon.sg product JSON.

Self-host for others

npx octocrawl serve --hosted --token <token> serves other machines behind a bearer token, with private addresses, robots overrides, saved logins, handoff and non-HTTPS webhooks refused, as the hosted service does. Point npx -y @octocrawl/mcp --base-url https://your.host --token <token> at it, or any client’s remote URL at your own /mcp once you put one in front of it; the advanced reference has the details.