Connect Octocrawl MCP
Connect your AI agent to Octocrawl at https://mcp.octocrawl.dev/mcp. Hosted MCP supports scrape for page content and map for site URLs. An account key enables browser-backed scraping and the Amazon.sg scrape_product tool, subject to service limits. With an account (a key or sign-in), it also runs crawl and batch jobs. Without a key, hosted access reads over HTTP within a daily allowance.
For Monitor workflows, and for crawls and batches larger than hosted takes, start the local API with npx octocrawl serve and connect it through the @octocrawl/mcp package. Scheduled monitoring also requires local workers. Successful scrape results include an Evidence Record describing the capture; inspect any separate verification result before relying on extracted fields.
Octocrawl’s official website is octocrawl.dev, and its official source repository is github.com/77777R7/Octocrawl. See the introduction for REST, SDK and MCP entry points.
To see what map finds before connecting a client, try the free online Link Extractor. The guide to finding all pages on a website explains which URLs a sitemap, crawler or web archive can find and how to check the list.
Connect to hosted Octocrawl
One URL, no account: https://mcp.octocrawl.dev/mcp. Scrape and map over HTTP, a daily allowance per address; a key for more.
Add hosted Octocrawl to Codex as a Streamable HTTP server.
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcpRun codex mcp list, open a new Codex task, then use /mcp to check that the scrape tool is available.
Add hosted Octocrawl to Claude Code over HTTP. Run this in the project where you use Claude Code; --scope user adds it everywhere.
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcpRun claude mcp list: octocrawl shows as Connected. In Claude Code, /mcp lists scrape and map (scrape_product is listed only to a key with the browser lane); then send the first task below.
Merge this server into your project .cursor/mcp.json (or your user-level ~/.cursor/mcp.json). Keep existing servers.
{
"mcpServers": {
"octocrawl": {
"url": "https://mcp.octocrawl.dev/mcp"
}
}
}Approve the server when Cursor asks, then check its tools in Cursor Settings → MCP. In Cursor CLI, cursor-agent mcp enable octocrawl approves it and cursor-agent mcp list-tools octocrawl lists scrape and map.
For OpenCode 1.x, merge this entry into the mcp object in opencode.json. Keep your existing settings and servers.
{
"mcp": {
"octocrawl": {
"type": "remote",
"url": "https://mcp.octocrawl.dev/mcp",
"enabled": true
}
}
}Run opencode mcp list: octocrawl shows as connected. Then ask for the first task below.
Using another MCP client?
https://mcp.octocrawl.dev/mcpAdd a key for more
A key spends your account’s credits instead of the keyless daily allowance and opens the browser lane (pages that only appear in a browser, and scrape_product for Amazon.sg). Sign in to make one: an account gets 500 credits a month, a page read over HTTP costs 1 and a page read in a browser 5. The key goes in an Authorization: Bearer header, never in the URL:
# Claude Code
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp --header "Authorization: Bearer oc_…"
# Codex: the token is read from an environment variable
export OCTOCRAWL_API_KEY=oc_…
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp --bearer-token-env-var OCTOCRAWL_API_KEY{ "mcpServers": { "octocrawl": { "url": "https://mcp.octocrawl.dev/mcp", "headers": { "Authorization": "Bearer oc_…" } } } }Cursor takes that block in .cursor/mcp.json ("Bearer ${env:OCTOCRAWL_API_KEY}" reads it from the environment); OpenCode takes the same headers object inside its remote entry. Over REST, curl -H 'authorization: Bearer oc_…' https://api.octocrawl.dev/v1/scrape.
Clients can also sign in instead of sending a key.
Sign in instead of a key
https://mcp.octocrawl.dev/mcp-oauth is the same server for clients that sign in with OAuth instead of sending a key. The client opens an Octocrawl page where you sign in and choose Authorize. Its calls then spend your account’s credits, as a key’s do, and show under the client’s name in Activity. Each connection is listed under MCP connections on API keys, where you can disconnect it; it also ends after 90 days unused. An account holds up to ten connections, besides its five keys.
Use one URL per client: /mcp for keyless use or a key, /mcp-oauth for sign-in. The setups below were checked on 2026-10-16 and 2026-10-17: each client signed in, ran a scrape on the account’s credits, and was refused after the connection was disconnected (record). Another client that follows the MCP authorization spec should sign in the same way, but has not been checked here.
Claude Code
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp-oauthThen run /mcp in Claude Code, select octocrawl and sign in in the browser it opens (claude mcp login octocrawl does the same from a terminal). The Octocrawl plugin sets up the same server: /plugin install octocrawl --marketplace 77777R7/Octocrawl, then sign in from /mcp the same way. Clear authentication in /mcp signs it out. The key setup above keeps working on /mcp. (Claude Code’s guide)
If you added the keyless server first (the home page’s claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp), it has the same name: run claude mcp remove octocrawl before the command above. Codex: codex mcp remove octocrawl; Cursor, OpenCode and Cline: replace the octocrawl entry in the JSON.
Checked with Claude Code 2.1.301 on 2026-10-16.
Cursor
{ "mcpServers": { "octocrawl": { "url": "https://mcp.octocrawl.dev/mcp-oauth" } } }Merge this into .cursor/mcp.json (or ~/.cursor/mcp.json for every project) with no headers, and sign in when Cursor asks for it. The key setup above keeps working on /mcp. (Cursor’s guide)
Checked with Cursor 3.25.6 on 2026-10-16.
Codex
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp-oauthCodex sees that the server signs in with OAuth and opens a browser to sign in as part of add. codex mcp login octocrawl signs in again later. codex mcp logout octocrawl removes the sign-in from that machine; Disconnect on API keys ends the connection itself. The key setup above keeps working on /mcp. (Codex’s guide)
Checked with Codex CLI 0.162.1 on 2026-10-17.
OpenCode
{ "mcp": { "octocrawl": { "type": "remote", "url": "https://mcp.octocrawl.dev/mcp-oauth", "enabled": true } } }Merge this into the mcp object of opencode.json with no headers, then run opencode mcp auth octocrawl and sign in in the browser it opens. opencode mcp list shows whether it is connected. opencode mcp logout octocrawl removes the sign-in from that machine; Disconnect on API keys ends the connection itself. The key setup above keeps working on /mcp. (OpenCode’s guide)
Checked with OpenCode 1.18.26 on 2026-10-17.
Send your first task
Use Octocrawl's scrape tool on https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Overview with formats ["markdown"]. Show the final URL, the HTTP status and the robots.txt decision from its evidenceRecord, then the first heading of the Markdown.The answer carries the page’s Markdown and an Evidence Record: final URL, fetch time, HTTP status, the robots.txt decision and hashes of what was read. A page that could not be read comes back as a result whose status says so (blocked with a blockReason, failed with a failureReason such as timeout, or empty_verified), not as an invented page. Over the daily allowance the tool answers with the time until it resets (00:00 UTC) and how to get a key. For a walk-through of a real task in Claude Code, from finding the page with map to citing its Evidence Record, see Claude Code web scraping.
Check the first result before continuing
A connected server and a listed tool establish the connection. To assess the task, inspect the actual tool result:
- Compare the returned Markdown with the heading or content you asked for. HTTP 200 and nonempty text can also describe a navigation shell or an error page.
- Keep the fetch
statusseparate fromverification.status.not_requestedmeans no task contract was checked.passedmeans the configured checks passed; it does not establish independent field accuracy. - Inspect
readinesswhen present.not_loaded,not_extractedandnot_servedneed different next steps. A missing readiness field is unknown. - Keep incomplete results and failures visible. Do not convert missing content to a zero value or retry paid access automatically.
The tool descriptions keep each option to a line or two. Every option’s full behaviour, with examples, is in the scrape options reference, and crawl and batch in Batch, crawl and map.
If the client lists no tools, check the selected server’s connection error. If the tool returns a quota or access refusal, follow that reason. Changing the MCP client does not establish that the target content can be read. See result states and the Evidence Record reference.
Crawl and batch jobs
A connection with an account (a key on /mcp, or sign-in on /mcp-oauth) also lists the job tools:
crawlandbatch_scrapestart a job;get_crawlandget_batchreport its status;get_crawl_pages,get_batch_items,get_crawl_errorsandget_batch_errorspage through what it read;wait_crawlandwait_batchwait for it;cancel_crawlandcancel_batchstop it.
Without an account the tools are not listed, and a call says it needs one. A job runs on Octocrawl’s servers in slices, so it keeps going after the client disconnects:
- A crawl reads at most 500 pages, and a batch at most 1,000 URLs.
- Each page costs credits as a scrape does: 1 over HTTP and 5 in a browser.
- An account runs two jobs at a time, with up to ten more waiting. A job reads 8 pages at once, 2 of them in a browser, across up to 100 sites a slice.
- Results are kept 7 days after the job ends. Once it has ended,
get_crawl,get_batchand the waits carryexport: a link that downloads every page as one gzip JSONL file, good for 24 hours, and asking again gives a fresh one, so a thousand pages need not pass through the chat. A job that runs out of credits is not lost: it waits, its status says so and where to ask for more, and it goes on by itself when credits come back. - A wait returns after at most 55 seconds (
timeoutMs, default 30 seconds; a wait on a server on your computer takes up to 120 seconds, but Claude Code ends an HTTP MCP request that has sent nothing for 60 seconds) with the status so far. Call it again, or check later withget_crawlorget_batch.
Use Octocrawl's batch_scrape tool on https://example.com/ and https://books.toscrape.com/ with fastMode true. Wait for it, then list its items with each URL, status and first heading.Claude Code, Codex and OpenCode each ran a job end to end through these tools on 2026-10-20 (record). Codex asks before it runs batch_scrape, since starting a job is not read-only. In codex exec, which cannot ask, set default_tools_approval_mode = "approve" on the server’s entry.
Over REST the same jobs are POST https://api.octocrawl.dev/v1/batches and POST /v1/crawl with a key. GET /v1/jobs lists your jobs, newest first. Signed in on octocrawl.dev, Jobs in the dashboard shows each job’s progress while it runs, what it cost, and an export of its results once it ends. maxCredits caps what a job spends, on the tools and over REST. webhook is REST only: it posts the job’s end to an https address, signed with a secret the start returns. The hosted API reference has the signature and a verification example.
Every signed-in account can run hosted jobs. If they are ever paused, a start answers 503 jobs_paused with Retry-After, and the same job runs on your computer meanwhile.
What hosted Octocrawl does not do
- The Monitor tools, and crawls over 500 pages or batches over 1,000 URLs: on your computer (below), where they run without these limits. The hosted answer names this.
- Proxies, CAPTCHA solving and stealth: not offered on the hosted service. A page that blocks a plain request comes back as
blockedwith the reason; the access grant for those routes works on your own server. - Private network addresses, robots.txt overrides and saved logins: refused in hosted mode; robots.txt is obeyed for every URL.
The allowances and what is recorded are on Limits and Privacy.
Run it on your computer
With Node.js 22.13 or later, start the API in any terminal and keep it running:
npx octocrawl serveIt listens on http://127.0.0.1:8787, on this computer only, with every tool: scrape, map, crawl, batch, saved logins, handoff to your own Chrome, the Amazon.sg product tools and the Monitor and delivery tools, and no daily limit. Pages are read over HTTP; for pages that only appear in a browser, run npx playwright install chromium once and the browser lane is used too.
A page behind your own sign-in can often be read here too, with no new login for a site you are already signed in to in Chrome. The agent is told to ask you first, then reads the page in that Chrome, or with a login you save with npx octocrawl login import <site> and save again when it expires. A site can still refuse. Saved logins stay on this computer. Scrape a website with login shows each step.
Add the server to your client
The server runs on this computer as npx -y @octocrawl/mcp and talks to the API from the step above.
Add the stdio server to Codex.
codex mcp add octocrawl -- npx -y @octocrawl/mcpRun codex mcp list, open a new Codex task, then use /mcp to check that the scrape tool is available.
Add the stdio server to Claude Code. Run this in the project where you use Claude Code; --scope user adds it everywhere.
claude mcp add octocrawl -- npx -y @octocrawl/mcpRun claude mcp list: octocrawl shows as Connected. In Claude Code, /mcp lists its tools; then send the first task.
Merge this server into your project .cursor/mcp.json (or your user-level ~/.cursor/mcp.json). Keep existing servers.
{
"mcpServers": {
"octocrawl": {
"command": "npx",
"args": ["-y", "@octocrawl/mcp"]
}
}
}Approve the server when Cursor asks, then check its tools in Cursor Settings → MCP. In Cursor CLI, cursor-agent mcp enable octocrawl approves it and cursor-agent mcp list-tools octocrawl lists the tools.
For OpenCode 1.x, merge this entry into the mcp object in opencode.json. Keep your existing settings and servers.
{
"mcp": {
"octocrawl": {
"type": "local",
"command": ["npx", "-y", "@octocrawl/mcp"],
"enabled": true
}
}
}Run opencode mcp list: octocrawl shows as connected. Then ask for the first task.
Using another MCP client?
npx -y @octocrawl/mcpIf the client reports no tools, check that npx octocrawl serve is still running; if a call fails to connect, the API is not on 127.0.0.1:8787 (pass --base-url to the server command). The repository also has a managed local service for macOS (npm run first-use:local) that adds the Monitor scheduler and the delivery worker behind http://127.0.0.1:8791/mcp; the Monitor guides use it. Then continue with Monitor → HTTPS Webhook or Amazon.sg product JSON.
Self-host for others
npx octocrawl serve --hosted --token <token> serves other machines behind a bearer token, with private addresses, robots overrides, saved logins, handoff and non-HTTPS webhooks refused, as the hosted service does. Point npx -y @octocrawl/mcp --base-url https://your.host --token <token> at it, or any client’s remote URL at your own /mcp once you put one in front of it; the advanced reference has the details.