Octocrawl / Blog

Hosted crawl and batch jobs: give your agent a whole site as Markdown

A pixel-dithered lighthouse on a cliff at dusk, its beam sweeping across a dotted blue sky.

Your agent can now hand Octocrawl a whole documentation section or a list of URLs and get every page back as clean Markdown, without running a server. Hosted crawl and batch jobs run on Octocrawl’s servers, on your account’s credits: a crawl reads up to 500 pages of a site, a batch up to 1,000 URLs, and each page comes with its Evidence Record.

On 2026-10-21 we asked Claude Code to crawl a section of the MCP specification through the hosted server and tell us which JSON-RPC methods each page defines. It started the job, waited 10 seconds, read all 7 pages in two calls and answered. The run is below.

What can an agent hand to hosted Octocrawl now?

Before, hosted Octocrawl read one page at a time (scrape) or listed a site’s URLs (map). Crawls and batches ran on your own computer. Now an account runs both on the hosted server:

Job What it does Hosted limit
crawl Follows links from a start page within its section of the site and reads each page. 500 pages a job
batch_scrape Reads a list of URLs you already have. 1,000 URLs a job

A job runs on the server, so it keeps going if the client disconnects. The tools around it read its status (get_crawl, get_batch), wait for it (wait_crawl, wait_batch), page through what it read (get_crawl_pages, get_batch_items) and stop it (cancel_crawl, cancel_batch). The same jobs are on the REST API as POST /v1/crawl and POST /v1/batches, with GET /v1/jobs to list yours, and in the TypeScript SDK and the Python client.

What does a hosted crawl look like from Claude Code?

We connected Claude Code (version 2.1.301, with Claude Opus 5.5) to https://mcp.octocrawl.dev/mcp with an account’s key, turned its own WebFetch and WebSearch off, and gave it this prompt:

text
Use Octocrawl's crawl tool on https://modelcontextprotocol.io/specification/2025-11-25/server/ with maxPages 15 and fastMode true. Wait for it with wait_crawl, then read every page with get_crawl_pages. Give me a table of the pages it read: each page's title, URL and the JSON-RPC methods it defines. Then tell me how many pages the job read, how long it took and how many credits it used.

It took 6 turns and 30 seconds:

  1. crawl started the job at once and answered with its taskId.
  2. wait_crawl came back when the job ended: completed, 7 pages, 0 failed, 7 credits. The section has 7 pages, so the crawl stopped there, under its limit of 15.
  3. get_crawl_pages returned the first 4 pages with a cursor, and a second call with that cursor returned the other 3.

Claude’s table, as it answered:

Page JSON-RPC methods it defines
Overview None (an overview of prompts, resources and tools)
Prompts prompts/list, prompts/get; notifications/prompts/list_changed
Resources resources/list, resources/read, resources/templates/list, resources/subscribe; notifications/resources/list_changed, notifications/resources/updated
Tools tools/list, tools/call; notifications/tools/list_changed
Pagination None of its own: it adds cursor and nextCursor to the list methods
Completion completion/complete
Logging logging/setLevel; notifications/message

The job ran from 19:02:02 to 19:02:11 UTC, about 10 seconds for 7 pages.

Why does Markdown matter for an agent?

Because the agent pays for every character it reads. The 7 pages above are about 3.2 MB of HTML as the site serves them, mostly navigation, scripts and styling. Octocrawl returned 41,254 characters of Markdown for the same pages: the headings, text, tables and code the specification is made of. That is what fits in Claude’s context with room left to think, and why two get_crawl_pages calls were enough.

How do you know each page is what the site served?

Every page in a job carries the same Evidence Record a single scrape does. For the Tools page in the run above:

  • finalUrl: https://modelcontextprotocol.io/specification/2025-11-25/server/tools, the address that answered.
  • fetchedAt: 2026-10-21T19:02:10.360Z, when it answered.
  • httpStatus: 200, read over HTTP.
  • robotsDecision: allowed. Hosted Octocrawl obeys robots.txt for every page of a crawl.
  • outputSha256.markdown: a hash of the exact Markdown Claude received, so you can tell later whether a page changed.

A page that cannot be read stays in the job with its reason (blocked, failed, timeout) instead of disappearing, and get_crawl_errors lists them.

What does a job cost?

A page in a job costs what a single scrape costs: 1 credit read over HTTP, 5 in a browser. The run above used 7 credits. Every account gets 500 credits a month free.

  • maxCredits caps a job: it stops before the next page would pass the cap.
  • An account runs two jobs at a time, with up to ten more waiting.
  • Results are kept 7 days after a job ends. The dashboard’s Jobs page lists them with their progress.
  • Over REST, a job can post its end to your server as a signed webhook.

When should you run it on your own computer instead?

Run Octocrawl locally with npx octocrawl serve when a job is larger than hosted takes (more than 500 pages or 1,000 URLs), when the pages need your login, or for Monitors on a schedule. The same tools work against the local server, with no daily limit.

How do you start?

Make an account at octocrawl.dev and create a key, or sign in from your client instead. Then connect it:

bash
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp --header "Authorization: Bearer oc_…"

Ask your agent to crawl a section or batch-scrape a list, and to wait for the job before it reads the pages. Crawl and batch jobs has the tools, the limits and the setup for Codex, Cursor and OpenCode.

FAQ

Do hosted crawl and batch jobs need an account?

Yes. Without a key, hosted Octocrawl offers scrape and map only. Any account can run jobs, on its own credits.

Does a crawl obey robots.txt?

Yes. Hosted Octocrawl reads each site’s robots.txt and skips the pages it disallows, recording the decision on every page.

What happens if my agent disconnects in the middle of a job?

Nothing: the job keeps running on the server. Reconnect and read it by its taskId, or find it on the dashboard’s Jobs page.

How is this different from running Octocrawl locally?

The tools and results are the same. Hosted jobs need no server on your side and are limited to 500 pages a crawl and 1,000 URLs a batch; a local server has no such limit and can also read pages behind your login.