Advanced reference
Use the browser preview or Connect MCP for a first result. REST, the SDK and self-hosting are for developers who need explicit configuration and persistent task control; they come from the published packages (npx octocrawl serve, @octocrawl/sdk, octocrawl-client on PyPI), not from a repository checkout.
REST and SDK
The API (npx octocrawl serve, on 127.0.0.1:8787) has POST /v1/scrape for one URL, POST /v1/map for a site’s URLs, POST /v1/batches for an explicit URL array with GET /v1/batches/:id/items?limit=...&cursor=... (limit 1 to 50) for its paginated outcomes, and POST /v1/crawl with GET /v1/crawl/:id for following links. Monitor and Delivery have separate REST resources. The TypeScript SDK is published as @octocrawl/sdk and the Python client as octocrawl-client (0.3.2, 2026-10-09); each takes the API’s base URL and an optional token.
For a URL list before a scrape or batch, use the Link Extractor in your browser or call POST /v1/map. See how to find all pages on a website for the sources each method reads and the pages it can miss.
A server started with tokens (--token, which can be repeated, or W2L_API_TOKEN and the comma-separated W2L_API_TOKENS) accepts a request only with Authorization: Bearer <token> naming one of them. Give each client its own token; restarting the server without a token revokes it. Tokens are compared as fixed-length SHA-256 digests in constant time. The SDK sends its token option, or W2L_API_TOKEN from the environment when no token is passed.
Scrape results, batch items and crawl pages carry metadata: the page’s <title>, <meta name="description">, language (<html lang>, else the <meta http-equiv="content-language"> tag), <meta name="keywords">, <meta name="robots">, the first <link rel="icon"> as an absolute URL, and <link rel="canonical">. Each of these seven is null when the page does not declare it; Open Graph fields (ogTitle, ogImage, …) appear only when the page states them. A scrape response’s metadata also carries the call’s own facts, such as scrapeId, statusCode, contentType and cacheState; batch items and crawl pages carry the page’s fields only. document.title is a different value: the content’s own title, usually its first heading.
A URL that answers with a file (PDF, CSV, JSON, plain text, XLSX, XLS or ZIP, by its Content-Type or, under application/octet-stream, by its bytes) is saved byte for byte as received under the task root, files/<sha256>.<ext> (hosted Octocrawl reads and hashes it and saves nothing, so its path is null), and is never sent to the browser. The result’s file block says what it is (kind, detectedBy, contentType), its declared and received size (declaredBytes, bytes), the cap that applied (maxBytes), sha256 and path, and markdownFrom: pdf_text, text or null. A PDF’s markdown is its text layer with a line <!-- page N --> before each page, and file.pdf gives pageCount, pagesRead, each page’s number, printed label and start/end offsets in the Markdown, the PDF’s own info, and warnings (tables_unverified whenever there is text, no_text_layer per page without text: no OCR is run, page_cap, time_budget, page_error). A PDF with text is success, or partial when a cap or budget stopped it; one with no text layer at all is failed with empty_unverified; one that cannot be opened is failed with parse_error. CSV, JSON and text files give their text as markdown; XLSX, XLS and ZIP give none and are success. The operator’s W2L_MAX_FILE_BYTES (default 50 MiB, at most 500 MiB) caps a file; maxFileBytes on scrape, batch or crawl can lower it, and a file over the cap is failed with body_too_large and nothing saved. Images, audio, video, fonts and word-processor or presentation documents are failed with unsupported_content_type.
JSON extraction
A json format takes a JSON Schema: { "type": "json", "schema": { … } }, with prompt and modelFallback optional. Octocrawl fills the schema from the page itself, and every value carries its evidence. A { "type": "json" } entry without a schema is refused (invalid_request): Octocrawl does not extract JSON from a prompt alone, which would need a model for every page. prompt (at most 4,000 characters) only instructs the model fallback. The plain string "json" is different: it returns Octocrawl’s own entity JSON for the page, with no schema of yours.
The schema may use this subset, which covers what Pydantic’s model_json_schema() and zod-to-json-schema usually write:
| Keywords | What Octocrawl does with them |
|---|---|
type (one or a list), properties, required, items (one schema), additionalProperties, enum, const, $defs, definitions, local $ref (# or a pointer into the schema) |
Maps values by them and checks the result. |
anyOf / oneOf of a schema and { "type": "null" }, or of primitive types only |
Maps the value by the non-null schema or the types; a missing nullable field is null with a field_unavailable issue. |
minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf, minLength, maxLength, pattern, minItems, maxItems, uniqueItems |
Checks the result; never fills a value in. A page value that breaks a check makes the result incomplete with a field_unavailable issue. A pattern has at most 2,000 characters; one that can backtrack catastrophically, such as ^(a+)+$, is refused with invalid_request. A pattern with a lookaround, a backreference, a counted repetition above 16 (such as [0-9a-f]{32}), a \p{…} or \u{…} escape or a character outside the Basic Multilingual Plane, and any text holding such a character, is matched only against text of up to 2,048 UTF-16 code units within 100 ms; text it cannot decide counts as breaking it. |
title, description, $comment, examples, deprecated, readOnly, writeOnly, format, default; $schema (draft-07, 2019-09 or 2020-12) and $id at the root only |
Accepts and ignores them: format is not checked and default is never filled in. |
A schema is at most 64 KiB, 8 levels deep and 100 properties. Any other keyword, a union of objects, arrays or references, a keyword other than an annotation beside $ref, and $schema or $id below the root are refused with unsupported_parameter; details.parameters names the keyword where it was sent, such as formats[0].schema.properties.author.allOf.
A result is complete, incomplete (a required field has no source: missing_required; or the page was not a success: page_unsuccessful, page_partial) or invalid (the model’s answer did not fit the schema twice). json.evidence, like evidenceRecord.fieldEvidence, gives each value’s source: a label row, h1[0] or title for the content title (dom), finalUrl or requestedUrl (fetch), document.pageType (inferred), document.product.<name> for a product list or map the extractor reported empty (inferred), or model. For a number read from the page’s text, json.evidence also quotes that text (text).
Numbers are read as the page writes them: a decimal . or ,; thousands grouped by ., ,, a space or an apostrophe in groups of three (or India’s lakh groups); a currency symbol or code before or after; ,- for a whole amount. 12,99 € is 12.99, and 1.299,00 €, 1 299,00 € and CHF 1'299.– are 1299. A single . or , before exactly three digits (1.299 €, $1,299) is read only when the value settles it: a review count, an amount in a currency without minor units (JPY, KRW, ₩, 円, …), or a JSON-LD or product:price:amount price, whose format writes . as the decimal point. The page’s language, currency or domain is not used to guess. Otherwise the field is left out (null when nullable) with a field_unavailable issue quoting the text. The model fallback (modelFallback: true and W2L_EXTRACT_BASE_URL / W2L_EXTRACT_MODEL set, with W2L_EXTRACT_API_KEY when the provider needs one) fills only what the page did not give, or a page value that breaks the schema; every other value keeps what the page said and its evidence. It asks the provider for strict structured outputs with a strict-safe copy of the schema, or for the schema as given when strict mode cannot express it; json.modelUsage.strict and strictReason say which and why.
Start the API only after reviewing its network and task-store settings (npx octocrawl serve --help). For full request shapes and examples, read the repository’s docs/onboarding.md, docs/batch-scrape.md and examples/monitor-workflow.ts at the tag of the version you run, since the contract moves between versions.
Evidence Record
Every scrape result (full or compact, so MCP scrape too), batch item and crawl page carries evidenceRecord: the evidence for that page in one shape, whichever lane fetched it. Its JSON Schema (draft 2020-12) is packages/contracts/schemas/evidence-record.v1.json in the repository, which describes every field; each record names it with schemaVersion: "w2l.evidence/1". Octocrawl writes every field on every record, and a value it did not observe is null, never 0 or a guess. The record sits beside the existing fields (evidence, snapshot, compliance, lane), which do not change. The Firecrawl-compatible /fc routes keep Firecrawl’s shape and do not carry it.
w2l.evidence/1 changes only additively: a later revision of the schema file accepts every record an earlier one accepted. A field added later is optional in the schema, so older records stay valid, though Octocrawl writes it on every record from then on: contentEncoding, pageActions, access, readiness and document at the top level, identity.device and identity.requestHeaders, robotsDecision.overrideBasis, access.completion, egress, session, paidCalls and grant, access.egress.exit and access.egress.kind, and bytes and contentType on artifacts. Enums only gain values (lane gained my_browser); no field changes its meaning. Anything else would get a new schemaVersion and schema file.
| Field | Meaning |
|---|---|
schemaVersion |
w2l.evidence/1. |
requestedUrl |
The URL as requested. |
finalUrl |
The last URL Octocrawl requested for the page, after redirects. On the browser lane a script or a meta refresh that loaded another document is a redirect; a URL the page set with the history API (pushState, replaceState), which nothing requested, is not. null when it sent none: a robots.txt disallow, a DNS failure or a policy refusal came first. |
redirectChain |
{ urls, complete }. urls runs from requestedUrl to finalUrl ([requestedUrl] without a redirect, [] when nothing was requested). complete is true when every hop is listed: on the HTTP lane, which requests each hop itself, and on the browser lane, which lists each redirect Chromium followed and each document a script or a meta refresh loaded (false there when a follow-up navigation did not start at requestedUrl, a document came without a request, or the chain of a page that kept moving on was cut to its first URL and last 20). It is false on the provider lane, which sees only where its vendor started and ended. |
fetchedAt |
When the reported response was received, UTC ISO 8601 with milliseconds: its headers on the HTTP lane, the capture of the rendered page on browser lanes, the vendor’s answer on the provider lane. null without a response. |
httpStatus |
The status of the response that answered finalUrl: on the browser lane, of the document the page shows when it is captured, not of the first navigation. null without one. |
status, reason |
Octocrawl’s verdict, and the failure, block or budget reason (null for other statuses). |
lane |
http, browser_local, browser_local_authed, browser_proxy, provider or my_browser (your own Chrome, on a site you allowed). |
robotsDecision |
{ decision, robotsUrl, robotsSha256, unreachable, crawlDelayMs, userOverride, overrideBasis }, or null when the result ended before robots.txt was checked. decision is allowed, disallowed or no_robots (the site has none); unreachable says why robots.txt could not be fetched, which counts as a disallow. userOverride is true when the fetch went ahead although decision was disallowed, and overrideBasis says why: user_named_url (a server on your computer reading a URL you named in a scrape or batch), robots_override (an override you recorded) or ignore_robots_txt (a crawl or map that set it, on your computer); null when nothing was set aside. A hosted server never overrides. |
rawSha256 |
SHA-256 of the body Octocrawl read, as evidence.rawBodySha256: for a file (PDF, CSV, …), the bytes received in any lane, equal to its artifact’s sha256; for a web page, the response body as UTF-8 text on the HTTP lane, the rendered HTML on browser lanes, the vendor’s HTML on the provider lane. |
outputSha256 |
{ markdown, json }: SHA-256 of the UTF-8 bytes of the markdown delivered in this response, and of json.data as canonical JSON (RFC 8785: keys sorted at every level, no whitespace). null for what was not delivered. |
extractor |
{ name, version, commit }: extract-tf and its version (extract-tf/16 in 0.3.2) for a web page, pdf-text (pdf-text/1) for a PDF, file-text (file-text/1) for another file; and the source commit when the operator sets W2L_SOURCE_COMMIT, else null. |
fieldEvidence |
For a JSON request, each field’s JSON Pointer mapped to { source, locator }: the source is jsonld, microdata, meta, hydration, dom, text, inferred, model, fetch (the fetch’s own URL, not the page) or pdf, and the locator says where, such as table[0] tr[3] "UPC", h1[0], title, finalUrl, or page 14 "KPI 2" for a Label: value line of a PDF (page N is the <!-- page N --> marker). JSON-LD, microdata and meta values outside the Amazon adapter have no locator yet (null), nor has model. A field that is absent or null has no entry. null without a JSON request. |
artifacts |
Files saved for the result (hosted Octocrawl saves none), each { kind, path, sha256, bytes, contentType }: a file saved as received (kind: "file", with its size and Content-Type), and, when W2L_CAPTURE_RAW_DIR is set, the page snapshot (kind: "snapshot", size and type null) and the screenshot format’s image (kind: "screenshot"); otherwise []. |
proxy |
host:port of the environment proxy the request went through (local mode). null when it went direct or the lane does not report its route, which the provider lane never does. |
identity |
{ userAgent, mode, contact, device, requestHeaders }: the User-Agent observed on the request for the page (null when none was sent or observed), the crawl mode, the contact a research-mode User-Agent declares (W2L_CONTACT), the device emulated (desktop or mobile), and the names of the custom headers you sent with a SHA-256 of each value, never the value (null when no request was sent). |
access |
{ route, executor, executorVersion, profile, externalCostUsd, completion, egress, session, sessionApplied, allowance, paidCalls, grant }: how the page was reached, read from its own trace. route is http, http_compat (the browser-compatible HTTP transport), browser, enhanced_browser (Patchright), installed_chrome (the Google Chrome installed on the machine), hosted_chrome (Google Chrome on Octocrawl’s own server), authed_browser (a saved login), user_browser (your own Chrome, through the my_browser lane or a handoff) or vendor; null, with the client, when no lane produced the result (a rung the deadline cut, that failed, or whose identity was refused). executor is the client that sent the requests (undici, impit, playwright, patchright, your browser, or the vendor’s id), with its version when the lane reports one; profile is the browser profile the HTTP transport sent (chrome142), null elsewhere. externalCostUsd is the run’s third-party spend (for a page from the cache, the stored fetch’s): 0 when no paid service was called, null when one stated no price. egress is where the page left from: { proxy, source, switchedFrom, exit, kind } with the proxy’s host:port (never its credentials) and source pool (your W2L_EGRESS_PROXIES), environment (HTTPS_PROXY) or direct (no proxy on a request that got a page response); switchedFrom names the pool egress the task last moved off before this page was read, null otherwise; exit is { ip, country, observedAt }, the address (and two-letter country, when given) your W2L_EGRESS_ECHO_URL saw for that pool proxy, null without one, when the echo did not answer in the page’s time, or for a proxy not from the pool (the echo uses a connection of its own, so behind a gateway that changes exit per connection it shows where that proxy was seen to leave from, not the page’s own address); kind is the kind of address the page left from (google, datacenter, isp or residential) when the lane states it, which only hosted_chrome will, null otherwise; null as a whole when nothing says: a vendor’s service, your own browser, or a lane that stopped before any page request (robots.txt, an address check, a deadline). session is the task’s cookie session as { id }, never its cookies, null when the page was read with none. sessionApplied is what a saved login put into the browser, { cookies, localStorageOrigins }, counts alone, null with no saved login; it says the login was sent, not that the site took it. allowance is how you allowed the site of a page read in your own Chrome (my_browser): { basis, allowedAt }, basis clicked (on Octocrawl’s page, for this request) or remembered (from an earlier click, with no page shown), null on every other read. completion is how the page was reached: unattended (Octocrawl’s own lanes, or a provider’s service), authorized_session (a saved login), user_browser (your own Chrome, with no step of yours) or handed_to_person (your own Chrome, after you got through a check); null when no page was read. paidCalls lists each paid provider call (provider, rung, ceiling, charge, reported cost, outcome), and grant the access grant the run used (its SHA-256, tier and attestation time). completion, egress, session, sessionApplied, allowance, paidCalls and grant are optional in the v1 schema file, so older records stay valid. |
readiness |
{ state, basis }: whether the page was ready for its task when it was read, from the result’s own signals (ADR 0007). state is ready; not_loaded, read before what it shows had come (a loading indicator still showing, an HTTP shell its scripts fill in or an HTTP page with no main content, a page still changing when the my-browser lane read it, a wait or a list the deadline cut short, a rendered page with no main content and at most 4,000 visible characters); not_extracted, read as served but not extracted (no main content in a rendered page with more than 4,000 visible characters, a file with no text or that could not be read); or not_served, the route, the access or the region did not serve it (a block, an HTTP error, robots.txt, a network failure, a timeout with no answer, a spent budget). basis lists the signal codes it was read from, never page text. Scrape responses, batch items and crawl pages carry the same readiness, with waitedMs when the lane recorded a wait. Optional in the v1 schema file, so older records stay valid. |
document |
{ captureId, generation, responseMatch }: the capture the content was read from, the same as evidence.document on a full response. captureId is a random id for that one read (each HTTP read and each browser read of a page gets its own, so a later read of the same URL has another). generation is how many documents the page’s main frame had loaded when the browser lane read it, null on the HTTP lane and when the lane could not count. responseMatch says whether httpStatus and the content type are the response of the document read: matched; mismatched, the page moved on while it was read or its document came without a response that could be paired with it; unobserved, the lane could not see the documents (a vendor connector with no CDP session); null on the HTTP lane, which reads the response body itself. null when no web page was read (no response body, or a file such as a PDF, which rawSha256 and its artifact identify) or the lane keeps no capture (the my-browser lane). Never page content. Optional in the v1 schema file, so older records stay valid. |
To check a cited value, hash the markdown you received as UTF-8 and compare it with outputSha256.markdown; for JSON, serialize json.data with sorted keys and no whitespace first.
Errors
When the API refuses or fails a request, it returns an error body instead of a result:
{ "error": "unsupported format: summary (supported: markdown, links, json, html, rawHtml, images, tables, screenshot, attributes, list)", "code": "unsupported_format", "details": { "formats": ["summary"] } }Branch on code; error is written for people. details appears only with unsupported_parameter and unsupported_format and lists the rejected names as sent, in parameters and formats. Some refusals also carry agentHints, sentences that say what to change, and a 429 from a server of your own on a native route carries retryAfterSeconds. The Firecrawl shim (/fc) returns the same error, code and details after "success": false, with hints as agent_hints and no retryAfterSeconds.
| Code | HTTP | Meaning | Typical cause | What to do |
|---|---|---|---|---|
invalid_json |
400 | The body is not valid JSON. | A truncated or hand-edited body. | Send one JSON object. |
invalid_request |
400 | A value is missing, malformed or out of range. | No url, a non-HTTP(S) URL, an invalid includePaths pattern, a { "type": "json" } format without a schema, a JSON Schema over its limits or with a malformed value, actions or skipTlsVerification on a hosted server. Also, for now, a batch refused because another is still being accepted or the active-batch limit is reached. |
Correct what the message names; retry a refused batch later. |
unsupported_parameter |
400 | The request names a parameter Octocrawl does not support, or a value it cannot honour. Nothing is ignored silently. | A Firecrawl option Octocrawl does not take, such as location or useIndex; proxy on a native route; limit on native crawl (it takes maxPages); "access": "enhanced" on a server without an access grant; a JSON Schema keyword outside the supported subset such as allOf. |
Remove or change what details.parameters lists. |
unsupported_format |
400 | A requested format is not produced. | A format name Octocrawl does not know, such as summary or changeTracking; on /fc, also json. |
Use one of the names the message lists. |
unauthorized |
401 | The bearer token is missing or is not one of the server’s tokens. | A server started with --token, W2L_API_TOKEN or W2L_API_TOKENS, which --hosted requires. |
Send Authorization: Bearer <token>; the SDK reads W2L_API_TOKEN when no token is passed. |
not_found |
404 | The task, monitor, run, destination, delivery or session does not exist, or no route matches. | A mistyped ID, another task directory, a Firecrawl v2 path on /fc. |
Check the ID and the path. |
conflict |
409 | The resource’s current state does not allow the request. | A monitor revision out of sequence or on a running monitor; a run queued on a paused, busy or unknown monitor; a retry of a delivery that is not dead-lettered; resuming a crawl that is completed, cancelled or still running; a handoff or saved login that this server does not offer or your Chrome did not allow; a reused idempotency key with another request. | Read the resource’s state first; the same request fails again. |
internal_error |
500 | Octocrawl failed unexpectedly. | A bug or a storage failure. | Retry once, then report it. A local server returns the underlying message; a hosted server (--hosted, remote MCP) returns internal error and writes the cause to its log. |
rate_limited |
429 | Too many requests in a minute. | Hosted Octocrawl, always; a server of your own only when started with --rate-limit-per-minute or W2L_RATE_LIMIT_PER_MINUTE. |
Wait Retry-After seconds. |
browser_busy |
429 | Hosted only: this scrape starts in the browser and the key already has one running there, or that scrape holds the account’s last credits. | A waitFor above 0, a screenshot or a json modelFallback before the first scrape ends. A page that only turns out to need the browser is not refused: it comes back with what HTTP read and a browser_busy warning. |
With an account, send several URLs as one batch job (POST /v1/batches, or batch_scrape over MCP). For this page, leave those out if HTTP can read it, or ask again after Retry-After seconds; the running call can take up to 5 minutes. |
quota_exhausted |
429 | Hosted only: the day’s allowance is used up, keyless, per key or for the whole service. | More pages than the allowance. | Wait for 00:00 UTC (Retry-After), use a key, or run Octocrawl on your computer. |
hosted_unavailable |
403 | Hosted only: the route is not served there. | Anything but POST /v1/scrape, POST /v1/map, the crawl and batch job routes, their GET reads and /health. |
Run Octocrawl on your computer, as the hint says. |
On hosted Octocrawl, unauthorized means a key it did not issue or has revoked, and its MCP endpoint refuses a body over 256 KiB with 413 and no code. These codes describe the request, not the page. A page that was fetched but blocked or failed is a normal result with its status and reason (see result states); on /fc it is HTTP 200 with success: false, no code, and the reason in data.metadata.error.
The SDK throws W2LError for every error response. It carries status, code (when the body had one), method, path and the parsed body; its message is still <METHOD> <path> failed: <status> <body>, or crawl not found: <id> and batch not found: <id> for those lookups. A request that never reached the API throws the underlying fetch error. scrape waits for the answer until the request’s timeout (default 300 000 ms) plus 30 s; on Node this replaces fetch’s own 300 s wait for the response headers, so the API’s failed/timeout answer at the deadline arrives instead of a TypeError. waitBatch / waitCrawl (and batchAndWait / crawlAndWait) retry a status request that fails with a network error, 408, 429 or 5xx, up to maxRetries (default 5) in a row, before throwing its error; they throw other errors at once, and WaitTimeoutError (taskId, timeoutMs, last: the last status read or null) when timeoutMs runs out. map waits for its timeout (default 60 000 ms) plus 5 s. W2LError also has retryAfterMs (from Retry-After, else null) and agentHints; a 429 on scrape or another single call is not retried for you.
import { W2L, W2LError } from '@octocrawl/sdk'
const w2l = new W2L({ baseUrl: 'http://127.0.0.1:8787' })
try {
await w2l.getCrawl(taskId)
} catch (error) {
if (!(error instanceof W2LError) || error.code !== 'not_found') throw error
// The ID is wrong or belongs to another task directory.
}An MCP tool error with one of the eight codes from invalid_json to internal_error starts with that code, for example unsupported_format: POST /v1/scrape failed: 400 {"error":...}, and their JSON-RPC error data is { "code": "unsupported_format", "status": 400 }. Successful tool results are unchanged.
Self-hosted operation
npx octocrawl serve runs the API on 127.0.0.1:8787 for the computer it runs on. octocrawl serve --hosted --token <token> serves other machines behind a bearer token: private addresses, robots overrides, saved logins, handoff and non-HTTPS webhooks are refused in that mode. The repository’s managed local service adds the Monitor scheduler and delivery worker with SQLite state; keep its task directory across restarts. The anonymous page preview on this site is a separate request-based service and runs no persistent Monitor or Delivery tasks. To run it in Docker on your own server, with Chromium and a data volume, see Run Octocrawl with Docker Compose.
Hosted Octocrawl, run by Octocrawl, serves the same POST /v1/scrape and POST /v1/map, with their GET /v1/scrapes/:id and GET /v1/maps/:id reads, at https://api.octocrawl.dev and the MCP tools scrape, map and scrape_product at https://mcp.octocrawl.dev/mcp: keyless within a daily allowance over HTTP, with a key for more pages and the browser lane; it also answers GET /health and the MCP server card at /.well-known/mcp/server-card.json (any other /.well-known file is 404). With an account it also runs crawl and batch jobs (POST /v1/crawl, POST /v1/batches, their reads, cancel and export, GET /v1/jobs) and lists the job tools; see Crawl and batch jobs. Every other route answers 403 hosted_unavailable with a hint to run Octocrawl on your computer, and every other tool answers tool not available in this deployment. Its allowances are on Limits. 127.0.0.1 in a snippet means the computer the client runs on.
Evidence and boundaries
The result states page explains user-facing outcomes. The repository’s records say what was tested where; a passing test on one computer is not evidence that a hosted service does the same.