Scrape a page

POST /v1/scrape reads one URL and returns what you ask for: Markdown, links, tables, fields as JSON, the raw HTML. Every result says whether the page was read (status) and carries an Evidence Record: the final URL, when it was fetched, the HTTP status, the robots.txt decision and hashes of the bytes read and of the output.

Hosted Octocrawl serves it at https://api.octocrawl.dev with no setup: without a key, 20 pages a day per address, over HTTP only. With a key from your account, it spends credits and can also read in a browser. The same route runs on your own computer with npx octocrawl serve (http://127.0.0.1:8787), with no daily limit.

First call

bash

curl -X POST https://api.octocrawl.dev/v1/scrape \
  -H 'content-type: application/json' \
  -d '{"url":"https://en.wikipedia.org/wiki/Octopus","formats":["markdown","links","tables",{"type":"json","schema":{"type":"object","properties":{"kingdom":{"type":["string","null"]},"phylum":{"type":["string","null"]},"class":{"type":["string","null"]},"order":{"type":["string","null"]}}}}]}'

Add -H 'authorization: Bearer oc_…' to spend your account’s credits instead of the keyless allowance. On 2026-10-17 at 09:28 UTC, without a key, that call returned this, shortened:

json

{
  "status": "success",
  "markdown": "…An **octopus** (pl.: **octopuses** or **octopodes**…) is a soft-bodied, eight-limbed …",
  "json": {
    "status": "complete",
    "data": { "kingdom": "Animalia", "phylum": "Mollusca", "class": "Cephalopoda", "order": "Octopoda Leach, 1818[1]" },
    "evidence": [
      { "path": "/kingdom", "source": "dom", "evidencePath": "table[0] tr[4] \"Kingdom:\"" },
      { "path": "/order", "source": "dom", "evidencePath": "table[0] tr[10] \"Order:\"" }
    ],
    "issues": []
  },
  "evidenceRecord": {
    "schemaVersion": "w2l.evidence/1",
    "finalUrl": "https://en.wikipedia.org/wiki/Octopus",
    "fetchedAt": "2026-10-17T09:28:44.066Z",
    "httpStatus": 200,
    "status": "success",
    "lane": "http",
    "robotsDecision": { "decision": "allowed", "robotsUrl": "https://en.wikipedia.org/robots.txt" },
    "rawSha256": "0962dd1b780cabdb7e8fc16f55dd713dbd16e685d8978d512c1c7f6e1e439cf6",
    "outputSha256": {
      "markdown": "ea6266cfa47ce54c4646fec4650ec09cb020b5da43ddb3dbd73fa614a18dfbdf",
      "json": "4657c798730a00302a262fe6c5a58b528082f7db0e09483e4f4a6a5f7dba595c"
    }
  }
}

The same response held 1,277 links, one table (the infobox, 15 rows) and 560 lines of Markdown. The requests and results of that day’s runs are in the run record.

Formats

formats takes any of these, markdown when it is left out:

Format	What comes back
`markdown`	The page’s main content as Markdown.
`links`	The URLs the page links to.
`tables`	Each data table of the content as rows, in the order the Markdown has them.
`{ "type": "json", "schema": … }`	Fields you name in a JSON Schema, each with where on the page it was read.
`html`, `rawHtml`	The cleaned HTML of the content, or the bytes as received.
`images`	The URLs of the page’s images.
`{ "type": "attributes", "selectors": … }`	Attribute values of the elements you name, such as every `a[href$=".pdf"]`.
`{ "type": "list" }`	One record per repeated item, such as products on a listing page.
`screenshot`	An image of the page. It needs the browser, so a key on hosted Octocrawl.

Keep only what you need

By default (onlyMainContent: true) the Markdown leaves out the header, navigation and footer. onlyMainContent: false keeps the whole page. includeTags keeps only the elements its CSS selectors name, and excludeTags removes elements before the content is taken.

Wikipedia served the Octopus page in two variants that day. Read at 16:00 UTC, the variant with rawSha256 0a1cc6ca…3f8de562 gave 560 lines and 39,370 tokens of Markdown. Read again at 18:18 UTC with only its headings and paragraphs:

json

{ "url": "https://en.wikipedia.org/wiki/Octopus", "formats": ["rawHtml", "markdown"], "includeTags": ["h1", ".mw-parser-output section > p", ".mw-parser-output h2", ".mw-parser-output h3"] }

The same bytes (0a1cc6ca…3f8de562 again) then gave 189 lines and 15,463 tokens, without the badges and the infobox. The selectors are listed in Scrape options.

Fields as JSON

A json format fills the schema you send from the page itself and gives each value its source in json.evidence. A value the page does not hold is not made up: the field is missing, or null when the schema allows it, and an issue names it. Reading an Amazon.sg book’s product page the same day, /brand came back with a field_unavailable issue, because a book has no brand, rather than the author’s name in its place.

json.status is complete, incomplete (a required field has no source, or the page was not a full success) or invalid. Check it, and issues, before you use a value. The supported schema keywords are in the advanced reference.

Pages that need a browser

Octocrawl reads over HTTP first and opens a browser when the page needs scripts to show its content. On 2026-10-17, quotes.toscrape.com/js/, whose quotes a script draws, came back with all 10 quotes after HTTP and then the browser (channelsTried: ["http", "browser_local"]).

fastMode: true keeps a call over HTTP alone. A page that needs scripts then returns HTTP’s verdict instead of its content.
waitFor (milliseconds) starts in the browser and waits that long after the page loads before reading it.
timeout bounds the whole scrape; when it fires, the answer is partial with what was read, or failed.

Without a key, hosted Octocrawl reads over HTTP only. With a key it can use the browser, one scrape at a time per key.

Read the result

status says what happened to the page. A scrape usually ends in one of these five; only success and partial carry the page’s content.

`status`	Meaning
`success`	The page was read.
`partial`	The deadline cut the read short; the content is what was read by then.
`blocked`	The site stopped the read: a CAPTCHA, a login wall, an access check. `blockReason` says which.
`failed`	Nothing usable was read, or the read was not allowed. `failureReason` says why.
`empty_verified`	The page was read and is empty, such as a search with no results.

Two pages from the same day: a Walmart product page answered with a CAPTCHA page, so the result was blocked with blockReason: "captcha". A Google search URL is disallowed by Google’s robots.txt, which hosted Octocrawl always obeys, so it was failed with failureReason: "policy_denied" and the page itself was never requested. Hosted Octocrawl does not solve CAPTCHAs or change who it says it is. agentHints, when present, says what to try next.

An error body ({ "error", "code" } with HTTP 4xx or 5xx) is about the call, not the page: a bad option, a missing key, the daily allowance used up. The codes are in the advanced reference.

Check the Evidence Record

evidenceRecord.rawSha256 is the SHA-256 of the bytes Octocrawl received, and outputSha256.markdown that of the Markdown it returned. Hash the markdown you got as UTF-8 and compare: if they match, it is the output of that read. Request rawHtml to keep the bytes themselves. robotsDecision records the robots.txt rule that applied, and fieldEvidence where each JSON value was read. Every field is described in the advanced reference.

Cost

On an account’s credits, a page read over HTTP costs 1 credit and a page read in a browser 5. A blocked or refused page costs nothing. Each answer gives X-Credits-Used and X-Credits-Remaining. An account gets 1,000 credits a month; sign in to make a key. Without a key the call is free, within 20 pages a day. Limits has the rest.

The same call goes through the TypeScript SDK, the Python client octocrawl-client, npx octocrawl scrape and the MCP scrape tool (Connect MCP). To read many pages, list them with POST /v1/map first.