GitHub

ScrapeOctocrawl web data API

Any page, scraped clean.

Turn any web data into Markdown, links, tables and source-checked fields, with an Evidence Record for every result.

Try it free Read the docs
Books to ScrapeWe love being scraped!
A Light
in the AtticShel Silverstein

A Light in the Attic

£51.77

In stock (22 available)

★★★★★

Warning! This is a demo website for web scraping purposes. Prices and ratings here were randomly assigned and have no real meaning.

Product Description

It’s hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. …

Product Information

UPCa897fe39b1053632
Product TypeBooks
Price (excl. tax)£51.77
Price (incl. tax)£51.77
Tax£0.00
AvailabilityIn stock (22 available)
Number of reviews0
  1. ![A Light in the Attic](https://books.toscrape.com/media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg)
  2.  
  3. # A Light in the Attic
  4.  
  5. £51.77
  6.  
  7. In stock (22 available)
  8.  
  9. ---
  10.  
  11. **Warning!** This is a demo website for web scraping purposes. Prices and ratings here were randomly assigned and have no real meaning.
  12.  
  13. ## Product Description
  14.  
  15. It’s hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. …
  16.  
  17. ## Product Information
  18.  
  19. | UPC | a897fe39b1053632 |
  20. | --- | --- |
  21. | Product Type | Books |
  22. | Price (excl. tax) | £51.77 |
  23. | Price (incl. tax) | £51.77 |
  24. | Tax | £0.00 |
  25. | Availability | In stock (22 available) |
  26. | Number of reviews | 0 |
successHTTP 2001 creditsha256 a6e572bec156

A real scrape of a page on books.toscrape.com, a sandbox made for practising scraping, recorded on 13 October 2026. Point at any line to see where it came from.

HOW IT WORKS

From a link to data
you can trust.

The same page, step by step, from the link to the record.

  1. 01Take the link

    One public URL.

    Octocrawl takes an http or https link. Hosted, it reads only public web addresses: a link into a private network is refused before anything is sent.

  2. 02Read robots.txt

    The site’s rules come first.

    It asks for /robots.txt before the page. books.toscrape.com has none (404), which counts as no rules. Hosted Octocrawl never requests a page the rules disallow: it answers with the reason, and charges nothing.

  3. 03Fetch the page

    HTTP first. A browser only when needed.

    The page came back over HTTP: 200, text/html, 9,279 bytes, 1 credit. Its content was in the HTML, so no browser was needed. A page drawn by JavaScript would go on to one, at 5 credits.

  4. 04Keep the content

    Menus and clutter fall away.

    It finds what the page is about (here, a product) and keeps it. The site’s header, the rating stars and the footer are left out. Ask for the whole page to keep them.

  5. 05Shape the output

    One read. Every format.

    From that one read: Markdown (397 tokens), 3 links, 1 table of 7 rows, and the 5 fields asked for, each with where on the page it was found.

  6. 06Write the Evidence Record

    Proof of where it came from.

    Last, the record: the URL as fetched, the time, the robots.txt decision, the lane, and SHA-256 hashes of the page’s bytes and of its Markdown and JSON. Hash the page yourself and you get the same value.

WHAT IT DOES

Everything the page says.
Nothing it doesn’t.

Clean Markdown, any format

The main content by default: the article, not the menus. Ask for the whole page when you want it, in any of eight formats from one read.

  1. ![A Light in the Attic](https://books.toscrape.com/media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg)
  2.  
  3. # A Light in the Attic
  4.  
  5. £51.77
  6.  
  7. In stock (22 available)
  8.  
  9. ---
  10. …

397 tokens of Markdown

  • markdown
  • links
  • html
  • rawHtml
  • images
  • tables
  • screenshot
  • json

Reads the hard ones, or tells you why

Pages drawn by JavaScript go on to a browser. PDFs come back as text. A page it may not read is never fetched, and the answer says why. Three real runs:

https://quotes.toscrape.com/js/

HTTP read
empty shell (empty_unverified)
then
read in a browser
status
success · HTTP 200
hosted cost
5 credits
  1. # [Quotes to Scrape](https://quotes.toscrape.com/)
  2.  
  3. [Login](https://quotes.toscrape.com/login)
  4.  
  5. “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
  6.  
  7. by Albert Einstein
  8.  
  9. Tags: change deep-thoughts thinking world
  10.  
  11. “It is our choices, Harry, that show what we truly are, far more than our abilities.”
  12.  
  13. by J.K. Rowling

An Evidence Record for every result

Every result says where it came from: the URL as fetched, when, the robots.txt decision, the lane, and SHA-256 hashes of what was read and of the Markdown and JSON delivered. Change one character of the page and see.

<h1>A Light in the Attic</h1>
<p class="price_color"></p>
<p class="instock availability">…
SHA-256 of the page’s 9,279 bytes, here in your browser
a6e572bec156bf80ff3149b89b6d218cdcf8866ccc26ccf69a5431db8e142c6a
rawSha256 in the Evidence Record
a6e572bec156bf80ff3149b89b6d218cdcf8866ccc26ccf69a5431db8e142c6a

Matches: this is the page that was read.

Checked fields, never guessed

Ask for fields by name. Each value says where on the page it was read; a field the page doesn’t state comes back null, with the reason. Add one it doesn’t have:

  1. {
  2. "title": "A Light in the Attic"✓ h1[0]
  3. "price": "51.77"✓ p.price_color
  4. }

FROM YOUR CODE

The same scrape,
one call away.

Hosted at api.octocrawl.dev and mcp.octocrawl.dev, or on your own computer with npx octocrawl serve. The code follows what you chose above.

WHAT IT COSTS

Pay for pages read.
Never for a block.

ResultCredits
A page read over HTTP1
A page read in a browser5
A page that doesn’t exist (404, 410)1
Blocked, refused, or nothing read0

Without a key: 20 pages a day per address, over HTTP. A free account: 1,000 credits a month and API keys. On your own computer: no limit. Free tiers · Limits

Your month

1,400 credits a month

1,400 of 1,000

Now scrape a page of your own.

It runs on the free tool page: five pages a day, no sign-up.

FAQ

Scraping,
answered plainly.

What is the difference between scrape, map and crawl?

Scrape reads one page and gives you what it holds. Map lists a site’s URLs from its sitemaps and links without reading each page. Crawl follows links from a start page and scrapes each one; it runs on your own computer with npx octocrawl serve, as does batch, which reads up to 1,000 URLs you give it.

Does it read pages that need JavaScript?

Yes. Octocrawl reads over HTTP first, and when that finds only an empty shell it reads the page in a browser. Hosted, a browser read costs 5 credits and needs an account’s key; without a key, hosted Octocrawl reads over HTTP only. On your own computer it uses a browser with no limit.

Does it respect robots.txt?

Hosted Octocrawl reads the site’s robots.txt first and never requests a page it disallows; the answer says so and costs nothing. On your own computer, a URL you give it yourself is read and the robots.txt decision is recorded in its Evidence Record; crawl and map follow the rules.

Which formats can I ask for?

Markdown, links, HTML, raw HTML, images, tables (also as CSV), a screenshot (read in a browser) and JSON fields from a schema, any of them from one read. The page’s metadata comes with every result.

How are fields read without a model?

From what the page states: JSON-LD, microdata, meta tags, table rows, definition lists and a PDF’s “Label: value” lines. Each value names where it was read. A field the page doesn’t state comes back null with the reason. A model fallback exists, but it is off unless you turn it on.

Can it read PDFs?

Yes: the text layer of a PDF, page by page, with page markers. It does no OCR, so a scanned page without text comes back empty rather than guessed. CSV, JSON and plain text come back as they were served.

What does a blocked page cost?

Nothing. A page blocked by a check, a challenge or robots.txt, and a call that read nothing, cost 0 credits. A page that doesn’t exist (404 or 410) is still the site’s answer and costs 1 credit, like a page read over HTTP; one read in a browser costs 5.

What is in the Evidence Record?

The URL asked for and the one fetched, the redirects, when it was fetched, the HTTP status, the robots.txt decision, the lane, SHA-256 hashes of the bytes read and of the Markdown and JSON delivered, the extractor’s version and where each field was found. See the reference.