Blog
AI AgentsClaude Code Web Scraping: Add a Web Tool With MCP
Give Claude Code a web scraping tool in one command: it can map a site, read pages as Markdown and cite the source URL and fetch time of every answer.
Octocrawl team
Web Scraping for RAG: Markdown Chunks That Keep Sources
Scrape web pages into clean Markdown for RAG, split them into chunks that keep the source URL, fetch time and a text hash, and skip pages that did not change.
How to Find All Pages on a Website, With Their Sources
List every page of a website from its sitemaps and links in seconds, see where each URL was found, and learn why some pages are left out of the list.
Scrape a Website With Login Without Sharing Your Password
Read pages that need your login in the Chrome you already use. Your password and cookies stay in your browser, and you pass any check yourself, not a bot.
Extract Structured Data From a Web Page, Without AI
Pull fields like price and availability from a web page into JSON, each with the place on the page it was read from, and a reason when a field is missing.
How to Scrape a List of URLs and Keep Every Failure
Scrape a list of up to 1,000 URLs into Markdown files and a CSV, with one row per URL, the reason for every failure and the evidence behind each page.