Blog

Web Scraping for RAG: Markdown Chunks That Keep Sources

Scrape web pages into clean Markdown for RAG, split them into chunks that keep the source URL, fetch time and a text hash, and skip pages that did not change.

Octocrawl teamOctober 10, 2026