GitHub

URL to Markdown.

Paste a public page and get clean Markdown for an AI agent, a RAG pipeline or your notes. Free, no sign-up. When a page can’t be read, you get the reason.

5 free previews a day · Public pages only · Use it from your agent

Get code

Use it in your code or agent

The same page, format and options, through hosted Octocrawl (api.octocrawl.dev and mcp.octocrawl.dev: no key within a daily allowance, a key for more and the browser lane) or on your own computer with no limit (npx octocrawl serve, npx -y @octocrawl/mcp). A browser-lane run may differ from this HTTP preview.

    Advanced reference ↗Connect MCP ↗
    See how it works

    HOW IT WORKS

    From a link
    to clean Markdown.

    No install and no account: paste the address and take the text.

    1. Paste a public URL

      Any page anyone can open without signing in: an article, documentation, a product page, a PDF. You get five conversions a day.

    2. Octocrawl checks, then reads

      It reads the site’s robots.txt first, then fetches the page over HTTP and keeps its main content, leaving out menus, headers and footers. Choose Whole page in Options to keep them.

    3. Take the Markdown

      Read it rendered or as source, copy it, or download it as a .md file. Page info shows the final URL and what the page says about itself.

    WHAT IT IS FOR

    Markdown is what
    models read best.

    It keeps a page’s headings, paragraphs, lists, tables and links, and drops the markup, scripts and styling around them.

    1. AI agents and chat

      Give a model the page itself, not a summary of a summary, in far fewer tokens than the raw HTML.

    2. RAG pipelines

      Headings give natural places to split a page into chunks. See web scraping for RAG.

    3. Notes and archives

      Keep a readable copy of an article or a doc page in Obsidian, Notion or a Git repository.

    4. Moving content

      Bring pages from an old site or a wiki into a static site generator or a new docs tool.

    WHEN A PAGE CAN’T BE READ

    An empty file
    is not an answer.

    Many converters return a blank or half-blank result and say nothing. Octocrawl says what happened.

    1. robots.txt disallows the page

      It stops before fetching and says the site’s rules don’t allow it. If robots.txt can’t be reached at all, it stops too, and says that instead.

    2. The site blocks the request

      A verification page, a CAPTCHA or a block page is reported as blocked, never converted into Markdown about being blocked.

    3. The page needs a login

      It says so. This page doesn’t sign in; Octocrawl on your own computer can read pages you’re signed in to, in your own Chrome.

    4. The content is drawn by JavaScript

      This page doesn’t run scripts, so the result can be incomplete, and it is marked so. The hosted API can read such pages in a browser.

    FROM YOUR CODE OR AGENT

    The same conversion,
    from your own tools.

    Hosted Octocrawl answers without a key for 20 pages a day; a free account gives 1,000 credits a month and API keys. Each answer carries its Evidence Record.

    1. REST

      curl https://api.octocrawl.dev/v1/scrape -H 'content-type: application/json' -d '{"url":"https://example.com","formats":["markdown"]}'

    2. Claude Code, Cursor, Codex, OpenCode

      claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp. Connect MCP has the line for each client.

    3. On your own computer

      npx octocrawl serve runs the same API with no limit, and crawls and batches too.

    FAQ

    Questions,
    answered plainly.

    Is it free? Do I need an account?

    Yes, and no. You get five conversions a day without signing up, the same allowance as the preview on the home page (with at most 150 a day for the whole site); it resets at 00:00 UTC. With a free account you get 1,000 credits a month for the hosted API and MCP, and Octocrawl on your own computer has no limit.

    Does it work on pages that need JavaScript?

    This page reads a page over HTTP and does not run its JavaScript, so a page that draws its content in the browser can come back incomplete, and the result says so. With an account, the hosted API reads such pages in a browser (5 credits a page), and Octocrawl on your own computer can use your own Chrome.

    Are tables and links kept?

    Yes. Tables come through as Markdown (GFM) tables, and links keep their addresses. Tables used only to lay out a page aren’t kept as tables, and images stay as Markdown image links; they aren’t downloaded.

    Does it respect robots.txt?

    Yes. It reads the site’s robots.txt before fetching the page, and stops if the page is disallowed, or if robots.txt can’t be reached at all (a server error or no answer); the result then says which. A robots.txt that answers 404 counts as no rules. It never signs in, solves a CAPTCHA or gets past a verification page.

    How big a page can it convert?

    Pages up to 2 MiB and files such as PDFs up to 5 MiB, and it stops after 40 seconds. A very long page returns its first 1,000,000 characters of Markdown, and the result tells you when that happened.

    Can I convert many URLs, or call it from code?

    This page converts one URL at a time. For many pages, call the hosted API (https://api.octocrawl.dev/v1/scrape) or the MCP server from your agent, or run npx octocrawl serve on your own computer, which also crawls a site and reads up to 1,000 URLs in a batch.

    Do you keep the pages I convert?

    No. Your results live in this page and are gone when you leave it. Each conversion logs its state and the host of the page, never its path, as the privacy page says.