GitHub

Free Link Extractor: find all pages on a website

Paste a site and get its pages from its sitemap and its links, and where each was found. Or just one page’s links. Copy them or download a CSV. Free, no sign-up.

5 free previews a day · Public pages only · Use it from your agent

Get code

Use it in your code or agent

The same page, format and options, through hosted Octocrawl (api.octocrawl.dev and mcp.octocrawl.dev: no key within a daily allowance, a key for more and the browser lane) or on your own computer with no limit (npx octocrawl serve, npx -y @octocrawl/mcp). A browser-lane run may differ from this HTTP preview.

    Advanced reference ↗Connect MCP ↗
    See how it works

    HOW IT WORKS

    From a site to its pages.

    1. Paste a site

      Any public site. Whole site is the default; This page lists one page’s links instead. Five a day, no account.

    2. Octocrawl looks where the site says

      It reads robots.txt, follows its Sitemap lines (else /sitemap.xml, up to 50 files), and reads the start page’s links. Pages robots.txt disallows are left out and counted.

    3. Filter, copy, download

      Each URL says where it was found: the sitemap, a link, or the start page. Filter the list, copy it, or download a .txt or a CSV.

    WHO IT IS FOR

    A site’s pages, in one list.

    What the site itself says it has: its sitemap and the links it shows, with nothing invented.

    1. SEO and site audits

      See every page a site lists in its sitemap, and which ones it only links to. See how to find all pages on a website.

    2. Crawl and RAG lists

      Start a list of pages to read, for a RAG pipeline or an archive, then scrape them with the API or the CLI.

    3. Migrations and inventories

      Take a site’s URLs to a spreadsheet before you move it, and check none go missing.

    WHEN A PAGE CAN’T BE READ

    What it finds, and what it can’t.

    It lists what the site states: a page in neither the sitemap nor the start page’s links isn’t found, because it doesn’t crawl page after page.

    1. No sitemap

      Then only the start page’s links are listed. The result says the site has no readable sitemap.

    2. robots.txt disallows it

      It stops before fetching and says the site’s rules don’t allow it. If robots.txt can’t be reached at all, it stops too, and says that instead.

    3. The site blocks the request

      A verification page, a CAPTCHA or a block page is reported as blocked, never turned into a result about being blocked.

    4. Links drawn by JavaScript

      It reads the start page as the server sends it, so links a script adds aren’t found; the result says when the page looks drawn by JavaScript. The sitemap is read either way.

    FROM YOUR CODE OR AGENT

    The same map, from your own tools.

    Every answer carries its Evidence Record: where the page came from, when, and hashes of what was read.

    • REST API

      Hosted, no key for 20 pages a day; a free account’s key for 1,000 credits a month.

      curl https://api.octocrawl.dev/v1/map -H 'content-type: application/json' -d '{"url":"https://example.com","limit":5000}'API reference

    • MCP

      Claude Code, Cursor, Codex and OpenCode read pages through Octocrawl’s tools.

      claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcpConnect MCP

    • CLI

      On your own computer, with no daily limit, plus crawl and batch.

      npx octocrawl map https://example.comAdvanced reference

    • Playground

      Signed in, list up to 5,000 URLs of a site on your credits: 1 credit a map that finds some.

      octocrawl.dev/dashboard/map/Open the Playground

    MORE FREE TOOLS

    Other things to take from a page.

    URL to MarkdownWebsite to TextScrape APIAll free tools

    FAQ

    Questions,
    answered plainly.

    Is it free? Do I need an account?

    Yes, and no. You get five runs a day without signing up, whole site or one page (at most 150 a day for the whole site), reset at 00:00 UTC. Signed in, the Playground lists up to 5,000 URLs of a site for 1 credit, from 1,000 free credits a month.

    How does it find all pages on a website?

    It reads the site’s robots.txt and the sitemaps it names (or /sitemap.xml), following a sitemap index one level and up to 50 files, and the links on the page you paste. Each URL says where it was found. It doesn’t crawl page after page; for that, run npx octocrawl serve and use crawl. More in how to find all pages on a website.

    How many URLs does it list?

    Up to 500 for a whole site, and up to 500 links for one page. When there are more, the result says how many more it saw. The hosted API’s map and the Playground list up to 5,000.

    Does it respect robots.txt?

    Yes. Pages robots.txt disallows are left out, and the result counts them. It reads every sitemap file only if that file’s own robots.txt allows it, and it never signs in or gets past a verification page.

    Which URLs are left out?

    The same URL twice, other sites’ URLs (subdomains included), files such as images, and pages robots.txt disallows. Each kind is counted in the result. Tracking parameters and #fragments are dropped.

    Can I get one page’s links instead?

    Yes: choose This page above the box. It lists every link on that page, this site’s and the others, up to 500, with a CSV.

    More than five a day? Start free.

    A free account gets 1,000 credits a month for the hosted API, MCP and the Playground. No card. Or run Octocrawl on your own computer, with no limit.

    Start free Run it yourself