Octocrawl / Dashboard
Jobs
Hosted batch and crawl jobs on this account: their progress, what they cost and where their results are.
Batch and crawl jobs
| Started | Job | Status | Progress | Credits | Finished | Results |
|---|
No jobs yet. Hosted crawl and batch jobs run on this account's credits, as scrapes do.
To start one, connect an MCP client with this account (Connect) and ask it to crawl a site or batch-scrape a list of URLs: Crawl and batch jobs has the tools and limits. Each job shows here as it runs. On your own computer they run free and without these limits: npx octocrawl serve.
A job reads its pages a slice at a time, each priced as a scrape: 1 credit over HTTP, 5 in a browser. A running job's progress refreshes here every few seconds. Results are kept 7 days after a job finishes.
Read a job's results
Docs →Export on a finished job downloads every page it read as one gzipped JSON Lines file, one result with its Evidence Record a line. The link it opens works for 15 minutes.
MCP, with the job's id
get_batch_items { "id": "<job id>" }
get_crawl_pages { "id": "<job id>" }REST, with a key from API keys
curl https://api.octocrawl.dev/v1/batches/<job id>/items \ -H 'authorization: Bearer oc_YOUR_KEY' curl https://api.octocrawl.dev/v1/crawl/<job id>/pages \ -H 'authorization: Bearer oc_YOUR_KEY' curl https://api.octocrawl.dev/v1/jobs/<job id>/export \ -H 'authorization: Bearer oc_YOUR_KEY'