Why HTTP 200 is not enough to accept a scraped page
Demonstration recorded 16 October 2026. This is a synthetic fixture demonstration of Octocrawl’s existing result-display logic at source commit bc47bf0e62b07c45b517c2761f1ccb8c7e599073. It is not a live-site success rate, an extraction benchmark or a comparison with another provider.
HTTP 200 says the server returned a successful HTTP response. The response may contain a navigation shell, a verification message, the wrong region of a page or a source-side error instead of the table you need. Accept the content against the task you asked for, and keep that decision separate from the network response.
Octocrawl exposes fetch status, task verification and readiness separately. verification.status: "not_requested" means no task contract was checked. A configured check that passes establishes only that check, not independent accuracy of every field. Readiness can identify content that was not loaded, was not extracted or was not served.
Four controlled results, the same HTTP status
We passed these four synthetic objects to the same taskStatus function the public page uses. The function formats an existing verdict; it does not inspect HTML or perform verification.
| Fixture | Input alongside HTTP 200 / fetch success | Actual label | Acceptance boundary |
|---|---|---|---|
| No contract | verification.status=not_requested |
Page read · Task unverified | Review the needed content or define a task contract |
| Configured checks pass | verification.status=passed, readiness.state=ready |
Configured checks passed | Limited to the configured checks |
| Expected table absent | Failed minTables check observes 0 tables, at least 1 asked; readiness not_extracted |
Configured checks failed | Review region/extraction; another paid fetch is not automatically a fix |
| Source does not serve data | verification.status=passed, readiness not_served, basis source_data_error |
Target data unavailable | A weak passing check cannot establish that the missing target was served |
A blocked result is another case: preserve its access/policy reason instead of treating the block message as target content. A partially useful result needs a separate review decision. A missing field is unknown; a zero is a value that needs evidence in the source.
Reproduce the verdict demonstration
In the Octocrawl source checkout with dependencies installed, run:
node --import tsx --input-type=module <<'JS'
import { taskStatus } from './apps/public-web/src/taskStatus.ts'
const cases = [
{ name: 'not requested', http: 200, status: 'success',
verification: { status: 'not_requested' } },
{ name: 'configured checks pass', http: 200, status: 'success',
verification: { status: 'passed' },
readiness: { state: 'ready', basis: [] } },
{ name: 'target missing', http: 200, status: 'success',
verification: { status: 'failed', checks: [
{ type: 'minTables', passed: false, observed: '0 tables, at least 1 asked' }
] }, readiness: { state: 'not_extracted', basis: [] } },
{ name: 'source did not serve data', http: 200, status: 'success',
verification: { status: 'passed' },
readiness: { state: 'not_served', basis: ['source_data_error'] } }
]
for (const input of cases) {
console.log(input.name, JSON.stringify(taskStatus(input)))
}
JSThe third result includes:
{
"label": "Configured checks failed",
"detail": "The page was served but the target content was not extracted. Review extraction; another paid request is not a fix. Configured checks failed. minTables: 0 tables, at least 1 asked",
"tone": "partial"
}The fourth label is Target data unavailable, even though its synthetic verification says passed. The command above contains the full synthetic inputs and prints their actual outputs. The existing rendered regression suite was also run locally: 13 cases passed, including unverified results, explicit failed checks and unavailable source data. It exercises a browser with injected responses, not independent live acquisition.
Use the distinction in a pipeline
Keep the original result and reason even when it produces no chunks. Send partial results to review. Do not embed the text of a blocked/error page simply because a Markdown field is nonempty. If no task check was requested, record the result as unverified instead of renaming fetch success to “accurate.”
The RAG guide shows source URL, timestamp and hash retention; splitting and embedding stay in your pipeline. The MCP guide shows what to inspect after connecting, and the result-state reference describes the available routes and limits. None of this establishes that every hard site is handled correctly; validate the source region and values your own task requires.
