Imported from Sachin0496/brand-ai-readiness-audit (skills/crawl-reachability/SKILL.md). Install upstream with npx skills add Sachin0496/brand-ai-readiness-audit --skill crawl-reachability. Copyright stays with the author (MIT).

Crawl & Reachability

The first gate in the pipeline, and the one most likely to make everything downstream moot. A page that cannot be retrieved, or that arrives without its content, cannot be selected as a source no matter how good the content is.

arXiv:2604.25707 §4.1 reports a 76.44% fetch success rate across pages that answer engines had already chosen to cite. Roughly one in four selected pages could not be retrieved. Reachability is not a hygiene item; it is a hard gate with a measured failure rate.

When to use

After url-validation clears the target. Run it standalone to answer "can a crawler read this site at all" or "does this page render server-side".

Inputs

Input	Required	Notes
`--workspace`	no	Run directory. Defaults to the most recent run.
`--max-pages`	no	Overrides the preflight page budget.
`--json`	no	Machine-readable output.

Reads 01-url-validation.json for the target, scope, robots policy and budget. Refuses to run if preflight has not cleared.

Procedure

Discover. robots.txt Sitemap: directives, then the conventional sitemap paths, expanding at most three sitemap indexes; fall back to a bounded same-scope homepage crawl when discovery yields fewer than 12 URLs. Probe for llms.txt.
Sample. Round-robin across page types so the sample spans product, pricing, article, docs, about and hub pages rather than 20 blog posts. This is what makes a later population claim defensible.
Fetch, honouring robots.txt per URL and the crawl-delay, saving each page's HTML into the workspace for downstream skills. No JavaScript is executed — deliberately, because that is what a crawler sees.
Audit the sitemap in depth (see the table below).
Diagnose render faults per page.
Check per-page reachability: status, canonical correctness, X-Robots-Tag, noindex, soft-404s, redirect behaviour.
Write 02-crawl-reachability.json including the page inventory.

Run it:

python3 scripts/check_reachability.py [--workspace PATH] [--max-pages N] [--json]

Checks performed

Sitemap

Check id	Detects	Severity	Claim level
`sitemap_missing`	No retrievable sitemap	high	1
`sitemap_not_declared`	Sitemap exists but robots.txt does not point to it	medium	1
`sitemap_malformed`	Bad namespace, unwrapped `<loc>`, over size limits	medium	1
`sitemap_empty`	Zero `<url>` entries	high	1
`sitemap_broken_entries`	Listed URLs returning 4xx/5xx	high/medium	1
`sitemap_redirect_heavy`	Over 30% of sampled entries redirect	medium	1
`sitemap_orphans`	Crawlable pages missing from the sitemap	high/medium	1
`sitemap_off_scope`	Entries outside the serving host	medium	1
`sitemap_scheme_mismatch`	`http` entries on an `https` site	medium	1
`sitemap_bad_lastmod`	`lastmod` not a valid W3C datetime	low	1
`sitemap_future_lastmod`	`lastmod` in the future	medium	1
`sitemap_stale`	Median `lastmod` older than a year	medium	1
`sitemap_uniform_lastmod`	Every entry shares one `lastmod`	low	1

Render faults

Check id	Detects	Recoverable	Severity
`empty_shell`	Empty mount point, no server-rendered content	no	critical
`state_blob`	Content present only inside a client state payload	yes	high
`script_heavy_shell`	Script-dominated document, nothing recoverable	no	high
`thin_served_html`	Visible text under 5% of served bytes	no	high
`meta_only`	Metadata renders, body does not	no	medium
`lazy_content`	Primary content deferred behind loading placeholders	no	medium
`client_routing`	Hash-fragment routes; sub-pages have no own URL	no	high
`noscript_gap`	Framework present, no `<noscript>` fallback	no	medium

The state_blob / empty_shell distinction is the point of this section. Both look identical to a browser-free reader, but the fixes differ by an order of magnitude of effort: state_blob means the text already ships with the page and needs hydrating into markup; empty_shell means it never left the server.

Page reachability

Check id	Detects	Severity	Claim level
`page_unreachable`	Sampled pages returning errors	high	1
`noindex_on_public_page`	`noindex` on a page that should be indexed	high	1
`x_robots_blocks`	`X-Robots-Tag` blocking indexing or AI agents	high	1
`canonical_mismatch`	`rel=canonical` pointing elsewhere unexpectedly	medium	1
`canonical_missing`	No `rel=canonical` on a site with URL variants	low	2
`soft_404`	200 response whose content says "not found"	medium	2
`slow_response`	Median response time over 3 seconds	medium	2

Output

02-crawl-reachability.json, including the page inventory downstream skills read:

{
  "pages": [ { "url": "...", "page_type": "product", "status": 200, "html_path": "pages/003.html" } ],
  "discovery": { "sitemap_found": true, "candidates": 214, "llms_txt": false },
  "sitemap": { ...deep audit... },
  "render": { "pages_with_blocking_faults": 3, "fault_counts": { ... } },
  "findings": [ ... ]
}

Guardrails

Read-only. robots.txt is enforced per URL, not merely reported. Crawl-delay is honoured. Requests are rate-limited and the page, byte and wall-clock budgets are hard limits. Never authenticates, never submits a form, never leaves the registrable domain.

Grounding

Fetch success rate 76.44% across cited pages (§4.1) — reachability is a measured failure mode, not a theoretical one.
Search-trigger rates 98.64% / 99.67% / 100% (§6) — retrieval is attempted on nearly every prompt, so a retrieval-time failure is a citation failure.
Handout appendix C — content assembled after load is invisible to a reader that does not execute scripts.

crawl-reachability

About