Site Audit (Crawler + LLM Citation Readiness)

Site Audit crawls your domain (BFS or via sitemap.xml), generates an issue report for every URL (technical SEO + content hygiene), and then automatically fires a Grounding-Audit batch over up to 100 pages. The findings are added as additional issues with prefix grounding_* into the same issue list — one dashboard view for technical SEO + LLM-Citation-Readiness.

What it does

• Crawl modes — bfs (link discovery from start_url, depth-limited) OR sitemap (reads the URLs from sitemap.xml, recursive incl. sitemap-index, ignores crawl_depth). In both modes your plan caps the number of pages per run: Free and Starter 100, Pro 500, Agency and Enterprise 5,000 pages. In sitemap mode the first that many sitemap URLs are checked; the start screen shows you the cap before you start. The automatic baseline check when a tracking project becomes active has its own limit of up to 1,000 pages.
• Issue classification — fix-priority sorted, filtered by severity (critical|high|medium|low|notice), issue_type, and status (open|fixed|dismissed). Eager-loaded url per issue.
• Auto-Grounding-Bridge — auto-dispatched after crawl completion: Grounding-Audit-Batch over up to 100 pages → findings appear as grounding_*-issues in the same crawl. No extra credits. Idempotent (re-run de-duplicates).
• Close-the-Loop — Mark issues as fixed/dismissed via UI or API — individually OR bulk per issue_type. Mandatory after each fix so the delta comparison metric ("fixed since last crawl") stays meaningful.
• AI brief (5 credits per brief) — narrative LLM explanation per issue on demand.
• Bridge timestamps — bridge_dispatched_at (batch dispatched) and bridge_completed_at (all grounding_* issues final). Deterministic polling condition instead of blind 1-3 min waits.

When to use

• Site-wide technical SEO inventory.
• Check whole-domain LLM citation readiness without 340× single-audit calls.
• Pre-relaunch: what still needs to be removed / fixed.
• Monthly health report for stakeholders.

UI workflow

1. Start — Form at /site-audit (start_url, max_pages, crawl_depth, crawl_mode, optional tracking_project_id). The start address is pre-filled with the active project's domain. A bare domain like example.com is enough: the https:// is added as soon as you leave the field. The page count is capped by your plan (Free and Starter 100, Pro 500, Agency and Enterprise 5,000). If the sitemap lists more pages, the start screen says: "Your plan checks up to N pages per run. The sitemap lists M pages; the first N will be checked."
2. Polling — Detail page /site-audit/{crawl} shows live counter (pages_crawled, total_issues). Auto-refresh.
3. Close-the-Loop — Per issue: 2 buttons ✓ Mark fixed and ⊘ Dismiss. Above the table: a bulk-action toolbar with dropdown "Select issue type" + Mark all as fixed / Dismiss all for mass fixes after layout/template changes.
4. Re-crawl — After critical+high are fixed, new crawl → trend block shows "fixed since last crawl" / "new since last crawl".

Large or slow websites

Large or slow websites are checked in several parts in the background; progress keeps moving, and the page shows "Part N of about M". You can leave the page in the meantime. If a run still cannot check every page (very slow website, overall time limit reached), it ends as completed with the badge Ended early and the note "N of M pages checked". The evaluation then covers only the checked pages. Credits are not refunded once pages were checked; a run that could not check a single page fails. It is refunded only when the error was on our side (time limit or a fault in our system) — not when the website is unreachable, rejects automated requests, or you cancel the run yourself.

From the "Errors (4xx/5xx)" card to the issue list

The Crawl-Efficiency card on the detail page counts how many crawled pages responded with an error status. The value Errors (4xx/5xx) is a button:

1. Click the value — the Issues tab opens, filtered to the status errors; above it you see "Issue list filtered to the error pages counted by the crawl-efficiency card." Severity and "Open only" are reset, because the card counts pages regardless of whether the issue has already been fixed. The error groups are expanded at the top.
2. The issues are named after what the page does: Page responds with an error status (4xx) and Page responds with a server error (5xx) — the page itself responds that way; it is not about a link on another page. "Page unreachable (timeout)" and "Rate limited (429)" belong to the same counter.
3. Two separate filters. Open only in the Issues tab hides fixed and dismissed issues; Status code in the Pages tab filters the crawled pages by their HTTP response (200, redirect, 404, 5xx). Both appear separately in the address — ?issue_status=open and ?http_status=404 — and do not affect each other: a 404 filter in the Pages tab no longer clears "Open only" in the issue list. A filtered address can be bookmarked or shared as a link.

API workflow (skill caller)

Mandatory sequence after fix:

# 1. Fetch issues
ISSUES=$(curl -sH "Authorization: Bearer $TOKEN" \
    "$BASE/v1/site-audit/$CRAWL/issues?status=open&per_page=100" | jq '.data')

# 2. Per issue: apply fix (layout edit / content update / migration)

# 3. MANDATORY: mark issue as fixed — otherwise the platform doesn't know
#    the work is done and the delta metric becomes useless:
curl -X PATCH \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"status":"fixed"}' \
    "$BASE/v1/site-audit/issues/$ISSUE_ID"

# 4. For layout/template fixes: bulk-mark instead of 50× single PATCH:
curl -X PATCH \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"filter":{"issue_type":"missing_alt_text","status":"open"},"new_status":"fixed"}' \
    "$BASE/v1/site-audit/$CRAWL/issues/bulk"

Polling pattern (deterministic since 2026-05-14):

while true; do
    R=$(curl -sH "Authorization: Bearer $TOKEN" "$BASE/v1/site-audit/$CRAWL" | jq '.data')
    STATUS=$(echo "$R" | jq -r .status)
    BRIDGE=$(echo "$R" | jq -r .bridge_completed_at)
    [ "$STATUS" = "completed" ] && [ "$BRIDGE" != "null" ] && break
    sleep 30
done
# → ALL issues are now final, including grounding_*

Anti-patterns

• Applying a fix without PATCH /issues/{id} with status=fixed → issue stays "open" in dashboard, delta comparison metric of next crawl iteration becomes useless.
• Querying issues right after status='completed' without checking bridge_completed_at → grounding_* issues missing (bridge still running).
• Bulk-mark without filter.issue_type → marks hundreds of unrelated issues. Server returns meta.applied_filter; caller MUST verify before next step.
• AI brief in a loop for 100+ issues — 5 credits per brief. Only for selected critical items.

Related modules

• Content Audit (Site Crawl) — content-quality scanner (what pages say, not what pages have). Complementary.
• Grounding Audit — single-URL LLM-citation-readiness. Direct call without crawl overhead.
• Page Deep Audit — Conversion Audit per URL (Lighthouse + Vision + persona fit). More granular per URL.