Site Audit (Crawler + LLM Citation Readiness)

Site Audit crawls your domain (BFS or via sitemap.xml), generates an issue report for every URL (technical SEO + content hygiene), and then automatically fires a Grounding-Audit batch over up to 100 pages. The findings are added as additional issues with prefix grounding_* into the same issue list — one dashboard view for technical SEO + LLM-Citation-Readiness.

What it does

• Crawl modes — bfs (link discovery from start_url, depth-limited) OR sitemap (loads all URLs from sitemap.xml, recursive incl. sitemap-index, ignores crawl_depth). Hard cap: 10,000 seeds, max_pages enforced.
• Issue classification — fix-priority sorted, filtered by severity (critical|high|medium|low|notice), issue_type, and status (open|fixed|dismissed). Eager-loaded url per issue.
• Auto-Grounding-Bridge — auto-dispatched after crawl completion: Grounding-Audit-Batch over up to 100 pages → findings appear as grounding_*-issues in the same crawl. No extra credits. Idempotent (re-run de-duplicates).
• Close-the-Loop — Mark issues as fixed/dismissed via UI or API — individually OR bulk per issue_type. Mandatory after each fix so the delta comparison metric ("fixed since last crawl") stays meaningful.
• AI brief (5 credits per brief) — narrative LLM explanation per issue on demand.
• Bridge timestamps — bridge_dispatched_at (batch dispatched) and bridge_completed_at (all grounding_* issues final). Deterministic polling condition instead of blind 1-3 min waits.

When to use

• Site-wide technical SEO inventory.
• Check whole-domain LLM citation readiness without 340× single-audit calls.
• Pre-relaunch: what still needs to be removed / fixed.
• Monthly health report for stakeholders.

UI workflow

1. Start — Form at /site-audit (start_url, max_pages, crawl_depth, crawl_mode, optional tracking_project_id). The start address is pre-filled with the active project's domain. A bare domain like example.com is enough: the https:// is added as soon as you leave the field.
2. Polling — Detail page /site-audit/{crawl} shows live counter (pages_crawled, total_issues). Auto-refresh.
3. Close-the-Loop — Per issue: 2 buttons ✓ Mark fixed and ⊘ Dismiss. Above the table: a bulk-action toolbar with dropdown "Select issue type" + Mark all as fixed / Dismiss all for mass fixes after layout/template changes.
4. Re-crawl — After critical+high are fixed, new crawl → trend block shows "fixed since last crawl" / "new since last crawl".

From the "Errors (4xx/5xx)" card to the issue list

The Crawl-Efficiency card on the detail page counts how many crawled pages responded with an error status. The value Errors (4xx/5xx) is a button:

1. Click the value — the Issues tab opens, filtered to the status errors; above it you see "Issue list filtered to the error pages counted by the crawl-efficiency card." Severity and "Open only" are reset, because the card counts pages regardless of whether the issue has already been fixed. The error groups are expanded at the top.
2. The issues are named after what the page does: Page responds with an error status (4xx) and Page responds with a server error (5xx) — the page itself responds that way; it is not about a link on another page. "Page unreachable (timeout)" and "Rate limited (429)" belong to the same counter.
3. Two separate filters. Open only in the Issues tab hides fixed and dismissed issues; Status code in the Pages tab filters the crawled pages by their HTTP response (200, redirect, 404, 5xx). Both appear separately in the address — ?issue_status=open and ?http_status=404 — and do not affect each other: a 404 filter in the Pages tab no longer clears "Open only" in the issue list. A filtered address can be bookmarked or shared as a link.

API workflow (skill caller)

Mandatory sequence after fix:

# 1. Fetch issues
ISSUES=$(curl -sH "Authorization: Bearer $TOKEN" \
    "$BASE/v1/site-audit/$CRAWL/issues?status=open&per_page=100" | jq '.data')

# 2. Per issue: apply fix (layout edit / content update / migration)

# 3. MANDATORY: mark issue as fixed — otherwise the platform doesn't know
#    the work is done and the delta metric becomes useless:
curl -X PATCH \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"status":"fixed"}' \
    "$BASE/v1/site-audit/issues/$ISSUE_ID"

# 4. For layout/template fixes: bulk-mark instead of 50× single PATCH:
curl -X PATCH \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"filter":{"issue_type":"missing_alt_text","status":"open"},"new_status":"fixed"}' \
    "$BASE/v1/site-audit/$CRAWL/issues/bulk"

Polling pattern (deterministic since 2026-05-14):

while true; do
    R=$(curl -sH "Authorization: Bearer $TOKEN" "$BASE/v1/site-audit/$CRAWL" | jq '.data')
    STATUS=$(echo "$R" | jq -r .status)
    BRIDGE=$(echo "$R" | jq -r .bridge_completed_at)
    [ "$STATUS" = "completed" ] && [ "$BRIDGE" != "null" ] && break
    sleep 30
done
# → ALL issues are now final, including grounding_*

Anti-patterns

• Applying a fix without PATCH /issues/{id} with status=fixed → issue stays "open" in dashboard, delta comparison metric of next crawl iteration becomes useless.
• Querying issues right after status='completed' without checking bridge_completed_at → grounding_* issues missing (bridge still running).
• Bulk-mark without filter.issue_type → marks hundreds of unrelated issues. Server returns meta.applied_filter; caller MUST verify before next step.
• AI brief in a loop for 100+ issues — 5 credits per brief. Only for selected critical items.

Related modules

• Content Audit (Site Crawl) — content-quality scanner (what pages say, not what pages have). Complementary.
• Grounding Audit — single-URL LLM-citation-readiness. Direct call without crawl overhead.
• Page Deep Audit — Conversion Audit per URL (Lighthouse + Vision + persona fit). More granular per URL.