Knowledge Base

The Knowledge Base is the per-project knowledge store that article generation draws factually correct, company-specific information from instead of guessing. You fill it three ways: file upload (PDF, DOCX, Markdown, TXT), a single URL, or an automatic website analysis that crawls an entire domain and synthesizes structured knowledge documents from it. Every document is split into semantic text blocks (chunking) and indexed with embeddings. When searching, the system combines embedding similarity (cosine) with classic keyword matching and lets Claude Haiku do a final relevance re-sort (reranking). That way search also finds answers that share no words with the question. In article generation (AI Content Editor) the matching excerpt automatically lands in the prompt as "KNOWLEDGE BASE (binding sources)".

What it does

• File upload — PDF, DOCX, Markdown, TXT up to 10 MB. Text extraction, chunking and embedding run asynchronously in the background.
• URL ingest — read in a single web page by URL. Fetch via ScraperAPI, SSRF protection blocks private/internal addresses, content is extracted to Markdown (navigation, header, footer, scripts stripped).
• Website analysis — an agentic 5-phase pipeline: crawl the domain (sitemap first, BFS fallback), classify subpages by LLM (language-agnostic), extract the most relevant pages and synthesize up to 9 structured knowledge documents (company profile, products, target audience, brand voice, USPs, FAQ, social proof, contact, site overview).
• Hybrid search — embedding cosine similarity plus keyword matching, merged and finally sorted by Claude Haiku reranking. Finds paraphrases that share no words with the query.
• Automatic RAG usage — article generation and storyline sections automatically pull relevant chunks as binding context when needed.
• Download — download the original file (upload) or the extracted/synthesized Markdown (URL ingest and website analysis).
• Asynchronous processing — ingest, URL fetch and website analysis all run as background jobs; every start endpoint answers with 202 and an ID to poll.

When to use it

• Generated articles should contain company-specific facts (products, pricing, USPs, FAQ) instead of the AI inventing them.
• You have reference material (data sheets, internal docs) and want to use it as a content source.
• You want to ingest an entire website as a knowledge base instead of pulling every subpage individually via URL ingest.
• Storylines sections or individual articles should consistently draw from the same internal sources.

Workflow

1. Start the ingest — raw text via POST /v1/projects/{project}/knowledge-base {title, content} or a web page via POST /v1/knowledge-base/url {url}. Both run asynchronously, response 202 {document_id, status: 'processing'}.
2. Poll status — GET /v1/projects/{project}/knowledge-base until the document shows status: 'ready'. Only then are the chunks created and embedded; failed means an extraction or chunking error.
3. Search — GET /v1/projects/{project}/knowledge-base/search?q=…&top_k=… returns the most relevant chunks (hybrid cosine + keyword, Haiku-reranked) with a score.
4. Bulk: an entire website — instead of many individual URLs, start POST /v1/projects/{project}/knowledge-base/site-analysis {url, max_pages?}, then poll GET /v1/knowledge-base/site-analysis/{id} (progress_percent 0–100) until status ∈ {completed, partial, failed}.
5. Wrap-up — no separate call is needed for article generation: use_knowledge_base + knowledge_mode (all/specific) + knowledge_document_ids[] in the generation request of AI Content Editor automatically uses the same retrieval pipeline.

API

Example:

POST /v1/projects/42/knowledge-base/site-analysis
{"url": "https://example.com", "max_pages": 40}

Credits & limits

• File upload / URL ingest / text ingest: 5 credits, charged only once chunking succeeds (status: ready) — no charge on failed.
• Search: 1 credit per query, charged only on a 2xx response.
• Website analysis: 25 credits flat, covers crawl, classification, synthesis and ingest of all resulting documents. Charged only on completed/partial, not on failed.
• File upload limit: 10 MB per file.
• Website analysis limit: max_pages 5–80 (default 40); only one analysis runs per project+host at a time (409 on a parallel start).
• All start endpoints are asynchronous — fetch the result by polling the respective status endpoints.

Related modules

• AI Content Editor — consumes the knowledge base as RAG context during article generation.
• Storylines — section generation uses the same retrieval pipeline for consistent sources across the whole cluster.