AI document chatbot architecture: the decisions that matter

Situation: RAG needs nearest-neighbour search, but per-tenant isolation, deletion on request and low operating cost matter more than the last millisecond at phase-1 scale.

Choice: Store chunks and embeddings in the tenant database with an HNSW index, filter by collection on every query and combine with GIN full-text search; partition large collections and keep a retrieval interface so the store can move later.

• Rejected: Managed vector database: strong scale but a second system, cross-tenant isolation to design and per-use billing.
• Rejected: IVFFlat index: smaller build but needs training data and degrades with churn.
• Rejected: Brute-force scan: exact but O(n) per query.
• Consequence: Isolation and erasure follow the tenant database.
• Consequence: Index memory is a real cost, so chunk and dimension quotas are enforced.
• Consequence: Filtered ANN needs iterative scan and ef_search tuning, validated against the golden set.

Situation: Model calls bill per token with no ceiling; one abusive user, prompt loop or bug can produce a large bill before any dashboard alert is read.

Choice: Every call atomically reserves estimated tokens and USD against both the user and org counters; no reservation means no call; settlement records actual usage and a reaper releases abandoned reservations; defaults apply when no limit is configured.

• Rejected: Post-hoc metering with alerts: cannot stop spend, only report it.
• Rejected: Gateway-level limits only: protects the account but not per-user fairness inside an app.
• Rejected: Pre-paid credits only: safe but blocks plans that include a monthly allowance.
• Consequence: Users sometimes see a 'budget reached' message instead of an answer, which is the intended behavior.
• Consequence: Estimates are conservative so a small portion of budget is briefly held.
• Consequence: Counter rows are hot, so they are single-statement updates and can be sharded later.

Situation: Pure vector search misses exact terms (SKUs, names, error codes); pure keyword search misses paraphrases; weak retrieval makes the model guess.

Choice: Fuse vector top-40 and full-text top-40 with RRF in one query, send a bounded top-N within a context budget, and refuse to answer from thin evidence when the fused score is below the threshold.

• Rejected: Vector-only: simpler but poor on exact-match queries.
• Rejected: Keyword-only: cheap but poor on meaning.
• Rejected: Always answer from model knowledge: fluent but ungrounded.
• Consequence: Quality is measured with recall@k and groundedness on a golden set.
• Consequence: Some questions get 'not found in your documents', which is shown honestly.
• Consequence: A reranker can be added later behind the same interface.

Situation: Documents are large and untrusted, parsing and embedding are slow and billable, and users re-upload the same files.

Choice: Presigned upload, then staged jobs (parse, chunk, embed) with SKIP LOCKED, sandboxed parsing, content-hash dedupe at document and chunk level, batch embedding under an org budget and resumable stages.

• Rejected: Inline processing on upload: times out and exposes the web tier.
• Rejected: Re-embed everything on change: simple but wastes embedding spend.
• Rejected: Third-party ingestion pipeline: extra vendor and uncapped usage fees.
• Consequence: Documents show clear per-stage status and errors.
• Consequence: Re-uploads and edits cost only the changed chunks.
• Consequence: Embedding model changes require a new collection and a budgeted reindex.

Indexes are chosen from the questions the app asks most often. Each line below names a table and the question its main index answers.

• conversations: (owner_id, updated_at DESC, id) — conversation list keyset page.
• messages: (conversation_id, created_at, id) — load a thread in order and build the history window.
• collections: (org_id, name) — list knowledge bases for an org.
• documents: (collection_id, created_at DESC) — document list keyset page.
• chunks: (embedding) — HNSW vector_cosine_ops (m=16, ef_construction=64) for approximate nearest neighbour.
• ingest_jobs: (status, run_at) where status = 'queued' — worker claim with SKIP LOCKED.
• usage_events: unique (request_id, kind) — metering is idempotent per call, retries never double count.
• usage_daily: unique (day, org_id, user_id) — rollup upsert is idempotent.

Forge64 matches your description to one of 25 reference architectures, writes the plan and data model into the project, generates the app, and runs a quality gate over the code before you see it. Each app gets its own Postgres database and sign-in with no per-user fee. Hosting on Forge Cloud and Forge Payments are rolling out.

This is a reference design for the category, not a promise that every item is generated today. Built-ins such as background jobs, file uploads and search are rolling out; email sent from your app is coming soon, so anything here that needs it is something you add yourself for now.

What is the most important architecture decision for an AI document chatbot?

pgvector with HNSW inside the tenant Postgres, not a separate vector database. Store chunks and embeddings in the tenant database with an HNSW index, filter by collection on every query and combine with GIN full-text search; partition large collections and keep a retrieval interface so the store can move later.

Do I have to make these decisions myself?

No. Forge AI plans them from your description and writes the decisions into the project as architecture records you can read and change.

What do the alternatives cost me?

For example, for "Reserve-then-settle budgets that fail closed before the provider call": Post-hoc metering with alerts: cannot stop spend, only report it. Gateway-level limits only: protects the account but not per-user fairness inside an app.

• How to build an AI document chatbot
• Example prompt for an AI document chatbot
• AI document chatbot security checklist
• AI document chatbot scaling checklist
• Often combined: SaaS dashboard
• Often combined: CRM
• All app types