Icon

AI

Icon

RAG Pipeline Architecture in Production: Our pgvector Build

·

By Chandresh Patidar

Kraftbase thumbnail: RAG Pipeline Architecture in Production: Our pgvector Build

Let the model choose, but only from what the search returned.

One of our clients runs a wellness app: guided meditation, breathwork and yoga programs, plus live classes. Members use the mobile app, and the client's team runs the catalog from an admin panel. They wanted an AI companion in the app that people could ask for something to do right now, like "something short for anxiety before bed".

That meant a RAG pipeline that recommends from a real catalog, not one that answers questions from documents. Every answer becomes a card someone taps to play, so an invented title is a broken screen. This post covers what the assistant had to do, the decisions we made and why, and how the pieces fit together.

a blue background with lines and dots

What we needed

  • Real items only. Every suggestion has to exist, be published and have something playable, because it turns into a card.

  • Personal answers. Skip what someone has already finished, never suggest what they said they don’t like, and take their mood and history into account.

  • A reply that streams to the app while the person waits.

  • No dead ends. New content with no embedding yet, or an outage at the embedding provider, shouldn’t break the chat.

  • A small team and the Postgres database the app already had. No appetite for new infrastructure.

We haven’t measured latency, cost per turn or retrieval accuracy in production, so this post doesn’t claim any.

Where we started

An earlier version had its own scorecard: a deep evaluation of memory and recommendations that marks each check pass, fail or blocked. On 1 October 2026 it scored 11 pass, 7 fail and 4 blocked.

The failures were the kind users notice. Dislikes weren’t stored in a form search could use, so a program could come straight back after someone said “not for me”. The similarity checks were blocked rather than passed, because the test catalog was too thin to measure them, and the evaluator won’t invent data to make a number look healthy. Several of the decisions below came out of that run.

The decisions

One embedding per item, kept in Postgres

Most RAG guides start with chunking long documents. Our “documents” are catalog items: a program or a live class, which a person picks whole. So each item gets exactly one embedding, built from its title, description, category, length and teachers. Chunking would only have meant ranking fragments and working out which program each one belonged to.

The vectors live in Postgres with the pgvector extension, on the same rows as the programs. There’s no separate vector database to keep in sync, and an unpublished program can’t linger in an index somewhere else. The trade-off: vector search shares the database with everything else the app does.

Rules go in the query, not the prompt

Some rules must never break. Only published items. Only items with something playable. Nothing the person has finished or dismissed, and nothing from a category they dislike.

We put all of those into the same database query as the vector search. A program with no playable video can’t come back from search at all, however close a match it is. A rule in the prompt is a request the model can ignore. A rule in the query means the item never reaches the model.

This is also where the scorecard’s dislike failure got fixed. Dislikes are now stored as IDs of programs, teachers and categories, and every search excludes them, whatever the model asks for.

Make the model search before it answers

The assistant uses a small model, gpt-4o-mini, with search exposed as tools. Left to decide for itself, a small model will sometimes answer a recommendation from general knowledge, with plausible programs that don’t exist.

So for a recommendation, the first step is always a search. The model gets at most three rounds, and the last one can’t call tools, so a turn always ends with an answer. Spotting what someone is asking for (a recommendation, a live class, a teacher) is done with simple pattern matching, not another model call. It’s crude, but it’s instant and free.

Never trust the model’s IDs

The model replies with the IDs of the items it suggests and a reason for each. The server then keeps only IDs that a search actually returned in that turn. It checks each one against the database again (still published, still playable) and shows the database’s title, never the model’s.

If the model’s answer is broken, or none of its picks survive, the server falls back to what the search found, rather than show nothing. The model can phrase the reply, but it can’t invent what’s on a card. Retrieval narrows what’s possible, and the server decides what’s shown.

Degrade instead of failing

Embeddings are made in the background when someone on the client’s team adds or edits a program, so a brand-new item can exist before its vector does. If vector search finds nothing or fails, search falls back to a keyword match, and then to the newest playable programs. Live classes fall back to the published schedule.

New content is findable straight away, and an embedding outage lowers the quality of answers without breaking the chat.

How it works

RAG pipeline overview, drawn top to bottom: when content changes it is embedded into Postgres with pgvector; when a user asks, the request is understood, a small model searches in up to three rounds, the server checks every ID, and playable cards stream back

Two halves: content is embedded when it changes, and every question is answered only from what search returns.

  1. Someone on the client’s team adds or edits a program or live class, and its text is embedded in the background. A backfill script catches anything that was missed.

  2. A member asks for something in the app.

  3. The server works out what kind of request it is and loads their profile, mood, recent chat and what they’ve finished or dismissed.

  4. The model searches the catalog, matching on meaning and on the hard rules in one query, for up to three rounds.

  5. The server checks every suggested ID, swaps in the database’s titles and streams the reply back with playable cards.

What it’s like to run

The server runs in Docker next to Redis, which counts each person’s requests against an hourly limit. The deploy pipeline applies database changes twice to a throwaway database, to prove they’re safe to run again, before it ships anything. There’s no cache yet, so every turn calls the model.

Testing has three layers. A mocked suite checks the rules that must never break, like invented IDs or excluded items reaching the screen. A live run uses real model calls against the real catalog with temporary test users. And the deep scorecard checks memory and recommendations. We ran the mocked suite on 5 November 2026: 27 of 28 tests passed. The one failure was a test whose expected query had drifted from the code, while the fallback it covers still ran. We haven’t rerun the deep scorecard since the fixes, because it needs real model calls against a real database.

What we learned

  • Retrieval isn’t the hard part. Trusting the output is. The server check did more for answer quality than any prompt change.

  • Put rules where they can’t be ignored. In the query, not the prompt.

  • A scorecard that can say “blocked” is worth more than one that always passes.

  • What we’d add next: a queue with retries for embeddings, a cache for repeated questions, and embeddings for teacher search, which only matches keywords today.

When this design fits

This shape suits an assistant that recommends from a catalog: products, courses, listings, content. There the unit someone picks is the unit you retrieve, and every answer has to point at something real. If your answers come from long documents, you’ll need chunking and passage-level retrieval instead. If vector search starts competing with the app’s own queries, that’s the point to move vectors to their own database.

Related reading

On the interface side of the same problem, see our posts on AI chatbot UX best practices, AI agent UX patterns and designing trust into AI interfaces. For another production system we built, read our scalable notification system architecture.

FAQ

1. Do you always need chunking for RAG?

No. Chunk when documents are long and the answer sits in a small part of one. Our "documents" are catalog items that people pick whole, so we embed each item once. Chunking would only make it harder to map a match back to something playable.

2. Do you need a separate vector database?

Not for us. pgvector in the Postgres the app already used means one source of truth, filters in the same query as the search, and nothing to sync when a program is unpublished. We'd move vectors out only if search started slowing down the app's normal queries.

3. How do you stop a RAG chatbot from hallucinating?

Don't rely on the prompt alone. We make the model search before it recommends, accept only IDs a search returned in that turn, check them in the database again, and show the database's titles. The model can phrase the answer, but it can't invent what's on the card.

4. What similarity threshold should you use?

We return the top 5 matches above a cosine similarity of 0.3, with the hard rules in the same query. Treat that as a starting point, not a rule: the right value depends on your embedding model and your text. Tune it against an evaluation set, not by eye.

5. What happens when embeddings are missing?

Search falls back to a keyword match on titles and descriptions, and then to the newest playable items. New content is findable before its embedding exists, and an outage at the embedding provider lowers answer quality instead of breaking the feature.

6. How do you evaluate a RAG pipeline?

Split it. A mocked suite checks the rules that must never break, like invented or excluded items reaching the screen. Live runs against the real catalog check behavior end to end. When the data can't support a metric, mark it blocked instead of calling it a pass.

AI engineering

Adding AI search or recommendations to your product?

Tell us what users should find. We'll show you how we'd ground it in your own data.

Let the model choose, but only from what the search returned.

One of our clients runs a wellness app: guided meditation, breathwork and yoga programs, plus live classes. Members use the mobile app, and the client's team runs the catalog from an admin panel. They wanted an AI companion in the app that people could ask for something to do right now, like "something short for anxiety before bed".

That meant a RAG pipeline that recommends from a real catalog, not one that answers questions from documents. Every answer becomes a card someone taps to play, so an invented title is a broken screen. This post covers what the assistant had to do, the decisions we made and why, and how the pieces fit together.

a blue background with lines and dots

What we needed

  • Real items only. Every suggestion has to exist, be published and have something playable, because it turns into a card.

  • Personal answers. Skip what someone has already finished, never suggest what they said they don’t like, and take their mood and history into account.

  • A reply that streams to the app while the person waits.

  • No dead ends. New content with no embedding yet, or an outage at the embedding provider, shouldn’t break the chat.

  • A small team and the Postgres database the app already had. No appetite for new infrastructure.

We haven’t measured latency, cost per turn or retrieval accuracy in production, so this post doesn’t claim any.

Where we started

An earlier version had its own scorecard: a deep evaluation of memory and recommendations that marks each check pass, fail or blocked. On 1 October 2026 it scored 11 pass, 7 fail and 4 blocked.

The failures were the kind users notice. Dislikes weren’t stored in a form search could use, so a program could come straight back after someone said “not for me”. The similarity checks were blocked rather than passed, because the test catalog was too thin to measure them, and the evaluator won’t invent data to make a number look healthy. Several of the decisions below came out of that run.

The decisions

One embedding per item, kept in Postgres

Most RAG guides start with chunking long documents. Our “documents” are catalog items: a program or a live class, which a person picks whole. So each item gets exactly one embedding, built from its title, description, category, length and teachers. Chunking would only have meant ranking fragments and working out which program each one belonged to.

The vectors live in Postgres with the pgvector extension, on the same rows as the programs. There’s no separate vector database to keep in sync, and an unpublished program can’t linger in an index somewhere else. The trade-off: vector search shares the database with everything else the app does.

Rules go in the query, not the prompt

Some rules must never break. Only published items. Only items with something playable. Nothing the person has finished or dismissed, and nothing from a category they dislike.

We put all of those into the same database query as the vector search. A program with no playable video can’t come back from search at all, however close a match it is. A rule in the prompt is a request the model can ignore. A rule in the query means the item never reaches the model.

This is also where the scorecard’s dislike failure got fixed. Dislikes are now stored as IDs of programs, teachers and categories, and every search excludes them, whatever the model asks for.

Make the model search before it answers

The assistant uses a small model, gpt-4o-mini, with search exposed as tools. Left to decide for itself, a small model will sometimes answer a recommendation from general knowledge, with plausible programs that don’t exist.

So for a recommendation, the first step is always a search. The model gets at most three rounds, and the last one can’t call tools, so a turn always ends with an answer. Spotting what someone is asking for (a recommendation, a live class, a teacher) is done with simple pattern matching, not another model call. It’s crude, but it’s instant and free.

Never trust the model’s IDs

The model replies with the IDs of the items it suggests and a reason for each. The server then keeps only IDs that a search actually returned in that turn. It checks each one against the database again (still published, still playable) and shows the database’s title, never the model’s.

If the model’s answer is broken, or none of its picks survive, the server falls back to what the search found, rather than show nothing. The model can phrase the reply, but it can’t invent what’s on a card. Retrieval narrows what’s possible, and the server decides what’s shown.

Degrade instead of failing

Embeddings are made in the background when someone on the client’s team adds or edits a program, so a brand-new item can exist before its vector does. If vector search finds nothing or fails, search falls back to a keyword match, and then to the newest playable programs. Live classes fall back to the published schedule.

New content is findable straight away, and an embedding outage lowers the quality of answers without breaking the chat.

How it works

RAG pipeline overview, drawn top to bottom: when content changes it is embedded into Postgres with pgvector; when a user asks, the request is understood, a small model searches in up to three rounds, the server checks every ID, and playable cards stream back

Two halves: content is embedded when it changes, and every question is answered only from what search returns.

  1. Someone on the client’s team adds or edits a program or live class, and its text is embedded in the background. A backfill script catches anything that was missed.

  2. A member asks for something in the app.

  3. The server works out what kind of request it is and loads their profile, mood, recent chat and what they’ve finished or dismissed.

  4. The model searches the catalog, matching on meaning and on the hard rules in one query, for up to three rounds.

  5. The server checks every suggested ID, swaps in the database’s titles and streams the reply back with playable cards.

What it’s like to run

The server runs in Docker next to Redis, which counts each person’s requests against an hourly limit. The deploy pipeline applies database changes twice to a throwaway database, to prove they’re safe to run again, before it ships anything. There’s no cache yet, so every turn calls the model.

Testing has three layers. A mocked suite checks the rules that must never break, like invented IDs or excluded items reaching the screen. A live run uses real model calls against the real catalog with temporary test users. And the deep scorecard checks memory and recommendations. We ran the mocked suite on 5 November 2026: 27 of 28 tests passed. The one failure was a test whose expected query had drifted from the code, while the fallback it covers still ran. We haven’t rerun the deep scorecard since the fixes, because it needs real model calls against a real database.

What we learned

  • Retrieval isn’t the hard part. Trusting the output is. The server check did more for answer quality than any prompt change.

  • Put rules where they can’t be ignored. In the query, not the prompt.

  • A scorecard that can say “blocked” is worth more than one that always passes.

  • What we’d add next: a queue with retries for embeddings, a cache for repeated questions, and embeddings for teacher search, which only matches keywords today.

When this design fits

This shape suits an assistant that recommends from a catalog: products, courses, listings, content. There the unit someone picks is the unit you retrieve, and every answer has to point at something real. If your answers come from long documents, you’ll need chunking and passage-level retrieval instead. If vector search starts competing with the app’s own queries, that’s the point to move vectors to their own database.

Related reading

On the interface side of the same problem, see our posts on AI chatbot UX best practices, AI agent UX patterns and designing trust into AI interfaces. For another production system we built, read our scalable notification system architecture.

FAQ

1. Do you always need chunking for RAG?

No. Chunk when documents are long and the answer sits in a small part of one. Our "documents" are catalog items that people pick whole, so we embed each item once. Chunking would only make it harder to map a match back to something playable.

2. Do you need a separate vector database?

Not for us. pgvector in the Postgres the app already used means one source of truth, filters in the same query as the search, and nothing to sync when a program is unpublished. We'd move vectors out only if search started slowing down the app's normal queries.

3. How do you stop a RAG chatbot from hallucinating?

Don't rely on the prompt alone. We make the model search before it recommends, accept only IDs a search returned in that turn, check them in the database again, and show the database's titles. The model can phrase the answer, but it can't invent what's on the card.

4. What similarity threshold should you use?

We return the top 5 matches above a cosine similarity of 0.3, with the hard rules in the same query. Treat that as a starting point, not a rule: the right value depends on your embedding model and your text. Tune it against an evaluation set, not by eye.

5. What happens when embeddings are missing?

Search falls back to a keyword match on titles and descriptions, and then to the newest playable items. New content is findable before its embedding exists, and an outage at the embedding provider lowers answer quality instead of breaking the feature.

6. How do you evaluate a RAG pipeline?

Split it. A mocked suite checks the rules that must never break, like invented or excluded items reaching the screen. Live runs against the real catalog check behavior end to end. When the data can't support a metric, mark it blocked instead of calling it a pass.

AI engineering

Adding AI search or recommendations to your product?

Tell us what users should find. We'll show you how we'd ground it in your own data.