Technical Article
Designing an Enterprise RAG Platform: Architecture, Retrieval, and Production Considerations
An engineering-focused look at building an enterprise Retrieval-Augmented Generation platform for document ingestion, semantic retrieval, and LLM-powered knowledge assistance.
Retrieval-Augmented Generation, or RAG, has become one of the most practical ways to bring large language models into enterprise applications.
The basic idea is simple.
Instead of expecting an AI model to already know everything about an organization’s internal knowledge, we retrieve relevant information from enterprise data and provide it to the model as context when answering a question.
In practice, however, building a useful RAG system is not quite as simple as connecting a document to an embedding model and then calling an LLM.
Once you start thinking about an enterprise environment, a number of other questions appear.
How are documents uploaded and processed?
How should documents be divided into chunks?
Where should embeddings and metadata be stored?
How do we make retrieval useful rather than simply retrieving the nearest text?
How do we make sure a user does not retrieve information they are not allowed to see?
What happens when documents change?
How do we troubleshoot an incorrect answer?
And perhaps most importantly, how do we design the system so that we are not tied to one particular model or AI provider?
These questions influenced the architecture of the AI-powered knowledge platform I worked on.
The goal was not to build another AI demo. The goal was to approach RAG as an engineering problem and create clear boundaries between document processing, retrieval, orchestration, and model execution.
Why RAG?
Large language models are very good at understanding and generating natural language.
They are not, however, a replacement for an organization’s continuously changing knowledge base.
Enterprise information is usually spread across many documents and systems. It can also change frequently and may contain information that should only be available to specific users.
Trying to put all of that information directly into a model prompt is obviously not practical.
RAG provides another approach.
When a user asks a question, the application first looks for relevant information. That information is then supplied to the language model as context.
The model can use that context to generate the answer.
At a high level, the flow looks like this:
User Question
|
v
Query Processing
|
v
Semantic Retrieval
|
v
Relevant Knowledge
|
v
Prompt Construction
|
v
LLM
|
v
Generated AnswerThe important part of this architecture is that retrieval and generation are treated as separate responsibilities.
The model is responsible for generating the response.
The application is responsible for finding the information that the model should use.
That distinction becomes increasingly important as the system grows.
Platform Architecture
The platform was designed as a set of logical components rather than as one large application responsible for everything.
A simplified view of the architecture is:
+----------------------+
| Client / UI |
+----------+-----------+
|
v
+----------------------+
| Authentication / API |
+----------+-----------+
|
v
+----------------------+
| Orchestration |
| Service |
+----+------------+----+
| |
Documents| |Questions
| |
v v
+----------------+ +----------------+
| Ingestion & | | Retrieval |
| Processing | | Service |
+-------+--------+ +-------+--------+
| |
v v
+----------------+ +----------------+
| Embeddings | | PostgreSQL |
| / Indexing | | + pgvector |
+----------------+ +----------------+
|
v
+---------------+
| Redis / Cache |
+---------------+
|
v
+---------------+
| Model / LLM |
+---------------+There are several reasons for keeping these responsibilities separate.
Document processing is generally an asynchronous or background activity.
Retrieval happens when a user asks a question.
Model execution has its own performance, cost, and provider-specific considerations.
Authentication and authorization have to apply to the overall request and, importantly, influence what information can be retrieved.
Keeping these concerns separate makes the system easier to change and easier to troubleshoot.
1. Document Ingestion
Everything starts with the documents.
The ingestion layer is responsible for getting documents into the platform and preparing them for indexing.
A typical flow looks something like this:
Document Upload
|
v
Validation
|
v
Text Extraction
|
v
Content Processing
|
v
Chunking
|
v
Metadata Enrichment
|
v
Embedding Generation
|
v
Vector IndexThere is an important architectural reason for keeping ingestion separate from query processing.
A document might need to be processed only once, or reprocessed when its contents change. There is no reason to repeat that work every time someone asks a question.
The ingestion pipeline can therefore handle responsibilities such as:
- File validation
- Text extraction
- Content processing
- Chunk creation
- Metadata extraction
- Embedding generation
- Indexing
- Processing status
- Error handling
- Reprocessing
This also gives the application a clear place to deal with documents that cannot be processed successfully.
For example, if a document fails during text extraction, that should be an ingestion problem rather than something that affects the query pipeline.
2. Chunking
One of the first practical decisions in a RAG implementation is deciding how documents should be split.
A large document usually cannot be passed directly to the model as context.
The document therefore needs to be divided into smaller pieces, commonly referred to as chunks.
At first this sounds like a simple problem: take a document and split it every few hundred words.
In reality, the way a document is divided can have a noticeable effect on retrieval quality.
If chunks are too small, important information can be separated from the context that gives it meaning.
If chunks are too large, retrieval can return a lot of text that is only partially relevant to the question.
The ideal chunk is therefore not necessarily a fixed number of words.
It should represent a useful piece of information.
Metadata can also be stored alongside each chunk.
For example:
- Document identifier
- Source
- Section
- Document type
- Creation information
- Modification information
- Access-related information
This metadata becomes useful later when filtering search results.
For enterprise systems, I consider metadata almost as important as the vector itself.
A vector tells us something about semantic similarity.
Metadata can tell us whether that piece of information belongs to the right document, category, tenant, user scope, or access boundary.
3. Embeddings and Vector Search
Once documents have been processed and divided into chunks, the next step is to create embeddings.
An embedding converts a piece of text into a numerical representation that captures aspects of its semantic meaning.
The resulting vectors can then be stored and searched for similarity.
For this platform, PostgreSQL with pgvector provides the vector storage and similarity-search capability.
Conceptually:
Document Chunk
|
v
Embedding Model
|
v
Vector Representation
|
v
PostgreSQL + pgvectorOne reason PostgreSQL is attractive for this type of architecture is that vector data does not have to exist in complete isolation from the rest of the application’s data.
The same database can contain:
- Document information
- Chunk information
- Metadata
- Relationships
- Processing status
- Vector representations
This creates opportunities to combine semantic search with traditional relational filtering.
For example:
Semantic Similarity
+
Document Metadata
+
Access ConstraintsThat combination becomes particularly important in enterprise scenarios.
The closest semantic match is not necessarily the correct result if the user does not have permission to access the underlying document.
4. Query-Time Retrieval
Document ingestion and user queries follow different paths.
When a user asks a question, the application needs to find the pieces of knowledge that are most useful for answering it.
A simplified retrieval flow looks like this:
User Question
|
v
Query Embedding
|
v
Vector Similarity Search
|
v
Candidate Chunks
|
v
Filtering / Ranking
|
v
Relevant ContextThe goal is not to retrieve as much information as possible.
The goal is to retrieve the right information.
This distinction is important.
If the system returns a large amount of loosely related content, the model has more information to process, but that does not necessarily mean it has better information.
Too much irrelevant context can make the final answer worse.
The retrieval layer can therefore be responsible for several tasks:
- Query processing
- Query embedding
- Vector similarity search
- Metadata filtering
- Access-aware filtering
- Ranking
- Context selection
- Source information
This also creates a useful service boundary.
If retrieval needs to change later, the rest of the application does not necessarily need to change with it.
5. Retrieval and Generation Are Separate
One of the architectural decisions I consider particularly important is keeping retrieval separate from model execution.
It is tempting to build a single function that does everything:
Question
|
+--> Search
|
+--> Build Prompt
|
+--> Call LLM
|
+--> Return AnswerThat approach may work for a small prototype.
As the system becomes more complex, it becomes harder to understand where problems are occurring.
A cleaner design separates:
- Query processing
- Retrieval
- Context construction
- Prompt generation
- Model execution
- Response handling
The resulting flow becomes:
User Question
|
v
Query Processing
|
v
Retrieval
|
v
Context Builder
|
v
Prompt Builder
|
v
Model Service
|
v
ResponseThis separation makes testing much easier.
Suppose the application produces an incorrect answer.
There are several possible causes.
The question may have been interpreted incorrectly.
The wrong documents may have been retrieved.
The correct documents may have been retrieved but the wrong chunks selected.
The prompt may have been constructed poorly.
Or the model may simply have generated an unsupported answer.
Without clear boundaries, these problems tend to get mixed together.
With clear boundaries, each stage can be investigated independently.
6. Prompt Construction
Once relevant information has been retrieved, the application needs to construct the input that will be sent to the model.
A simplified representation is:
System Instructions
+
User Question
+
Retrieved Context
|
v
Prompt
|
v
LLMThe prompt should clearly distinguish between the user’s question and the retrieved enterprise information.
The retrieved content should provide context for the answer rather than becoming a source of arbitrary instructions.
This is particularly important when documents come from external or user-controlled sources.
A practical prompt design should make clear:
- What role the model is expected to perform
- What the user is asking
- What information was retrieved
- How the retrieved information should be used
- What to do when the available information is insufficient
Another important consideration is context size.
It is tempting to send every retrieved chunk to the model.
That is usually not the best approach.
The objective should be to provide the model with the most useful context rather than the maximum amount of context.
This makes retrieval and context selection closely connected to prompt quality.
7. Redis and Caching
Redis can be used as a supporting infrastructure component for caching and fast-access application data.
There are several areas where caching can potentially help:
- Frequently repeated queries
- Short-lived retrieval results
- Session-related information
- Frequently accessed metadata
However, caching should be introduced based on actual application behavior.
Not every RAG operation needs to be cached.
For example, if the underlying documents change frequently, caching retrieval results for too long can cause the application to return information that is no longer current.
Cache invalidation therefore becomes important.
A useful caching strategy should consider:
- Expiration
- Invalidation
- Document versions
- Query characteristics
- User-specific access boundaries
The last point is particularly important.
If retrieval results are dependent on user permissions, a shared cache must not accidentally return information retrieved under one user’s authorization context to another user.
8. Authentication and Security
Enterprise AI systems need to treat security as part of the architecture.
Authentication answers:
Who is the user?
Authorization answers:
What is the user allowed to access?
For a normal application, these concepts are already important.
For a RAG system, they become even more important because retrieved information is eventually provided to the language model.
If the system retrieves a document that a user should not be able to access, the model can potentially expose that information in its response.
The security flow should therefore be considered before retrieval:
User
|
v
Authentication
|
v
Identity / Claims
|
v
Authorization
|
v
Retrieval Filters
|
v
Permitted Knowledge
|
v
LLM ContextThis means access control should not simply be implemented at the UI level.
The retrieval layer also needs to understand the access boundaries of the information it is searching.
A useful principle is:
Do not retrieve information that the requesting user is not authorized to access.
Trying to remove sensitive information after it has already entered the model context is not a replacement for access-aware retrieval.
9. Orchestration vs. Model Execution
Another architectural decision is separating orchestration from model execution.
The orchestration layer coordinates the overall workflow.
For example:
- Receive the request.
- Validate the request.
- Authenticate the user.
- Perform retrieval.
- Prepare the context.
- Construct the prompt.
- Invoke the model.
- Process the response.
- Return the result.
The model execution layer has a much narrower responsibility.
It interacts with the selected AI model.
Conceptually:
+----------------------+
| Orchestration |
| Service |
+----------+-----------+
|
v
+----------------------+
| Model Execution |
| Service |
+----------+-----------+
|
v
Model / LLMThis separation is useful because AI models and providers change quickly.
The orchestration layer should not need to be redesigned every time the underlying model changes.
For example, a future implementation might use:
- A different LLM provider
- A different model
- A self-hosted model
- A different inference configuration
The orchestration layer can continue to perform the same business workflow while the model execution component changes behind the boundary.
10. Evaluating RAG Quality
One of the lessons from working with RAG systems is that evaluating only the final answer is not enough.
When an answer is wrong, we need to understand why it is wrong.
There are at least two separate areas to evaluate.
Did the system retrieve the right information?
This evaluates the retrieval layer.
Questions include:
- Was the relevant document retrieved?
- Was the correct section retrieved?
- Was useful information ranked too low?
- Was irrelevant information ranked too high?
- Did metadata filtering work?
- Did access filtering work?
Did the model use the retrieved information correctly?
This evaluates the generation layer.
Questions include:
- Did the answer stay grounded in the retrieved context?
- Did the model introduce unsupported information?
- Was the answer relevant?
- Was the response complete enough?
- Did the model correctly interpret the retrieved information?
The distinction can be represented as:
User Question
|
v
Retrieval
|
+----------+----------+
| |
v v
Retrieved Context Retrieval Metrics
|
v
LLM
|
v
Generated Answer
|
v
Generation MetricsIf the correct document was never retrieved, changing the prompt may not solve the problem.
If the correct information was retrieved but the model ignored it, then the investigation needs to move further down the pipeline.
This is why I prefer to think about RAG evaluation as two related but distinct problems:
retrieval quality and generation quality.
11. Common Engineering Challenges
Building a RAG system introduces several practical engineering challenges.
Retrieval Quality
Poor chunking or poor embeddings can result in relevant information never reaching the model.
This is one of the reasons I would not treat the LLM as the only part of the system that needs optimization.
The retrieval pipeline deserves just as much attention.
Context Quality
Even when the correct document is retrieved, the selected context may contain too much irrelevant information.
Ranking and context selection therefore matter.
The goal is to give the model useful information, not simply more information.
Data Freshness
Enterprise information changes.
A document may be updated after it was originally indexed.
The platform therefore needs a strategy for updating or replacing previously indexed content.
Otherwise, the retrieval layer can continue returning outdated information.
Security
Retrieval must respect the authorization boundaries of enterprise information.
Security is not just about protecting the API or the user interface.
It also needs to be considered when selecting the information that will become model context.
Cost and Performance
There are multiple sources of latency and cost:
- Document processing
- Embedding generation
- Database operations
- Vector search
- Model inference
- Network communication
Looking only at LLM latency does not give a complete picture of application performance.
Observability
When something goes wrong, the system needs enough information to explain what happened.
Useful telemetry can include:
Request
|
+-- Authentication
|
+-- Query processing
|
+-- Retrieval latency
|
+-- Retrieved document IDs
|
+-- Number of chunks
|
+-- Prompt/model execution
|
+-- Response latencyThis makes production troubleshooting much easier.
For example, if a user receives a poor answer, observability should help answer:
- What query was processed?
- How long did retrieval take?
- Which documents were retrieved?
- How many chunks were selected?
- Did model execution succeed?
- Where did most of the latency occur?
Without this information, debugging an AI application can quickly become guesswork.
12. Designing for Maintainability
A RAG platform should be designed as a software system, not as a collection of AI scripts.
Clear boundaries make individual components easier to understand and maintain.
A practical architecture can look like this:
+--------------------+
| Authentication API |
+--------------------+
|
v
+--------------------+
| Orchestration |
+--------------------+
|
+-----+-----+
| |
v v
+---------+ +-----------+
|Ingestion| | Retrieval |
+---------+ +-----------+
| |
v v
+---------+ +-----------+
|Embedding| |PostgreSQL |
+---------+ | pgvector |
+---------+ +-----------+
|
v
+-------+
| Redis |
+-------+
Orchestration
|
v
+---------------+
| Model Service |
+---------------+
|
v
LLMThis modularity means individual parts can evolve independently.
For example:
- The embedding model can change.
- The retrieval strategy can change.
- The model provider can change.
- The user interface can change.
- The ingestion pipeline can support additional document types.
- The caching strategy can change.
The rest of the platform does not necessarily need to change with each of those decisions.
This is particularly valuable in AI systems because the technology landscape changes very quickly.
13. Handling Failure Scenarios
Production systems should assume that individual components will sometimes fail.
Possible failure points include:
- Document upload
- Text extraction
- Embedding generation
- Database operations
- Vector search
- Redis
- Model invocation
- Network communication
- Authentication services
The platform should therefore have clear failure boundaries.
For example:
Document
|
v
Ingestion
|
+---- Failure ----> Processing Error
|
v
Embedding
|
+---- Failure ----> Retry / Failed Status
|
v
Vector IndexA failed document should not silently disappear.
The system should be able to represent its processing state and make failures visible to operators or administrators.
The same principle applies to query-time failures.
A model timeout, authentication failure, database failure, or temporary network issue should be handled differently from an invalid user request.
It is useful to distinguish between:
- Validation errors
- Authentication failures
- Authorization failures
- Temporary infrastructure failures
- Processing failures
- Model execution failures
Clear failure categories make monitoring and user-facing error handling more predictable.
14. Enterprise RAG as a Software Engineering Problem
One of the biggest lessons from working on RAG systems is that the most interesting engineering problems are usually not the first model call.
Calling an LLM is only one part of the overall solution.
The real engineering work starts around it.
How do documents enter the system?
How are they processed?
How are they indexed?
How do we retrieve the right information?
How do we enforce security?
How do we build the context?
How do we handle model failures?
How do we know why an answer was wrong?
How do we keep information current?
How do we replace the model later?
These are software engineering questions.
The AI model is an important component, but it is still one component within a larger system.
This is why I see enterprise RAG less as an “AI feature” and more as an application architecture that happens to include AI.
15. Lessons Learned
There are a few lessons that stand out from designing this type of platform.
The LLM is only one component
It is easy to focus most of the attention on the model.
In a production system, the surrounding architecture is just as important.
Retrieval deserves serious engineering attention
A powerful model cannot answer from information it never receives.
If retrieval is poor, the entire system suffers.
Security needs to be part of retrieval
It is not enough to authenticate the user at the API boundary.
The information being retrieved also needs to respect the user’s access boundaries.
Clear boundaries make systems easier to change
Separating ingestion, retrieval, orchestration, and model execution means individual components can evolve without forcing a complete redesign.
Observability matters
AI applications can fail in ways that are not always obvious from the final response.
Being able to see what was retrieved and where time was spent makes debugging much more practical.
AI systems still need normal software engineering
Authentication, authorization, databases, caching, error handling, monitoring, testing, deployment, and maintainability are still important.
The presence of an LLM does not remove those responsibilities.
Conclusion
RAG provides a practical way to connect language models with enterprise knowledge.
But a production-quality RAG platform is much more than a vector database and an LLM API.
The architecture needs to consider the entire journey of information:
Enterprise Knowledge
|
v
+----------------+
| Ingestion |
| Processing |
| Chunking |
+-------+--------+
|
v
Embedding Model
|
v
PostgreSQL
+ pgvector
|
v
Retrieval Layer
|
v
Context Builder
|
v
Model / LLM
|
v
User ResponseThe approach I prefer is to keep the boundaries clear:
- Ingestion is responsible for preparing knowledge.
- Retrieval is responsible for finding relevant knowledge.
- Authorization is responsible for determining what can be accessed.
- Orchestration is responsible for coordinating the workflow.
- Model execution is responsible for interacting with the LLM.
- Observability is responsible for helping us understand what happened.
This makes the platform easier to reason about, test, operate, and evolve.
Most importantly, it changes the way we think about RAG.
It is not simply an AI demo where a document is uploaded and a question is asked.
It is an engineering platform that brings together traditional application architecture, data processing, search, security, distributed services, and AI.
That is where I believe the real value of enterprise RAG begins.