SLM + RAG
A compact language model combined with retrieval to produce grounded responses without relying entirely on a large general-purpose model.
Problem
Bigger models aren’t automatically better systems.
A general-purpose language model can contain enormous amounts of knowledge, but that doesn’t mean it has the specific knowledge an application needs — or that it should be responsible for storing all of it.
Can a smaller model become significantly more useful when it’s given the right context at the right time?
Approach
Instead of trying to make the model memorize everything, I separated reasoning from knowledge.
The SLM handles language generation and reasoning, while a retrieval layer finds relevant information from an external knowledge base and supplies it as context.
The result is a system where the model doesn’t need to know everything. It needs to know how to use what it is given.
Reasoning ≠ Knowledge
- Reasoning
- Knowledge
The system
Query
The user asks a question in natural language.
Retrieve
The system searches the knowledge base for relevant context.
Augment
The retrieved information is assembled into the model's context.
Generate
The SLM produces a response using the supplied context.
Ground
The response is constrained by the retrieved information rather than relying solely on model memory.
Why an SLM?
The goal wasn’t to use the biggest model available.
Smaller language models can offer advantages where latency, compute, cost or deployment constraints matter.
But reducing model size also makes the question of knowledge more important. Retrieval provides another path: instead of making the model larger, give the model better context.
- More parameters
- More capability
- More compute
- Smaller model
- External knowledge
- Focused context
Neither column is the winner. They buy different things, and which one is correct depends on the constraints the system has to live inside.
The interesting part
Knowledge doesn’t have to live inside the model.
RAG changes where the system gets its information. The model provides the reasoning layer. Retrieval provides the knowledge layer.
Separating the two makes the system easier to update and gives the model access to information that doesn’t need to be permanently encoded into its parameters.
- Model
- Reasoning
- Retrieval
- Knowledge
Architecture
Both flows meet at retrieval. The knowledge pipeline runs ahead of time and ends at the vector store; the request path reads from it. Everything the model sees at generation time arrived through that one point, which is what makes the output attributable.
The model
Why this model
The selection criteria were the ones set out above — latency, compute, cost and deployment constraints — traded against how much reasoning the system still needed the model to do on its own. The specific checkpoint and its settings are not recorded in this repository, so they are left blank rather than reconstructed from memory.
The knowledge base
The retrieval system acts as the model’s external memory.
Finding the right context
Retrieval quality becomes just as important as generation quality.
A small model with excellent context can outperform a larger model working with irrelevant context.
The retrieval layer therefore becomes a critical part of the system rather than a preprocessing step.
Observations
This was an experiment, and it is worth being exact about what that means: no formal evaluation was run, so there are no numbers to show here. Presenting any would be inventing them.
What the system would have to be judged on is not in question, though — and knowing which measurements matter is most of the work of designing the evaluation.
The trade-off
Smaller models introduce constraints.
They may have less general reasoning capability and a smaller context budget.
Retrieval helps compensate for the knowledge problem, but it doesn’t eliminate the model’s limitations.
The interesting engineering problem is finding the point where model size, retrieved context and system constraints work together.
What I learned
The model is only one part of the system.
Building an AI application changed how I think about model performance. A better model doesn’t automatically produce a better product.
Retrieval quality, context construction, prompting, inference constraints and evaluation all affect the final result.
The intelligence isn’t just in the model. It’s in the system around it.
Stack
Outcome
The project explored a different approach to building useful language systems: rather than scaling the model indefinitely, combine a smaller model with a retrieval layer that supplies the knowledge it needs.
The result is an architecture where generation and knowledge are deliberately separated — making the system a study in how much capability can come from the architecture around a model, not just the model itself.