As a medium-sized company with a large number of monthly active users (MAU) and a few digital products, you will have realised by now that using free ChatGPT is not enough. Yes, you have learned about skills.md and the expensive Claude subscription. But is that enough? In this article, I will share the knowledge needed to design a multi-agent LLM system for mid-scale organisations.
The golden rule of working with LLMs: treat publicly available models as a brain, not as eyes and ears. An LLM doesn't actually know anything — it generates something that looks like an answer. That's why we, as developers, need to add plenty of validation along the way from the user's request to the model's response. Keep one simple rule in mind: if a human could mess it up, an LLM definitely will, because LLMs have no common sense.
The general pipeline for getting an answer from an LLM looks like this: indexing > retrieval > generation. The user types a prompt, it gets vectorized and sent to the database to look for relevant data. But how do we check the facts? For facts, we rely on external tools, not the LLM. Out of the box, LLMs are usually bad at precise facts — an LLM is not a rule engine. So if you're serious about your LLM product, never call the model without validation. Validation is what turns a bare LLM into a real agent.
Working with prompts
The easiest way to reduce the chance of incorrect answers is to get an explicit prompt out of the user. The very first check is input validation: we need to know for sure that the request matches the data we're prepared to serve. Another strategy is the family of zero-shot, one-shot, few-shot, and multi-shot methods, which help the LLM understand the format you want the answer in. With zero-shot, the model decides on its own how to approach the task. The more shots you give, the more context the model has about what exactly is expected of it.
You can also inject sample answers into the context using RAG. RAG gives the model access to up-to-date knowledge and can reduce hallucinations. Sample answers can be built around popular prompting techniques, such as:
- Persona — "Imagine you're a financial analyst…"
- Chain of thought — "Break the task down into substeps…"
- Chain of density — "Once you've prepared the final answer, go through the information again and make sure nothing important was left out…"
- Tree of thoughts — "Give me several possible answers, pick the best one, repeat."
- Graph prompting — graphs, like the ones in ComfyUI.
- Knowledge augmentation — "Second agent, challenge the first agent's answer…"
- It's always a good idea to explicitly ask the LLM to draw up an action plan. Step-by-step reasoning works through zero-shot CoT.
Keep your prompts short. One task, one prompt. If you want the model to assess a situation, give it explicit options: positive, negative, or neutral. Start with simple prompts and gradually layer on context — it works as a kind of lightweight fine-tuning. This is also why you shouldn't mix developer specs with product context: they belong in separate stores, so the LLM never gets confused about which context is the right one. Stick to SDD (spec-driven development) frameworks: Superpowers, GSD, grill-with-docs, OpenSpec.
Another framework for better prompts is RACE (Role > Action > Context > Execute/Expectations). Here's the structure:
- Role: have the model play a CFO, a copywriter, or a security specialist.
- Action: analyze, summarize, transform, generate.
- Context: describe the audience — who will use the results.
- Execute/Expectations: JSON, formal prose, word count.
Role: Facebook ad specialist with expertise in Gen Z engagement
Action: Create six ad copy variations for our new self-driving scooters, bearing in mind the additional value of the carbon tax.
Context: Brand voice guidelines: casual, informal, authentic, slightly irreverent.
Target audience: environmentally-conscious Gen Z (18-26) • USPs: recycled materials, carbon-neutral shipping, $10 donation to plant mangrove trees.
Maximum 115 characters per ad.
Execute: Provide 6 numbered ad variations with:
- Attention-grabbing headline
- Body copy
- CTA (Call to action)
Each one should appeal to a different emotion (urgency, rebellion, belonging, impact).
But what if the user overdoes it and sends a gigantic request? Say they dump their entire console output into our little agent, just to see what the AI makes of it. More verbs, better results — isn't that what everyone thinks? In that case, the agent decomposes the request into a series of subqueries, and we retrieve many chunks, each answering one small subquery. This is called Demonstrate-Search-Predict (DSP). First, the LLM generates a general answer to the main question, and then the set of subquestions forms vectors for better retrieval. You'll find detailed descriptions of these techniques under names like query expansion, query transformation, step-back prompting, and query rewriting. Keep in mind, though, that every extra step for retrieving, processing, and validating data costs more resources and adds response time — a sure path to poor tail latency.
In more technical terms, what matters isn't how many tokens you spend in one go, but the size of the context. Tokens aren't the same as words: they include punctuation, symbols, digits, and other text elements, and tokenization schemes vary from one LLM to another. As a rule of thumb, a two-word phrase like "cat escaped" takes about two tokens. This is where context limits come from: GPT-3.5 had a context window of 4,096 tokens, roughly five pages of text. GPT-4 raised that to 8,192 tokens (10 pages), and GPT-4-32k arrived with 32,768 tokens (40 pages). Gemini 1.5, meanwhile, works with a 1-million-token context window — over 1,000 pages. So can we take older models and feed them a huge context? No — that's a sure path to hallucinations. I've tried.
Also, be careful about letting users play with the top-p parameter, which controls the probability distribution — think of it as focused randomization. It's generally not recommended to adjust top-p and temperature at the same time. Note that top-p is OpenAI's term; other models may call it something else. A higher temperature means a more "creative" model, but in reality it simply makes the next token less predictable. If we asked the model to delete files and cranked the creativity all the way up… well, whose fault is that?
Agent orchestration
If you have just one agent (LLM + harness) handling every task, it will break quickly. This is called role overload. The fix is simple, just like with human employees: split the responsibilities. One person can single-handedly carry an entire project, but they can't keep doing it on a steady basis. That's exactly why businesses and professions exist — a team with defined roles solves problems systematically. At the very least, it lets agents keep working after errors occur. "If you failed, try again" logic is generally bad practice (just as it is in people management). You end up in an infinite loop where the agent hits the same error over and over and burns through your entire budget. Mind your limits.
Also, if you're simply calling an LLM directly, that's not an agent — it's just an interface wrapper around the LLM. An agent runs in a loop: it reasons about the best action or tool for the task, uses those tools, and updates its state. One popular structure looks like this: observe → decide → act → update state → repeat or stop. All of these steps are essential for debugging, too. The state can be the user's prompt, tool outputs, or the history of failures.
Agents need certain rules: common sense (gravity exists, so people can't fly), linguistic knowledge (the syntax of a language), and domain knowledge (understanding medicine, say). Knowledge is more or less "frozen" at the pre-training stage. Common sense, however, can be its own separate agent.
Which agents you build is entirely up to you, but each industry has established best practices. A standard "team" consists of reasoning (research, planning) > tools and integrations (APIs) > memory (remembering all actions, plans, and conversations) > and the worker agents themselves (writing content, sending emails, running code). For example, if you're building agents for red teaming, you'll definitely need a dedicated agent that blocks prohibited, dangerous, or irreversible actions. Or a supervisor agent might oversee several local control agents. Its main job is to resolve goal conflicts between those local agents by adjusting their objectives in line with global priorities. In short, a boss agent that keeps everyone on course toward the goal.
A Monitor Agent is responsible for watching external API integrations in a production multi-agent system. Teams usually want it to spot signs of API degradation and coordinate remediation. This can be tracked in numbers — for example, how often each external API exceeds its timeout within a given time window. A review might reveal, for instance, that executor agents are frequently waiting on decisions from planner agents. That means the data-exchange queue needs optimizing and backpressure routing should be added. Or, in a large multi-agent system, monitoring might show that end-to-end request latency spikes with every traffic surge, even though average CPU utilization across the cluster stays below 50%. In that case, we notify another agent or a human.
Here's another real-world example. In my own multi-agent system, the monitoring agent detected that an external scheduling API was showing latency and intermittent timeouts at certain hours. To stabilize behavior, I added adaptive backoff and graceful degradation, temporarily simplifying how scheduling requests were handled whenever latency crossed predefined thresholds. And whenever an agent failed, I returned structured error codes along with machine-readable details on what caused the failure and which constraints were violated.
Naturally, there's a dedicated agent for working with MCP. If an MCP server has database access, it's important to restrict queries to a predefined set of parameterized operations with strict input validation. Every tool should be isolated, and read-only tools can run in parallel. But if calls can modify the underlying data, they should run sequentially. Once again, agents need to understand when to call external tools and when to rely on their built-in knowledge.
In a multi-agent manufacturing system, a coordinator agent orchestrates calls to several MCP tools: one forecasts demand, another assesses production capacity, and a third handles scheduling. The right strategy is to use a shared domain model and map each tool's input and output schemas to that model before and after every tool call.
The role-based split described above also helps with debugging. You can only debug between steps — between agents and tools. Be sure to think about tracers, so you know what state the system was in and at what moment. This isn't about logging; it's about being able to answer questions when something goes wrong: inputs, outputs, validation errors, state changes, and final decisions. We need to be able to explain to management why an agent behaved a certain way at a specific point in time. New agents should be rolled out via blue-green deployment. And if agents have learned something while working with a particular cohort of employees, that new skill should be propagated into the agent system only after offline validation.
We're also talking about using different LLMs for different roles. Take DeepSeek: they have many models, such as V3, R1, and Janus Pro. DeepSeek is good at processing unstructured data and can show its chain of reasoning. It works at the Neural Network Layer. The common practice is to use DeepSeek-V3 for code generation and refactoring, and DeepSeek-R1 — which is built on transformers — for reasoning and decomposing complex tasks. Every LLM gets its own role, and when choosing one, consider the model's architecture (CNN, Transformer, RNN) and look at its loss function. V3 is also well suited to generating candidate bug fixes, while R1 is better at determining which fix best addresses the root cause. R1 can draw reasonably sound conclusions from complex datasets; its inference module is designed to make well-grounded inferences, which is one of R1's core functions.
As for combinations that work well: GPT + Codex, Claude + Claude Code, Gemini + Antigravity (though you can't get by without Claude here), and DeepSeek + Kimi + GLM + OpenCode.
CrewAI is used for agent orchestration, and its main goal is to make the system predictable. CrewAI assigns agents roles — marketer, developer, support operator — and that's where the predictability comes from, along with reduced flexibility. AutoGen, on the other hand, is more about LLMs communicating with each other, which gives you high flexibility and a higher chance of problems. AutoGen typically requires very active human involvement at every stage of the agents' work, with small, iterative tasks. This is also where you can plug in RLHF, in which a human helps the models produce results that match human expectations — and that requires direct human feedback. AutoGen can also be very token-hungry and drift into long, convoluted dialogues. If you need complex orchestration logic, look at LangGraph instead. LangGraph is more token-efficient than AutoGen, especially with a well-built Critic Agent. More advanced, visual tools include n8n. Or you can simply get by with the LangChain ecosystem:
- LangServe: turns your system into an API.
- LangChain: a set of modules for integrating LLMs with additional memory, prompts, and other tools.
- LangSmith: used to analyze, monitor, and evaluate your application.
And there's no need to build a homegrown monster out of a thousand components. You can use DeepSeek via GrokAPI, Cursor AI, or Perplexity, and Make for building assistants. Or you can do it all yourself: create a backend endpoint that authenticates against the DeepSeek API, and forward user requests to DeepSeek after sanitizing them.
RAG
RAG is simply extra information added to an existing LLM to help it answer better. RAG was originally designed for one simple job: retrieve information and summarize it. Complex questions also require the model to think, which goes beyond what standard RAG offers. Naive RAG is "retrieve and read," while so-called modular RAG is more about retrieve, read, and rewrite. Overall, there are two main ways to teach an existing LLM new knowledge: fine-tuning and RAG.
When to choose RAG and when to choose fine-tuning:
- RAG is great for feeding new knowledge directly into an LLM, and it can do so dynamically, in real time. Fine-tuning is static.
- RAG can work with a small data sample, while fine-tuning needs a good, large, clean dataset. It's worth stressing that unstructured data — articles, chat histories, meeting notes, reports — is no friend to either RAG or fine-tuning. The same goes for semi-structured data in JSON, XML, or HTML: working with data from files requires dedicated pipelines. Structured data, on the other hand — Excel, SQL databases, in some cases even PDFs (in my experience, the most problematic format), GitHub repositories — is the most desirable option. Different models also work better with different formats: Claude 3 prefers XML, while Llama 3 goes for SYS and INST tags.
- With RAG, it's easier to trace that answers came from the new knowledge layer. RAG is good at fact lookup. So it looks like RAG wins… But! Fine-tuning lets you change the model's behavior and writing style, add new skills, and reduce hallucinations, combined with span extraction.
- Fine-tuning means changing the model's weights based on the added data. It's like our long-term memory: if you sat through a long university lecture, you'll probably remember the gist but not the details. A fine-tuned model works the same way — it isn't great with details. So compared to RAG, fine-tuning is better suited to generalizing information. For tooling, I recommend Unsloth. You can update all of a model's weights with full-model fine-tuning (FMFT), or just a small fraction of its parameters with parameter-efficient fine-tuning (PEFT).
Staying on the subject of datasets: deduplication is another important part of working with LLMs. Duplicate data harms LLM training and also causes damage when it surfaces during answer generation. As data volume grows, the risk of duplicates grows linearly. Techniques such as fuzzy matching and hash-based deduplication can be used to eliminate them. More broadly, you should build a pipeline for quality control and data management in your system. These pipelines should include rules and tracking systems that let you identify problematic data and trace where it came from.
For more advanced RAG, you need richer metadata for precise retrieval. Metadata should concisely capture information that would otherwise be scattered throughout the document. One approach is a hierarchical index, where a short summary is generated for each level of the hierarchy.
To check our RAG, we can use Self-RAG, which lets the model look back at its own answer and evaluate it. Self-RAG iteratively verifies the reliability of the sources behind the final answer. Alternatively, there's Corrective RAG, which tackles the problem of faulty retrieval. Corrective RAG checks the quality of the retrieved data before generating an answer for the user. The system retrieves a document or piece of information and analyzes it; if the result is unsatisfactory, it either rewrites the query or turns to web search. This is a very popular approach for bots that must deliver accurate information.
Databases and chunks
A vector database is a specialized database designed to store high-dimensional vectors. As such, it's optimized for unstructured and semi-structured data represented as vectors. Its job is to provide efficient storage, indexing, and search. Your choice of vector database also has a significant impact on RAG performance. There are dozens of options today, so picking the best one can be a challenge. Our default vector DB is ChromaDB, but there are other ways to store data, such as RDBMS, NoSQL, and NewSQL. You can even add pgvector to PostgreSQL and get yourself a vector store. A vector store typically consists of three layers: indexing, storage, and processing.
Why vector databases instead of conventional ones? Libraries such as FAISS (Facebook AI Similarity Search) aren't designed for create, read, update, and delete (CRUD) operations, so they're a poor fit for dynamic systems where multiple users access and modify data. Conversely, if our database is immutable and we only provide read access, FAISS can be a good solution. There are also SQL databases that support vectors as an extension of the classic database. They let you efficiently index related metadata. However, they don't scale well, often limit vector size, and deliver lower performance. SQL databases are a good choice for internal projects that need to connect to existing enterprise databases (which are probably already in SQL), but not when scalability and performance matter.
When choosing how to store your data, it's important to understand what questions potential users will ask the RAG system. For example, if users will ask questions that require the model to look up several facts, it's better to use small chunks that each contain a direct answer. If the system will be more "reasoning"-oriented, chunks with more context work better. When there are too many chunks, speed drops, and the odds of finding the right piece of information are far from 100%.
Chunks can also be made more semantically aligned with incoming queries. A chatbot will most likely receive questions as input, so you generate hypothetical questions for each chunk, and those questions are represented as vectors (embeddings). An embedding is essentially a way of clustering text. Embeddings help reveal semantic relationships and improve understanding of the data. Say you have millions of product reviews and want to find all the similar ones. Or a recommender system built on the semantic relationships between a user's likes and dislikes. Embeddings aren't limited to text — they work for video and images, too. They convert human language into vectors that AI systems can analyze. Semantic similarity recognizes that two differently worded sentences actually mean the same thing.
You need semantic search. Because it accounts for the meaning and context of the data, semantic search makes search results and recommendations more relevant. In semantic search, the smaller the distance, the greater the similarity — this is Euclidean space at work. Another algorithm is the dot product, which is more about vector similarity than distance. There are many other options as well: Lin similarity, Jaccard similarity, Hamming distance, Manhattan distance. On top of that, techniques such as Hypothetical Document Embeddings (HyDE) come into play: for a user query, the system first generates a hypothetical answer that's semantically similar to the original question, then uses that answer to find matching chunks. In general, embedding models struggle with topics they're unfamiliar with. That's why the hypothetical answer is more about the pattern of an answer than its correctness.
Cosine similarity is the most common way to calculate the similarity between two embedding vectors: it measures the cosine of the angle between them. It's used to analyze relationships between sentences in embedding space by measuring the angular distance between their vectors. There are plenty of libraries for vectorization; popular ones include Word2Vec, Sentence2Vec, and Doc2Vec. Vectors can have thousands of dimensions, and more dimensions can capture finer-grained semantic information. If you know how to compare vectors using cosine similarity, you can write your own RAG.
The one approach that stands apart is keyword-based search, where matching happens on keywords. It works well when the query targets a specific name: a term or a company name. A popular model for this kind of search is BM25, the counterpart to vector search. BM25 finds documents that contain a particular term. BM25 is an example of sparse vectors, as is TF-IDF. With BM25, you get a formula along the lines of scorehybrid=1−0.43∙scoresparse+0.32∙scoredense. In plain terms: you have a product catalog with millions of items, and a user searches for something like "Samsung Note," "mid-range smartphone for my mom with a good camera," or "refurbished Xiaomi under $300." BM25 on its own can't handle that — and neither can vector search. Combine the two, and you'll get solid results.
Evaluating quality
The most basic metrics for validating LLM output are precision and recall. Precision is the share of retrieved documents that are relevant, and recall is the share of relevant documents that were successfully retrieved. These metrics show how often the right document was ranked high and retrieved. It's also useful to look at ranked list quality — which documents made it to the top.
It's good to have ground-truth data on hand. This is the ideal answer we'd expect from the RAG system, and it also serves as the benchmark for LLM quality. How to create ground-truth data is another matter. The simplest way is to hire an expert to write reference answers. For a support chat, for instance, the experts would be employees tasked with writing answers in the format To resolve [issue], you can try [solution]. Answers like these can be scored on Faithfulness — whether the answer is supported by the documents — and Groundedness — how well the answer matches the query and the data it was built on. Other ways to compare generated answers against reference ones include Ragas, a framework for RAG evaluation, as well as BLEU, ROUGE, and expert review.
Another test is the needle in a haystack. On its own it's fairly basic, but it really proves its worth when you need to find multiple needles — and far from every model can handle that.
For evaluating the LLM itself, the key metrics are accuracy — the share of correct predictions, or the quality of answers and reasoning. In spam filtering, for example, the fewer malicious emails slip past the LLM, the better it is. The second metric is latency, i.e., speed, measured as time to first token or time to last token.
Inference and speed
In a model's lifecycle, training comes first: the process in which the model learns from scratch. No LLM > LLM. Training takes anywhere from hours to weeks or even months. Training is when the model learns; inference is when it predicts — that is, generates an answer. Inference is next-token prediction, or simply generating a response to your request in real time. Your model has finished training and now runs on an NVIDIA A100, H100, or some other GPU cluster. During inference, you're most likely constrained by available memory, power, and time — and despite all of these constraints, inference still has to run efficiently for the end user.
Models served online (SaaS) have fixed weights. These are large models with billions of parameters (GPT-4, Llama-70B) that run in the cloud and require 40 GB GPUs. They're so big because they're very "deep" — they have many layers. Does that mean bigger is better? No. Modern models can't scale indefinitely; at some point, adding parameters brings no performance gain and can even hurt.
Large models are clunky and slow. Users don't need a good model — they need a fast one. In production, the typical latency budget is 200 ms max (p99). A 7B model will run slowly — say, 3 seconds per token. A 70B model will require 280 GB of GPU memory. For such large models to deliver value in the mass market, you need inference optimization to speed them up. The first method is quantization. Let me say up front that carelessly applied quantization severely hurts model accuracy and leads to hallucinations. You need to add RTN (round-to-nearest). At the same time, quantization is the fastest way to shrink a model, which is great for getting to production quickly. Quantization reduces the model's structure, which may require retraining.
To improve quantization quality, one popular technique is AWQ, which uses a small dataset to guide the quantization process. Layer by layer, the outputs of the optimized and non-optimized versions are brought closer to the reference. Some weights matter more than others, and those are the ones we try to leave untouched. GPTQ quantization reduces error during weight calibration. If your hardware is really weak, GPTQ is the way to go. GPTQ performs layer-wise second-order error correction using calibration data, producing 4-bit LLM weights with minimal perplexity loss even at scales from 7B to 70B parameters.
AWQ takes a different approach: it identifies the small fraction of salient weights whose quantization dominates the error and scales them to protect accuracy, while the remaining weights are quantized aggressively. Both methods are supported in Transformers, AutoAWQ, and AutoGPTQ, and are integrated with vLLM and TensorRT-LLM. AWQ is faster and strikes a good balance; GPTQ focuses on minimizing quantization error, is more mathematically rigorous, and delivers higher accuracy under extreme compression.
Another method is distillation, which simply trains a new, smaller model and requires a lot of resources.
Scaling
So what makes some LLM-based products better than others? Lots of clever engineering under the hood. If an agent remembers what you talked about a year ago, it's easier for users to come back to the product. Agent memory (window buffer memory) stores only recent messages; it's quick to read and quick to clear. Postgres memory is long-term memory that persists even after the session ends. Motorhead memory is a centralized memory store — a single shared layer that lets many agents access the same memory. As for a single agent's memory, the model's attention relies on two things: K (key) vectors and V (value) vectors. Every token is stored in the KV cache. Picture that ancient chat from ChatGPT's first launch that you decided to revive: the entire conversation context lives in the KV cache. All of this eats up a lot of memory, and every request needs its own KV cache, so the cache has to be cleared from time to time. With 1,000 users, you're looking at linear growth (good) and memory overflow (bad). Once memory overflows, the system stops working. So for a single user, the KV cache is fine; for millions of users, no amount of GPU memory will ever be enough.
The solution is specialized LLM servers optimized for multi-user scenarios and efficient memory usage — vLLM, for example. You can't run it natively on Windows, but it works with WSL 2. A more mature, enterprise-grade option is NVIDIA Triton Inference Server, built for corporate inference workloads. It supports multiple backends (TensorRT, ONNX Runtime, PyTorch, TensorFlow, Python, vLLM) through a single API. An inference server doesn't just forward user requests to the LLM; it adds optimizations on top. Suppose 100 users send requests at the same time. Their requests are grouped into a batch, which reduces the number of GPU launches. The batches then wait their turn in the scheduler. On top of that, the KV cache is reused for similar requests. And always hook up Prometheus for monitoring.
I'd recommend designing your agents around ONNX. That way, every model lives in a single portable file that describes its architecture, which reduces framework lock-in and lets you move the model between environments.
To run a model locally, you need llama.cpp — a C++ implementation for running LLMs on CPU. Just find the model's GGUF file on Hugging Face, choose a quantization method, and load it into llama.cpp. Remember: accuracy in lab conditions is great, but if a model eats too much memory and runs slowly, few people will care. Say you've fine-tuned a model and want to run it in llama.cpp on a local CPU — then 4-bit GGUF is your path. If you want the same model on a server GPU with vLLM, go with INT4 (AWQ/GPTQ). Smaller models can even run on a phone with no internet connection, or on an IoT device with very little memory and power. For phones, your options are TensorFlow Lite, ONNX, and Core ML.