AI is now at the heart of everything from customer analytics to research, and businesses are looking for ways to harness its potential. Many enterprises believe that training (or at least fine-tuning) a model is the best way forward. But is this actually true?
In an AI Explainer Series conversation, Box CTO Ben Kus and Senior Product Marketing Manager for AI Meena Ganesh discussed why pursuing model training may be the wrong choice for most enterprises. There’s a safer, smarter, more cost-effective alternative right at your fingertips: retrieval augmented generation.
Why enterprises should avoid training AI models
Say a consulting firm hopes to use AI for analytics and research on their own unstructured data — all the content they’ve amassed (and will continue to amass) through everyday work. That data might include meeting notes, client deliverables, presentations, research, and many more types of files. It’s highly specific stuff, so, as Ganesh asks, “Isn’t training a model the best way to provide context?”
At first, yes, model-training would seem to make the most sense. It allows the consulting company to tailor their AI solution to their own data and specific needs. But as Kus bluntly states, “Enterprises should probably never train an AI model.”
Why such a stark stance?
Training a large language model (LLM) is not as simple as it seems. “It’s incredibly expensive,” he says. ”It’s also hard to manage, and you’re going to end up having serious security data permissions issues.”
Training an LLM can cost hundreds of millions of dollars and require extensive computational resources. For this reason, the process is inaccessible unless you're a company like OpenAI. As Kus quips, “It’s sort of the philosophical equivalent of going out to find a small child to teach everything — their whole life — so that when they’re done, they’ll be able to come work in your company.”
Training an LLM can cost hundreds of millions of dollars and require extensive computational resources.
A somewhat dramatic simile, yet strangely apt. The analogy is clear: the effort, time, and resources necessary to train a unique LLM from scratch make it unrealistic for most enterprises.
Why fine-tuning isn’t much better
Fine-tuning an existing commercial LLM might seem like a viable option. As Kus describes it, “Fine-tuning, or reinforcement learning, is the idea of taking a model and updating it — not starting from scratch, but updating it so it’s specifically trained.”
This option is typically much easier and cheaper, but still, Kus cautions against it: “Definitely don’t do that either,” he says.
Fine-tuning requires ongoing expertise with data preparation, model management, compatibility updates, and more. More critically, as Kus explains, “They’re not reverse-compatible. If you fine-train a model on one version of that model, and a new model comes out, you have to fine-tune a new version on top of that new model.”
Each update or removal of sensitive data means starting the fine-tuning process all over again — an impractical cycle for any enterprise looking to operate efficiently.
They’re not reverse-compatible. If you fine-train a model on one version of that model, and a new model comes out, you have to fine-tune a new version on top of that new model.
Each update or removal of sensitive data means starting the fine-tuning process all over again — an impractical cycle for any enterprise looking to operate efficiently.
Another issue? Security.
“AI doesn’t keep secrets,” Kus warns. This means that any data or knowledge refined into the model could be inadvertently exposed to others who access it, creating risks of data leakage. Enterprises that work with sensitive information or have strict data permissions can’t afford these risks.
Retrieval augmented generation (RAG) and agents
A smarter alternative to training and fine-tuning (and the best way to leverage AI effectively) is secure retrieval augmented generation (RAG) paired with AI agents.
An AI agent is an AI-powered system that takes intelligent action to try to accomplish an objective. It uses an AI model as its brain, is given instructions to meet an assigned objective, and then uses tools to reach that goal. Agents built atop the best foundation AI models can understand, reason, and generate to automate workflows. Newer agents don’t just follow instructions like an assistant; they proactively plan and deliver comprehensive results.
Rather than have to figure out what it takes to fine tune something, you just need to prompt the agent.
Building on this, secure RAG gives the AI agent access to only the right data, specific to the user. This ensures AI agents operate in a permissions-aware environment that does not compromise data security.
Ganesh summarizes this two-part process:
- “Secure RAG is basically a way for AI to get context from vast amounts of content”
- “Then, we leverage agents and give them custom instructions, including guardrails as well as tools to access the right knowledge in a very permissions-aware manner”
So now, instead of laboriously training models, enterprises can simply provide AI agents with access to the right knowledge and guardrails via RAG. “Rather than have to figure out what it takes to fine tune something, you just need to prompt the agent,” Kus says. This combination “will handle almost all the use cases that used to require a fine-tuned or a reinforcement learning style of feedback.”
Why RAG + agents work so well together
Unlike when training a model, enterprises using RAG don’t need to worry about constantly updating or fine-tuning algorithms. Updates to the models behind these agents make them smarter automatically, without added work. “You don't need to manage models; you just need to manage the agents,” Kus adds.
Security is a cornerstone of this approach. Since agents can only retrieve information users are inherently permitted to access, they ensure compliance with data permissions and prevent leakage. The agents cannot access any information the user of the agent doesn’t have access to, which is why, in Kus’s words, “You solve the challenge associated with the idea of retrieving appropriate information without data leakage.”
The bottom line: focus on smart, scalable AI tools
Training and fine-tuning models requires intense resources, expertise, and ongoing management. For companies exploring AI, the focus should shift from costly and inefficient processes toward scalable and secure AI adoption. With RAG and agent-based solutions, enterprises don’t have to reinvent the wheel. They can simply harness AI’s ability to retrieve context, operate efficiently, and comply with permissions.
Of course, while RAG and agents work for most enterprises, there are exceptions. “Maybe you're doing something on the edge where you don't have all the power of a cloud-based system,” Kus says. These kinds of edge scenarios or hyper-specialized tasks may require custom training or fine-tuning, but for the average enterprise, the combination of RAG and agents largely suffices.
The recommendation from Kus is straightforward: “Use AI agents for retrieval augmented generation, pick a platform that does this well (like Box), and use that before you even consider training.”
Watch the full AI Explainer episode Stop before you train that model: A smarter path for enterprise AI. l and learn more about why Kus and Ganesh recommend using RAG and AI agents together to solve 99% of your custom enterprise AI model needs.
