Rarely, and almost never first. Most of what teams hope to fix by fine-tuning, such as wrong facts, the wrong tone or missed escalations, is fixed faster and more safely in the agent's brief, sources and bounds. Those changes stay readable and reversible. Fine-tuning earns its place only for narrow, high-volume tasks where a tuned model clearly wins.
What problem are you actually trying to solve?
Start by naming the failure. Most fall into four groups, and each has a better first fix than retraining:
- The agent states wrong or outdated facts. That is a sources problem. Give it the right system of record, with provenance, and remove stale copies.
- It writes in the wrong style. That is a brief problem. Add approved examples and clearer instructions.
- It handles cases it should hand over. That is an escalation rule in the brief, backed by bounds.
- It acts beyond what it should. That is a bounds problem, enforced outside the model. Training a model to be careful is no substitute for a limit it cannot exceed.
Fine-tuning changes how a model tends to respond. It does not give the model a fact it can cite, and it does not create a limit.
When does fine-tuning make sense?
| Situation | Fine-tune? | Why |
|---|---|---|
| Classifying very high volumes of short, uniform items, like routing codes | Possibly | A small tuned model can be cheaper and consistent |
| Producing a rigid output format a general model keeps breaking | Possibly | If brief changes have failed |
| Teaching the agent your policies or products | No | Policies change; sources can be updated the same day |
| Making the agent safer | No | Safety comes from bounds and certification |
| Fixing a drop in quality after a provider update | No | Investigate the change; see what model drift is |
What do you take on if you fine-tune?
More than a training run. The tuned model holds patterns from your data in its weights, so you need to know what data went in, whether it included personal data and how you would respond to a request to delete someone's information. Every retraining produces a new model that must go back through certification. If the base model is retired by its provider, you may have to start the tuning again.
There is also a model risk question. In banking, a tuned model is more clearly a model in the sense of supervisory expectations such as the PRA's SS1/23 principles or the US interagency guidance SR 11-7, with validation and documentation work that follows.
What does the decision look like in practice?
Say an insurer's claims team finds that its intake agent often misclassifies the cause of loss on home claims, coding escape of water as storm. The data science team proposes fine-tuning on 20,000 labelled past claims.
The process owner first asks why. Reviewing failed cases shows two causes: the agent reads the customer's own description, which often says "storm", ahead of the assessor's report, and the brief has no examples of the distinction. The team changes source priority so the assessor's report wins where it exists, and adds four examples. Recertification shows the misclassification largely gone. The fine-tuning project is shelved, saving months and a new model to validate. Our guide to improving an AI agent without retraining the model collects more of these fixes.
Who answers for the decision?
The process owner, with model risk or the equivalent function if your organisation has one. Fine-tuning should need a written case showing that brief, source and bounds changes were tried and failed, a data inventory for the training set, and a plan for recertifying each retrained version.
The rest of the sequence is in our guide to deploying an AI agent.
Where Pryme fits
Pryme qualifies agents rather than training them. We never update model weights for a customer. Improvements happen in the agent's brief, sources, bounds and certification, so every change can be read, tested in Attest on your own cases, and undone.
Questions people ask
Does fine-tuning reduce hallucination?
Not reliably. An agent that states facts it cannot trace to a source is better fixed by requiring grounding and refusing when the source is missing. Certification can then test that behaviour directly.
Is fine-tuning the same as training on our data?
It is one way of training on your data, changing the model's weights. Giving the agent access to your data as sources at the moment it works is different, and is usually what teams need.
Can we fine-tune later if we need to?
Yes. Keep your case sets, labelled outcomes and filed verdicts. If a narrow task later justifies tuning, you will have the evidence and the test material ready.