Few decisions in enterprise AI generate as much debate as this one: should we build or fine-tune a model of our own, or use a frontier model through an API? The debate often goes in circles because it is framed as a single choice for the whole organization. In practice it is a series of choices, one per workload, and the right answer for your contract review assistant may be wrong for your customer service voice agent.
This post offers a framework we use with clients. It is deliberately simple. The aim is to make the trade-offs explicit so the decision can be explained and revisited.
First, define the options clearly
"Custom LLM" covers several distinct approaches, and they differ a great deal in cost and effort:
- Frontier API. You call a hosted model from a major provider. You control prompts, retrieval and tools, but not the model or where it runs.
- Frontier model in your cloud tenancy. Some providers offer their models through cloud platforms with regional hosting and enterprise data terms. Control improves, but the model is still not yours.
- Open-weight model, self-hosted. You run a publicly released model in your VPC, data center or sovereign environment. You control the runtime and the data path.
- Fine-tuned model. You adapt an open-weight model to your domain, tone, formats or tasks using your own data.
- Model trained from scratch. Rarely justified outside a few organizations with very large, distinctive data and long time horizons.
Most real decisions sit between the first four.
The six questions
1. Where must the data and the model reside?
Start here, because it often settles the matter. If a workload handles classified information, regulated health records, export-controlled technical data or material bound by contractual residency terms, your options may be limited to models you host yourself. Ask your legal, privacy and security teams for the actual constraint, in writing, rather than working from assumptions in either direction.
2. What quality bar does the task require, and can a general model meet it?
Build a small evaluation set from real examples, with expected outputs that a domain expert has checked. Run the strongest general model you are permitted to use, with good prompts and retrieval. If it clears the bar, you may not need anything custom. If it misses in consistent ways, such as a specialized vocabulary, a strict output format or domain reasoning it does not know, fine-tuning may close the gap.
3. What will it cost at your expected volume?
API pricing is easy to start with and scales linearly with use. Self-hosting has a high fixed cost in GPUs, engineering and operations, but a lower marginal cost per request once utilization is high. The crossover point depends on volume, model size, latency targets and how well you keep hardware busy. Model your own numbers. A smaller fine-tuned model that does one task well can be far cheaper to serve than a large general model, but only if the volume justifies the setup.
4. What latency and availability do you need?
Voice agents, real-time operations and edge deployments often need low, predictable latency and must keep working when an external service is degraded. Self-hosted models give you more control over both. Batch workloads, such as overnight document processing, are far more tolerant.
5. How much control do you need over change?
Hosted models are updated and retired on the provider's schedule. For many tasks that is fine. For regulated workloads where a model change may require revalidation, or where outputs must stay stable for audit, owning the model version is a real advantage. If you use an API, pin model versions where the provider allows it, and keep your evaluation suite ready to rerun when a version changes.
6. Can you operate it?
A custom model is a product, not a project. Someone must own its training data, evaluation, serving infrastructure, security patching, monitoring and retraining. If you do not have, or cannot build or buy, that capability, the theoretical benefits of a custom model will not arrive.
A simple decision path
- If residency or control requirements rule out external hosting, go to self-hosted open-weight models, and fine-tune if evaluation shows a gap.
- If not, try the strongest permitted general model with good retrieval and prompts. If it meets the quality bar at an acceptable cost, use it.
- If it meets quality but not cost or latency at your volume, evaluate a smaller self-hosted or fine-tuned model for that specific task.
- If it misses quality in consistent, domain-specific ways, run a fine-tuning experiment and compare against the same evaluation set.
- Revisit the decision when volume, regulation or available models change.
Common mistakes
- Fine-tuning to add knowledge. Fine-tuning is better at teaching format, style and task behavior than at teaching facts. For facts that change, retrieval is usually the better tool.
- Skipping the baseline. Without a measured baseline from a general model, you cannot show that a custom model is better.
- Underestimating data work. Most of the effort in a fine-tuning project is collecting, cleaning, labeling and reviewing examples.
- Treating it as a one-time decision. Open-weight models and hosted offerings both improve quickly. A choice that was right a year ago may not be right today.
The likely answer: a portfolio
Most enterprises we speak with arrive at a portfolio. Sensitive or high-volume tasks run on private models they control. General reasoning and low-volume tasks use frontier models where policy allows. A routing layer decides which model handles each request, based on data classification, cost and quality targets, and records the decision.
The value of the framework is that each placement has a written reason. When a regulator, an auditor or your own board asks why a workload runs where it does, you have an answer.
We help organizations make and carry out these decisions, from evaluation design to private deployment in your cloud, VPC or on-premises environment. See our sovereign and on-premises AI services, or contact us to talk through a specific workload.