← Denys KozedubCase 01 · Private company LLM
Case 01 · Private company LLM · a software product with tens of thousands of daily users

A model that belongs to the company.

An open model, trained on one product and run on one card inside the company. Think of it as a senior employee who knows every screen, every setting and every API call of your product, and nothing else. Customers describe what they need in plain words; it sets the product up for them. No prompt, document or customer record ever leaves the building, and the bill does not grow with every message.

Watch it do a taskWhen it pays off
Short on time?Business reader: what it knows · is this for you · when renting is not a choice · when it pays offTechnical reader: how it is built · the loop · what is inside · control · the tech
Model35B · 3B active
Runs on1 card · 96 GB
Tools40–50 via MCP
Calls to outside AI0
Fig. 1 · Inside the company · the product and one GPU server
GuardQwen3Guard · filters
Help center & docsRAG · re-indexed per release
Customerschat in the product
The product’s API40–50 tools via MCP
Cachesanswers · prefixes
Audit & metricsOpenTelemetry · Langfuse
The company’s LLMQwen3.6 · LoRA · vLLM · 1 GPU
Outside AIChatGPT · Claude · Gemini
One card, inside the company’s walls

The model, the documents, the caches and the logs all run on the company’s own hardware. Hover any part of the map to see what it does.

What it knows

It knows your product like your best employee. And nothing else.

A general model knows a little about everything. This one is trained on one product: every setting, every screen, every API call, and the order things must be done in. Ask it about the weather and it will politely say that is not its job.

Live: what customers ask, what it does
in scope“Create a sales pipeline with steps New, Negotiation, Paid”Done: pipeline created, three steps in order. It knew that steps need a pipeline first.
Knows where everything is

Which setting, which screen, which API call, which field. It has seen every tool the product offers and hundreds of ways people ask for things.

Knows the order of things

A pipeline before its steps, a payout method before the first payment. When something is missing it asks, it never guesses.

Refuses everything else

No weather, no poems, no other companies’ accounts. That keeps it cheap to run and hard to misuse. Fine-tuning plus a guard in front make sure of it.

Is this for you?

Built for big flows. Honestly not for everyone.

Your own model is a serious piece of infrastructure. It wins when thousands of people talk to it every day, or when the data must never leave. Below that, a rented model does the same job for less.

Built for

Products and companies with thousands of AI conversations a day

Software products where customers set things up in a chat: website builders, online stores, accounting, delivery and booking platforms, marketplaces
Banks, fintech, insurance, telecom, clinics, defence and government suppliers: anywhere the data must never leave, by law, by contract or by common sense
Companies that want to own the thing: the model, the weights and the know-how stay with you, like your ERP does
Not for

A company with a few hundred conversations a day

A service company, a shop, an agency: an agent on a rented model knows your documents and acts in your systems for a fraction of the cost
Anyone who needs it next month: your own model takes four to six weeks to a first version and three to six months to production; a rented-model agent is live in days
The same agent can move onto your own model later. Nothing is thrown away
See the rented-model path on the homepage
When renting is not a choice

Some companies cannot show their kitchen to anyone. For them this is the only way.

To build any assistant, a model has to study the business from the inside: every API endpoint, the internal documents, years of real customer records, the rules nobody wrote down. Then tools are built on top of what it learned. If that model belongs to someone else, the whole inner kitchen has just crossed the perimeter, during the build and in every request after. A bank, a clinic, a defence supplier or a company with thirty years of records will not allow that. Careful companies don’t. For them the question is not own versus rented. It is only how to build their own well.

If any one of these is true for you · untick what is not
6 of 6 true for you

Renting is off the table. Only the build is left to decide.

Any one of these means no outside model may study the company, so the price comparison further down is beside the point. The only question left is how to build your own model well: the right open model, your data, your card, your rules. That is what the rest of this page is about.

What an outside model would have to see

Not a demo dataset. The real thing: the full list of endpoints and what each one does, the admin documentation, thousands of real tickets and records, the pricing logic, the exceptions. That is the material the tools are cut from. There is no building an assistant on less.

endpoints · docs · records · rules

Why “not used for training” is not enough

A business contract promises the provider will not train on your data and will not keep it. It does not change the fact that the data left your perimeter. A regulator, an auditor, a client’s lawyer or your own security team asks only that one question.

GDPR · NDAs · sector rules · security review

Who this usually is

Defence and government suppliers. Banks, insurers, payment companies. Healthcare and pharma. Law firms. Telecoms. And any company whose decades of records, prices and customer base are the business itself, not a by-product of it.

and everyone with a customer who says “never”
The story

Why a company builds its own model instead of renting one.

A software product with tens of thousands of people using it every day. Details changed, the shape of the problem is common.

01 · Before

Setup was a paid service

New customers did not configure the product themselves. A handful of integrators studied each customer’s business and set up funnels, fields, filters and dashboards by hand, by the hour. Slow, expensive, and only for customers who could pay for it.

02 · The first attempt

A chat on a rented model

The idea was right: let the customer describe their business in a chat and let an assistant do the setup. But on a cloud model every message is billed, and customers type anything at all. The projected bill ran into six figures a year, with no ceiling.

03 · The build

The company’s own model, on one card

An open model trained on the product, served on the company’s own GPU, with cheap layers in front of it so the expensive part runs only when it has to. Flat cost, data inside, and an assistant that knows the product better than a general model ever could.

How it is built

From the inside of your product to a model that runs it. Five phases.

No magic step. The work is to understand the product completely, cut it into tools a model can use, and prove with tests that the model uses them right.

Phase 01 · weeks 1–2

Study the product from inside

Every screen, every API route, every validation rule and every allowed value, read from the code and the API spec, plus what support actually gets asked. The output is a map of everything the product can do and how people ask for it.

One request, one loop

The brain thinks. The agent acts. Round and round until the job is done.

The model never touches your systems itself. It reads, decides which tool is next and hands the call to the agent. The agent executes it through an MCP server, brings the result back, and the brain decides again: another tool, a question, or the final answer. Before any of this, the cheap layers in front (guard, cache, FAQ) have already answered what they could; the next section shows that filter.

A requestfrom a customer, in plain words
The brainfine-tuned LLM · reads, decides, writes
The agentruntime · executes tools with the user’s login
MCP server · Docs search
MCP server · Product API
MCP server · Billing, email…
picks tool →← result← answer
The request

“Create a sales pipeline with steps New, Negotiation and Paid, and a filter for deals with no owner.” The guard and the caches have let it through; it reaches the brain.

1

The request arrives. The guard and the caches have already let it through.

2

The brain reads it, opens the right playbook and picks the first tool with its parameters.

3

The agent calls that tool on the MCP server, with the customer’s own login.

4

The result comes back to the agent and then to the brain, as data, never as instructions.

5

The brain decides: next tool, a clarifying question, or the answer. The customer sees nothing until the last round; a setup task is usually five to ten rounds.

6

One agent is enough for one product. When it grows, agents per service work under one coordinator: the same loop, one level up.

If you don’t need your own model

The same loop runs on a rented brain. Only the brain moves.

Most companies start here, unless the data must never leave (see when renting is not a choice). The agent, the tools, the MCP servers, the guard and the caches are exactly the same build. The difference is one line: where the brain’s requests go. Here they go to an API endpoint of a cloud model instead of a card in your rack.

Option A · a rented model · this section
The braina cloud model behind an API endpoint: Claude, GPT or Gemini
Where a request goesto the provider, or to Bedrock inside your own AWS account
What you payper request; grows with every user and every message
Who sees the datathe provider, under a business contract: not used for training, not kept
Live indays to two weeks
Option B · your own model
The brainQwen + your adapter, served by vLLM on your card
Where a request goesyour server, internal network only
What you paythe card, electricity, the engineer; the same at 10 or 5,000 users
Who sees the datanobody outside the company
Live infirst version in 4–6 weeks; production in 3–6 months

What Bedrock is

An AWS service that hosts other companies’ models (Claude among them) inside your own AWS account. Your agent calls an endpoint in your account; AWS runs the model. No servers to manage, and the traffic never leaves your cloud perimeter.

Amazon Bedrock · also: Anthropic, OpenAI, Google APIs directly

Nothing is rebuilt when you switch

The agent talks to the brain through one standard API shape, the same one vLLM speaks. Moving from a rented model to your own, or back, is a config change plus the test set run again. Tools, playbooks, guard and logs stay.

OpenAI-compatible API · same evals

What is still yours to decide

Which model, how much it may spend per customer, which actions need a person’s yes. Rented does not mean uncontrolled: the guard, the budgets and the audit sit in your agent, not at the provider.

guard · budgets · Human in the Loop
How it works · 1 of 3

Most messages never reach the expensive part.

Every message passes five layers, cheapest first. Each layer answers if it can and stops the message there. Only real work reaches the model that costs GPU time. Watch customer messages come in.

Message 1 of 12 · from a new customer“Create a sales pipeline with steps New, Negotiation, Paid”
→ stopped at: Agent
1GuardOff-topic, abuse, prompt injectionno model run0
2CacheA question somebody already askedno model run0
3FAQ“What is this? What can it do?”no model run0
4Docs answerA how-to nobody asked before1 short call0
5Agent“Set this up for me”~10 calls, the only expensive path1
Agent

The fine-tuned brain picks a skill, calls the product’s API step by step, checks each result, asks when unsure. About 10 calls per task.

Counters show how many of the messages so far each layer handled. In production the split is measured from logs; the cheap share grows with use.

How it works · 2 of 3

Watch it do a task, step by step.

When a message needs real work, the model plans, calls the product’s own API one step at a time, reads each answer and keeps going. In plain words on the left, the technical name on the right.

Pick a task
Create a sales pipeline with steps New, Negotiation and Paid, and add a filter for deals with no owner.
What to notice
It sees 6 tools, not 50A playbook (Agent Skill) opens only the tools for this kind of task. Small models stay accurate that way.
The model never touches the productIt returns a request; the runtime executes it with the customer’s own login and gives the answer back.
Errors are read, not hiddenA refusal from the API goes back to the model, which fixes its call or asks a question.
Risky actions waitDelete and mass update need a yes from a person first. Every step is logged.
Off topic gets a polite noWeather, bitcoin, poems: no playbook matches, no tool exists, and the model was trained to say so instead of improvising. It knows your product, not the world.
How it works · 3 of 3

What is inside. Five layers and four small models.

Nothing exotic: open models, open-source servers, and a few hundred lines of glue written for this product. Everything runs on hardware the company owns.

Brain

The fine-tuned model. Decides what to do: which playbook, which tool, which parameters, when to ask. The only part that is trained.

Playbooks

Agent Skills: one folder per scenario with instructions, the 5–7 tools it may use and examples. Loaded on demand.

Tools

40–50 atomic actions generated from the product’s API spec, exposed through MCP, validated against a schema before they run.

Runtime

Strands Agents: the loop that calls the model, executes tools with the user’s login, enforces policies, asks for confirmation, writes the audit log.

Knowledge

The help center, indexed for search (RAG). Lives next to the model, never inside it, so it is always current.

ModelRoleSize
Qwen3.6-35B-A3B + LoRAThe brain. 35B parameters, about 3B active per word, so it answers fast on one card. The adapter teaches it this product.35 GB in FP8
Qwen3-EmbeddingTurns text into meaning-vectors for search, the answer cache and playbook routing.0.6B
Qwen3GuardChecks every message in and out for abuse, injection and policy breaks. 119 languages.0.6B–8B
Qwen3-RerankerOptional. Re-orders search results when plain search is not precise enough.0.6B
vLLMServes the brain to hundreds of people at once, reuses shared prompt prefixes, loads a new adapter without a restart.server
Qdrant · RedisThe vector database for documents and the cache for answers and sessions.on existing servers
How it learns

Trained on behaviour, never on customer data.

The model learns how to use the product: which tool, which fields, how to chain calls, when to ask. It never memorises customer records; those are fetched through tools at the moment of the request, under that customer’s own login.

Step 01

Tools from your API spec

Every action the product’s API offers becomes a tool with an exact schema: required fields, allowed values, error texts. 40–50 of them, grouped into playbooks (skills).

Hardware for trainingOne rented card, hoursExperiments on a rented RTX 4090 or free Kaggle hours; the production adapter trains in hours on one 96 GB card.
Never in the training dataCustomer records · outputs of ChatGPT or ClaudeRecords would be memorised and leak between accounts. Other vendors’ outputs are off-limits by their terms and by the no-outside-model rule.
How it stays aliveWeekly or monthly retrainScrubbed logs and new tools go into a new adapter; it ships only if it beats the old one on the same tests.
Control

Six rules that are enforced by code, not by the model’s good will.

A model can be persuaded. A runtime cannot. The rules below sit around the model and apply to every single request.

Nothing leaves

The model server is on the internal network only. No outside AI provider is connected, so there is nothing to leak to.

vLLM internal · no external models

Only allowed tools

Each playbook lists its tools; everything else does not exist for the model. Arguments are checked against a schema before anything runs.

Cedar policies · JSON schema

Acts as the user

Every call runs with the customer’s own login. The assistant can do exactly what that person could do by hand, and nothing more.

user token · API permissions

Risky actions wait

Delete, mass update, anything that cannot be undone stops and asks a person first. The answer is logged with the action.

Human in the Loop

Budgets

Steps per task, tokens per day per customer, time per request. A runaway loop is cut off; a customer who hammers the chat gets a cooldown.

per-session · per-user · per-tenant caps

Everything is traced

Every model call and tool call is recorded with its cost and outcome. Undone actions and thumbs-down become training signals.

OpenTelemetry · Langfuse · audit log
When it pays off

Cloud grows with every user. Your own card does not.

Move the slider to the number of people who would talk to the assistant on a typical day. The chart shows the total you would have spent after each month, over three years, against the cheapest models of the three big vendors.

1,000AI users a day
about 25,000 model requests a day
Connect an API and move on.

At 1,000 AI users a day Average of the three costs about $2.5k a month. Your own card plus the engineer who keeps it alive ($10k a month) would not pay back within three years. Build the agent on a rented model; the same agent can move to your own model later.

Total spent, year by yearCloud is cheaper over three years, about 4.4× · at today’s list prices
$30k
$142k
After year 1
$59k
$266k
After year 2
$89k
$391k
After year 3
Rented cloud model (Average of the three)Your own model, card boughtBreak-even: beyond 3 years
Option A · a rented cloud modelno hardware
$2.5ka month
$30ka year · $89k over three years
Which model
Price per request$0.0033
Per year$30k
Over three years$89k
If cloud prices fall
Same volume on a top-tier model (Claude Sonnet 5.5)$26k a month

The bill grows with every user and every message. Nothing to buy, live in days; your prompts and data go to the provider under a business contract. The top-tier line is where the real story started: the company’s first chat ran on a model of that class.

Option B · your own model on your own card1 card · up to 5,000 users a day
$142kin year one: the server bought, 12 months of electricity and hosting, the engineer
$124kevery year after · $391k over three years
Hardware, bought once$18k
Electricity and hosting, per month$350
The engineer who keeps it alive (MLOps / LLM engineer), per month
Running costs per month, with the engineer$10k
Over three years$391k
… that is, per month if the server is spread over 3 years$11k
Renting the card instead of buying (with the engineer)$11k a month

The cost is the same whether 10 or 5,000 people talk to it. One card handles up to about 5,000 AI users a day; the next card is added only when the queue says so. The engineer is the biggest line: retraining, metrics, base-model refresh. Like your ERP admin, a normal budget line, not a project.

Per request12,000 tokens in · 1,000 outAn agent step carries the instructions, the tools and the conversation so far. That is why agent work is expensive on a per-token price.
Per AI user per day25 requestsTwo or three setup tasks of ~10 steps each, plus questions. Setup is mostly one-time; real usage drops after onboarding.
Cloud pricesCheapest tier of each vendorClaude Haiku 5.5 $0.10 / $0.50; GPT-5.4 nano $0.20 / $1.25; Gemini 3.1 Flash-Lite $0.25 / $1.50 per million tokens in / out. October 2026.
Own hardware$18k per server, onceOne RTX PRO 6000 card (96 GB) at ~$12–16k plus the box around it (CPU, 128 GB RAM, power, SSD) ~$3–5k. Electricity ~$150 a month; hosting ~$200 a month in a rented rack, close to zero in your own server room. Renting the same card: ~$700 a month.
Capacityup to ~5,000 AI users a day per cardThe project’s sizing: up to ~5,000 users a day on one card, 5,000+ on two to four (70–150 conversations at once per card). Measured on a rented card before anything is bought.
The engineer$10k a month by defaultAn illustrative average salary for an MLOps / LLM engineer; switch to part-time or “already on the team” above. Not in the numbers: a second server for failover.
Illustrative, not a quote. Every figure above is computed from the stated assumptions and the list prices on the stated date; prices change, and the switch “if cloud prices fall” shows how much that matters. Prompt caching can cut a cloud bill by up to half; your own card’s line does not move. Cloud models may be better or worse than your own on your tasks: the same test set decides, before anything is bought.

Beyond money: the data

No prompt, document or customer record is ever sent outside, not during the build and not in a single request after. For a bank, a clinic, a defence supplier or a company whose records are the business, this is the argument, not the price.

Beyond money: no dependency

No outside price change, rate limit, model retirement or outage can touch the assistant. The weights are the company’s, forever.

Beyond money: it knows the product

A general model knows a little about everything. This one is trained on the product’s own API and tested on its real tasks.

Hardware

It starts on one card.

One workstation-class GPU with 96 GB of memory holds the whole stack with room for hundreds of conversations at once. A second card is added when the queue says so, not before.

One RTX PRO 6000 · 96 GB · what fits
Brain · 35 GB
+6
Room for conversations · ~55 GB
Brain, Qwen3.6-35B-A3B in FP8~35 GB
Guard + embedding model~6 GB
Working memory for open conversations~55 GB
Conversations generating at the same moment~70–150
AI users a day, under uneven real load~1,000–5,000
Output speed, all users together~8,000 tokens/s

Estimates from memory math on a related model generation; measured on a rented card with a one-hour load test before any purchase. The measured numbers are what a client gets.

Rent first, buy when steady

Production starts on a rented card (~$700 a month) in a GPU cloud or inside the company’s own cloud account. Buying a server (~$18k) beats renting after about three years if it sits in your own server room; in a rented rack the two cost about the same. Buy when the card must physically stay in the building, rent for everything else.

Scaling signal: the queue

Requests wait in a queue sized so the first word appears within a few seconds. When the wait grows for more than a few minutes, a card is added. GPU utilisation is not the signal; wait time is.

Everything else is small

The agent runtime, the vector database, the cache and the monitoring run on ordinary servers the company already has. Training uses rented hours only when a new adapter is due.

The honest part

What you take on.

It is narrower than a frontier modelTrained for one product, it is excellent at that and ordinary at everything else. A general cloud model is stronger on vague, unusual requests. The same test set shows the gap, before you decide.
It needs a keeperSomeone retrains it, watches the metrics and refreshes the base model every six to twelve months. Count an MLOps / LLM engineer in the budget, the way you count your ERP admin; the cost block above does. Most of the pipeline is automated; the role is one person, not a team.
Small volume means cloud is cheaperBelow a few thousand AI users a day (a few hundred if the engineer is already on the team) the card sits idle and the API wins. Start on a rented model and move later; the agent, tools and tests carry over unchanged.
One card is one point of failureIf the card dies, the assistant is down until it is replaced. Failover means a second server. On a rented card the provider handles it.
Months, not daysFour to six weeks to a first version on real tasks; three to six months to production with the training loop, monitoring and handover in place. A rented-model agent is live in days. The time is the price of owning it.
Base models move fastA better open model appears every few months. Good news for quality, but the adapter must be retrained on it. Automated, cheap, still a task.
The tech, in plain words

Every name on this page, explained.

Qwen

An open model family from Alibaba (Apache 2.0 licence, free to use commercially). Production target: Qwen3.6-35B-A3B, a mixture-of-experts model with 35B parameters where only ~3B work on each word, so it answers at small-model cost with big-model quality.

How a pilot goes

Four to six weeks to a first version. Three to six months to production.

The first version proves the idea on your real tasks. The months after it turn that into something your team can run: the retrain loop, monitoring, failover, handover. Built so that every step is a script with a config file.

Weeks 1–2

Tools and the test set

Your API spec becomes 40–50 tools and a few playbooks. 50–200 real requests become the test set that will judge every version.

Weeks 2–4

The agent on the base model

The untouched open model, the runtime, the guard and the docs search, on a rented card, against a sandbox account. First score.

Weeks 4–6

Training and the gate

Traces generated and verified, the adapter trained, scored on the same tests. It ships only if it beats the base model.

Months 2–6

Production and handover

Real traffic on a rented card, then yours. The retrain loop runs on logs: logs in, new adapter out, eval gate in between. Monitoring, budgets, failover. Your engineer takes over with a dashboard that shows cost per request and which layer answered.

What stays with you at the end
The adapterthe trained weights, yours to keep and retrain
The tools and playbooks40–50 MCP tools generated from your API spec, with their tests
The test setthe real tasks that judge every future version
The pipelineevery step a script with a config file: traces, training, eval gate, deploy
The dashboardscost per request, which layer answered, tool errors, thumbs-down
The know-howyour engineer runs it after handover; no dependency on me or on any vendor
Contact

Have a flow like this?

Tell me how many people would talk to it and what they would ask for. I reply by email with whether your own model makes sense, or whether a rented one is the smarter start.

[email protected]