· Luuk
Running your own AI model: why we do it and how you can too
Everything you ask ChatGPT, Gemini or Claude ends up with an American company. So we run our own AI model in Europe. What it costs, how it works and how to start.
- Written by
- Luuk · STAIBL
- For
- Business owners and technical readers
- Reading time
- 6 minutes, or 1 for the essentials
In short
- Everything you ask ChatGPT, Gemini or Claude ends up with an American company.
- So we run our own AI model on a server in Europe. Nothing goes to OpenAI, Google or Anthropic.
- An answer usually starts within 2 seconds.
- A server for a mid-sized model costs €750 to €1,000 a month, plus a few hours of maintenance.
01
Why your own AI model?
We promise our clients that their data stays in Europe. With ChatGPT, Gemini or Claude you can't really keep that promise. Your data goes to an American company, and even if that company has servers in Europe, it still falls under American law such as the CLOUD Act. So when they say they don't train on your data, you have to take their word for it.
You also depend on their decisions. If the price goes up, a model disappears or a new limit arrives, you find out when it happens. And you pay per user or per word, while your own server costs a fixed amount per month however much your team uses it.
The strongest American models are still better at hard tasks, to be honest. Most work in a company just isn't that hard: internal questions, drafting emails and documents, analysing data. For that, it matters much more that the model is allowed to see your data.
When it's busy, we never fall back to an American provider. If our server really goes down, Mistral takes over, a French provider.
02
How it works
Your own AI model has four parts. You can think of it as a restaurant:
- The chef is the model. It understands language and writes every answer. Technically it's a file of tens of gigabytes; our chef is Gemma 4 with 31 billion parameters.
- The kitchen is the GPU, the graphics card that does the computing. The more memory, the bigger the chef that fits and the more dishes at once.
- The pass is the software that runs the model. It puts the orders in line and lets the chef make dozens at the same time. We use vLLM.
- The host is the gateway. Every application gets its own key, and work someone is waiting for goes before work that can wait. We use LiteLLM.
An evening in the kitchen
How a question travels through your own AI model, from the door to the plate.
2.1
Which chef do you choose?
Which model is best depends on your task and your hardware. Look at three things, then test for yourself:
- Does it fit in your kitchen? Count 2 GB of memory per billion parameters, or a half to a quarter of that for a compressed (quantised) version. A 30-billion model fits on a 48 GB card that way.
- Are you allowed to use it? Apache 2.0 and MIT allow almost anything, commercial use included. For other licences, read the terms.
- Can it do your work? Ask two or three models your own ten questions and compare the answers.
Gemma
by Google, 31 billion parameters
Chosen for our own work and our single 48 GB card: it fits, writes natural text and it's fast. Another model may suit another company better. A large model like DeepSeek is stronger at many tasks, but needs several cards.
Qwen
Alibaba
The widest choice of sizes, many languages, strong at maths and code.
Mistral
Mistral AI, France
European and multilingual. Mistral describes its newest large model as strong in cybersecurity.
DeepSeek
DeepSeek
Strong at reasoning and programming, under the open MIT licence. The large versions are among the strongest open models, but need several cards.
Llama
Meta
Lots of tooling around it, but a licence with restrictions for users in the EU.
You'll find them all on Hugging Face, with the size, the licence and how to start each model. New versions follow each other quickly.
2.2
For the technically minded
Your applications talk to the gateway the way they talk to OpenAI: the same API, a different address. Existing software usually moves over without rebuilding.
vLLM takes every running conversation along in each compute step (continuous batching) and remembers what it has already read (prefix caching). For us, 59% of all input text came straight from memory that way.
03
What it costs
What you need depends mostly on the model: the bigger it is, the more memory. After that, it's how many people use it at the same time.
Free
on your own computer
Try it out
With Ollama or LM Studio and a small model. Enough to test whether it handles your questions.
€750 to €1,000
per month
One GPU server
A model of around 30 billion parameters on a 48 GB card, at a European provider. A team can work with it every day.
On quotation
per month
Heavier
Large models, such as the biggest DeepSeek, or very many users. On data-centre cards such as the H100 or H200.
€12,000 to €35,000
one-off
Own hardware
The server in your own building, plus power, cooling and upkeep.
On top of that, count on a few hours of maintenance a month: updates, monitoring and a plan for when it goes down.
04
How to start
- 1
Pick one task
For example, answering internal questions from your handbook, or drafting quotes. Write down ten real examples with it, and you have your test straight away.
- 2
Try it for free
An afternoon with Ollama or LM Studio. If a small model already works, a bigger one certainly will.
- 3
Rent a European GPU server
From Hetzner, OVH or Scaleway, with enough memory for your model. No American cloud, not even with servers in Europe.
- 4
Install the software
The driver, Docker, vLLM with your model and a gateway such as LiteLLM. Make sure it's only reachable over a secure connection.
- 5
Connect your first application and measure
How long does someone wait for the first word? Agree on a standard, such as 9 out of 10 answers within 10 seconds.
- 6
Arrange the maintenance
Who does the updates, who notices when it slows down, and who takes over during an outage? For us that last one is Mistral, also in Europe. Without that plan, everything stops the moment your server goes down.
05
What we learned
From the first weeks, with the numbers.
89%
went right with 11 tools, against 72% with all 44
Fewer tools worked better
We tested it with all 44 tools and with only the 11 the model needed. With fewer, it went right more often, and the answer came faster too: 2.1 instead of 3.5 seconds. Do leave each tool's description in, though: that's how the model works out which one to use.
Technical: The prompt shrank from 15,092 to 4,103 tokens.
15×
a made-up session number
The model just filled something in
Ask a tool for something the model can't know, like a session number, and it simply makes something up. Better to remove those fields and fill them in yourself in your code. It costs nothing in quality.
3,073
requests in the 5 minutes after a restart
After a restart, everything came knocking at once
Every application retried at the same moment, and the server simply filled up.
Technical: Let software retry with an increasing wait (backoff), or catch it in the gateway.
1,200 to 1,500
tokens per second, whether 32 or 64 run at once
More at once didn't make it faster
The card just spreads the same computing power over more conversations. If you want more capacity, you need a second card. And never measure this kind of thing during real traffic.
Then two more things, without a number. An open model sometimes says "I've noted that" when it hasn't noted anything, so put a check behind every action that matters. And put whatever keeps a person waiting first: a colleague waiting for an answer goes before a report that can be ready overnight.
06
When you should not do it
If your data isn't sensitive, the main advantage is pretty much gone already. Beyond that, we'd advise against it if:
You hardly use it
A few questions a day don't justify your own server. A European provider like Mistral, paid per use, is cheaper then.
Nobody can maintain the server
Without updates and monitoring you'll have a slow or unsafe server within a few months.
You need the very strongest model
For complex analysis or code, open models still trail the best American ones.
Frequently asked questions
What is a self-hosted LLM?
Is your own AI model safe?
What does your own LLM cost per month?
Is your own model as good as ChatGPT?
Which model can I run myself?
How many people can use it at once?
Can my existing software talk to it?
Rather not build it yourself?
If you'd rather not build and maintain it yourself, we do it for you. With Eigenwijs you get your own AI on a server in Europe: connected to your documents, your email and your systems, and managed by us. If you stop working with us, you keep your server, your model and your data.