Self-hosted AI for German companies: keeping data in the EU

Almost every German company we talk to wants to use AI, and almost every one asks the same question first: where does our data go? That is the right question. Customer data, contracts, internal documents and technical drawings should not end up in a system nobody has checked. The good news: in 2026 there are solid options to use powerful AI while keeping data in the EU or even inside your own building.

This article explains the three main hosting options, what they mean for GDPR and the EU AI Act, and how to choose. It is not legal advice; involve your data-protection officer early.

Why "just use ChatGPT" is not an answer

Consumer AI accounts are convenient, but for business use they raise problems: unclear data processing terms, possible use of inputs for model training, data transfers to the USA and no audit trail. Employees pasting customer emails into a consumer chatbot is one of the most common data-protection risks we see today.

The solution is not to ban AI. It is to provide a sanctioned, well-configured alternative that is better than what people would use on their own.

Option 1: Managed models in EU regions

Large providers offer their models through cloud platforms with EU data residency, for example Azure OpenAI in Sweden or Germany, AWS Bedrock in Frankfurt (with models from Anthropic, Mistral and others) or Google Vertex AI in European regions. Under enterprise contracts, inputs are not used for training, and you sign a data processing agreement.

Advantages: access to the most capable models, no hardware, pay per use, quick to start. Disadvantages: the provider is often a US company, so transfers and access requests remain a topic for your DPO; costs grow with volume; you depend on the provider's model versions.

Typical use: document assistants, email drafting, summarisation and agents that work with business data that is not highly sensitive.

Option 2: Open-weight models on EU servers

Models such as Llama, Mistral and others can be downloaded and run on your own GPU servers, rented from a German or European hosting provider. Data never leaves infrastructure you control.

Advantages: full control, no per-request fees, models can be fine-tuned on your data, no transfer to third countries. Disadvantages: you need GPU capacity and operations know-how; open models are very good but may trail the very best managed models on complex reasoning; you are responsible for updates and security.

Typical use: internal knowledge assistants over sensitive documents, classification and extraction at high volume, and use cases where predictable costs matter.

Option 3: Fully on-premise

The same open-weight models running on hardware in your own data centre or even on a single workstation with a strong GPU. For computer vision, models like the YOLO family run on small edge devices next to the camera. Our ALPR parking system is an example: plate recognition can run on site, and only events leave the location.

Advantages: maximum control, works offline, no data leaves the building. Disadvantages: hardware investment, maintenance and limited capacity for very large models.

Typical use: production, healthcare, critical infrastructure, works-council-sensitive scenarios and anything with strict confidentiality requirements.

How to choose

Ask four questions:

1. How sensitive is the data? Public product information is different from patient records or HR files.
2. How much volume do you expect? A few hundred requests per day are cheap with managed models; millions of documents per month may justify your own GPUs.
3. What quality do you need? Test the candidate models on your own examples. An evaluation set of 50–200 real cases tells you more than any benchmark.
4. Who will operate it? Self-hosting needs monitoring, updates and on-call support.

Many companies end up with a mix: managed EU models for general productivity, self-hosted models for sensitive data.

GDPR in practice

Whatever you choose, the basics stay the same:

• Legal basis and purpose: document why personal data is processed by the AI system.
• Data processing agreement: with every provider that processes personal data on your behalf.
• Data minimisation: send only what is needed; mask names and IDs where possible.
• Retention: define how long prompts, outputs and logs are kept.
• Transparency: tell employees and, where relevant, customers that AI is used.
• Data protection impact assessment (DPIA): often required for systems that evaluate people or process sensitive data at scale.

The EU AI Act in two minutes

The EU AI Act classifies AI systems by risk. Most business assistants and document tools fall into the lower risk classes with transparency obligations. Systems used for things like recruitment decisions, credit scoring or critical infrastructure can be high-risk and come with requirements for documentation, human oversight and quality management. Since 2025, companies must also ensure sufficient AI literacy among staff who use AI systems. Classify each use case early; it shapes the architecture.

Architecture patterns we use

• Retrieval-augmented generation (RAG): the model answers from your documents, retrieved from a vector database in the EU, with citations. The model never needs to be trained on your data.
• Model gateway: one internal API in front of several models, with authentication, logging, cost limits and the ability to switch models without changing applications.
• PII filters: personal data is detected and masked before it reaches a model.
• Evaluation and monitoring: test sets, quality dashboards and alerts when answers drift.

What it costs

A first productive assistant with retrieval over a well-defined document set, hosted on managed EU models, typically costs €25,000–€60,000 to build, plus usage fees. A self-hosted setup adds GPU costs, from a few hundred euros per month for a rented GPU server to five-figure investments for on-premise hardware. A two-week AI sprint is the fastest way to get real numbers for your case.

Frequently asked questions

Is a self-hosted model as good as the big commercial ones?

For many business tasks (summarising, classifying, extracting data from documents, answering questions from a defined knowledge base) current open-weight models are good enough, especially combined with retrieval. For complex reasoning across many steps, the leading managed models are often still ahead. Test both on your own examples.

Can we start with a managed model and switch later?

Yes, if you build on a model gateway and keep prompts, retrieval and evaluation independent of a specific provider. That is how we build every AI system, so switching becomes a configuration change, not a rewrite.

What hardware do we need for self-hosting?

That depends on the model size and the number of simultaneous users. Smaller models run on a single modern GPU; larger models need several GPUs or quantisation. Renting GPU servers from a European provider is a good way to measure real requirements before buying hardware.

How do we stop employees from using unapproved tools?

Offer a better approved tool, explain why it matters, and back it with a short AI policy. Bans without alternatives rarely work.

Conclusion

You do not have to choose between AI and data protection. Managed models in EU regions, open-weight models on European servers and fully on-premise setups each have their place. Start with one concrete use case, test models on your own data, involve your DPO, and build on an architecture that lets you switch models as the market evolves. If you want to know where you stand, take our AI readiness check.

Book a 2-week AI sprint