# Why frontier models and privacy are not mutually exclusive

> LLMs no longer run in your own data center - at least not without significant investment. If you still don't want to send sensitive data unfiltered to the cloud, you don't need your own GPU cluster, but a router in front of it.

- URL: https://blog.goroot.de/post/hybrid-llm-data-sovereignty/
- First published: 2026-08-19
- Lastmod: 2026-08-19
- 1549 words, 8 min
- Tags: ai, llm, data-sovereignty, self-hosted, privacy
- Autor: Michael Kolb — goroot (https://blog.goroot.de/)

---


Sooner or later, the question comes up in practically every project that's currently getting an LLM feature: Is it even allowed to send what’s being processed here to the cloud? Support tickets with names and contract numbers, internal notes with salary data, code with customer secrets in the context window - as soon as a feature goes beyond the demo stage, a dataset appears that no one wants to send to OpenAI, Anthropic, or Google. And then the reflex kicks in: "Then we'll just host the model ourselves." At this point, the reality of costs intervenes.

## Why "just self-hosting" fails in the face of cost reality

Running a model of frontier quality - GPT, Claude, or Gemini class - yourself is not an option that is realistically on the table for a team of five, fifty, or a hundred people. These models run on clusters with two- to three-digit numbers of H100s. Even if you can afford it, you'll never get the same model quality as the providers themselves, who continuously fine-tune their models.

The more interesting insight: You don't need this model class for every request. Much of what companies send to an LLM is trivial - summarizing, rephrasing, categorizing, simple extraction. An 8B or 14B model on a single consumer GPU is perfectly adequate for this. The question is not "own data center or cloud," but: which request really needs the expensive model, and which does not.

## The real problem is the context, not the model

Data sovereignty is often treated as a deployment decision: "Now we're using an on-premise model." That's too narrow. Whether a request may leave your own infrastructure cannot be answered at the model level, but only per request - at the level of the context contained in the prompt.

Two support tickets from the same queue can have entirely different requirements. "How do I change my password?" is not critical for the cloud. "Mr. Meier, born on April 3, 1981, customer number 88213, complains about the billing of his disability insurance" is. A static deployment - everything local or everything in the cloud - treats both cases the same and is therefore either unnecessarily expensive or unnecessarily risky.

## The router: a sensitivity-aware gateway instead of gut feeling

The component that makes this decision before any model sees the request has several names in practice, depending on where you come from. Providers focusing on PII detection[^2] call it an **AI Gateway** or **LLM Gateway**: the prompt is scanned at the gateway, and if a hit is detected, the request is automatically routed to an on-premise model instead of the cloud API. In research on routing architectures, the term **sensitivity-aware routing** or **semantic routing** appears - a router in front of the inference API decides based on data sensitivity, task complexity, and user group which model receives the request. Closest to what I have in mind is the **hybrid local-cloud architecture**: a trusted local "local controller" handles planning, routing, and simple reasoning itself and only delegates to an untrusted "cloud language model" when necessary.[^1]

What matters is the principle, not the name: classification happens before any prompt crosses a boundary, and in case of doubt, processing is done locally, never delegated to the cloud. Fail closed, not fail open. In code, the decision logic - deliberately simplified - looks something like this:

```go
// Route decides which tier of the stack processes a request.
// Sensitive contexts never leave the infrastructure - even if the local model
// is actually too weak for the task.
func Route(ctx context.Context, req Prompt) Tier {
	if classifier.ContainsSensitiveContext(ctx, req) {
		return TierDatacenterGPU
	}
	if req.Complexity <= complexityTrivial {
		return TierLocalOllama
	}
	return TierCloud
}
```

The classification itself does not need to be complicated. A set of rules based on regex patterns for customer numbers, IBANs, and names from your own CRM catches most cases. For everything that cannot be cleanly captured by pattern matching, a very small local model often suffices - for example a 1B to 3B model that answers only one question: "Does this text contain personal or confidential information? Yes/No." It runs in milliseconds on any halfway recent GPU and does not need to be good at the actual task - only at recognizing when caution is warranted.

## The cascaded architecture: workstation → local GPU in the network → cloud

From the routing principle follows a three-tier cascade:

```markdown
Request
│
▼
Sensitive context detected? ──yes──▶ local GPU in the datacenter (RTX 3090, Qwen2.5-14B etc.)
│ no
▼
Trivial enough for the workstation? ──yes──▶ Ollama locally on the laptop
│ no
▼
Cloud model (Claude, GPT, Gemini, ...)
```

**Tier 1 - Ollama on the workstation.** Autocomplete, short summaries, simple rephrasing run directly on employees' machines without any network request being necessary. Latency near zero, costs near zero.

**Tier 2 - a GPU in your own network.** Everything with sensitive context that requires more performance than a laptop can deliver goes to a central GPU in the datacenter or homelab, served via vLLM or Ollama in server mode. This tier is the real core of data sovereignty: it makes "local" practical without requiring a cluster. With an appropriate queue, this is also scalable for teams with multiple simultaneous users.

**Tier 3 - cloud.** Everything that is neither sensitive nor trivial goes where the best available model quality is. This is the pragmatic part: you don't have to solve every complex task locally just because you can.

## What an RTX 3090 can handle for \~5 employees

The 24 GB of VRAM on the RTX 3090 is why it appears so often in homelab and small-team setups: enough for 8B models with minimal quantization loss and for 14B models in Q4, with room to spare for context.

| Model (quantization) | VRAM requirement | Approx. throughput¹ |
| --- | --- | --- |
| Llama 3.1 8B (Q4_K_M) | \~5 GB | 60-90 tokens/s |
| Qwen2.5 14B (Q4_K_M) | \~9 GB | 35-45 tokens/s |
| Qwen2.5 32B (Q4_K_M) | \~19 GB | 15-20 tokens/s |

¹ Publicly circulating approximate values for single requests, not measured by us - benchmark yourself for production use with multiple concurrent users, especially for parallel requests.

For a team of five people who do not continuously max out the GPU, a single 3090 is usually sufficient for the second tier of the cascade. If more is needed, a DGX Spark for a few thousand euros. Always assuming that truly compute-intensive tasks end up in tier 3 anyway. The rough calculation behind this: 350 W TDP, say 6 hours of active load per workday, amounts to about 42 kWh per month - at €0.30/kWh, that's roughly €13 in electricity. Add depreciation of the card over two to three years. That's far below what five people would spend on cloud API costs if every sensitive request went to a frontier model - especially since compliance-relevant requests tend to be long (lots of context, many documents) and therefore disproportionately expensive with token-based billing.

## Limits: when local models are simply too weak for the job

The honest part of this architecture: a Qwen2.5-14B in Q4 is no substitute for a frontier model. For complex reasoning, long documents with many cross-references, or tasks that require real-world knowledge beyond the training data, you quickly hit limits. If you're not aware of this, you'll end up with worse results and justify them as a data protection compromise - something no one will accept in the long run.

The solution is not to push the limit, but to introduce a third option between "local but worse" and "cloud but risky": redaction before handing off to the cloud. Names, customer numbers, and other identifiers are replaced with placeholders before the cloud call and restored after the response. The local model only handles anonymization - a task for which even a small model is sufficient - while the actual, complex task goes to a strong cloud model without plaintext PII ever leaving your infrastructure. This involves more effort than pure routing, but far less than running your own frontier model.

## What this means for an organization

The principle behind this is the same as with [network segmentation for my IoT devices]({{< ref "network-segregation-iot-not" >}}): you don't blindly trust the entire system, but define boundaries that certain data simply cannot cross. And this is independent of whether an application later behaves incorrectly. For IoT devices, these are VLANs and firewall rules. For LLM requests, it's a router in front of the inference API.

If you scale this from a five-person team to a larger organization, this technical decision quickly becomes a governance question: who maintains the classification rules when data categories change? How do you prove to customers or regulators that a specific prompt never left your own infrastructure? It's this audit trail that turns "we use a local model" into a verifiable statement rather than an assertion. What starts as a homelab solution with a single GPU becomes a solution you can explain to an auditor.

[^1]: Related approaches that go deeper in the same direction: "Multi-Agent Firewall Architecture for Privacy Protection" and "PrivScope: Task-scoped Disclosure Control for Hybrid Agentic Systems" - both describe how to control the disclosure of data to cloud models per sub-task rather than per request.

[^2]: Personal Identifiable Information - the machine detection of personally identifiable and personally related data in texts.