Definition: Model Serving
Model serving is the infrastructure and software layer that takes a trained AI model and makes it available to run inference requests in production, handling request routing, batching, scaling, and monitoring so applications get fast, reliable responses.
Core characteristics of model serving
Model serving sits between a trained model and the applications that consume its predictions, as a persistent service built for concurrent, ongoing load rather than a script that runs once.
- Request handling: accepts and queues inference requests from multiple applications at once
- Batching and scheduling: groups requests to maximize GPU throughput without breaching latency targets
- Autoscaling: adds or removes compute capacity as demand fluctuates
- Observability: tracks latency, error rates, and token usage per request
Model Serving vs. AI Inference
AI inference is the actual computation, one forward pass through the model producing an output. Model serving is the system around it: which requests run together, on which hardware, and what happens if a node fails. A single inference call can happen on a laptop; model serving turns that call into a service thousands of people rely on continuously.
Importance of model serving in enterprise AI
Serving decisions shape both cost and user experience. Gartner reports that global spending on AI-optimized cloud infrastructure reached 42 billion US dollars in 2026, with 23.3 billion going to inference versus 19 billion to training, the first year inference spend overtook training spend. Serving choices are no longer a niche concern either: Bitkom’s 2026 KI-Studie finds 41 percent of German companies now actively use AI in production, with a further 48 percent planning deployment, meaning most Mittelstand companies will make a serving decision soon.
Methods and procedures for model serving
Enterprises choose between deployment patterns with different cost, control, and compliance trade-offs.
Hosted API vs. self-hosted serving
A hosted API bills per token with no infrastructure to manage. Self-hosted serving runs the model on company-controlled GPUs, trading a fixed cost for full control over data flow. Many enterprises use a hybrid AI deployment that routes routine queries to a self-hosted model and escalates hard cases to a hosted frontier model.
- Hosted APIs suit variable, low-to-moderate volume without dedicated infrastructure staff
- Self-hosting suits sustained, high-volume workloads or strict data residency needs
- Hybrid routing balances cost against capability for mixed workloads
Serving frameworks and continuous batching
Inference engines such as vLLM, TensorRT-LLM, and Triton Inference Server use continuous batching and paged attention to keep GPUs busy across concurrent requests instead of one at a time, and handle quantization so larger models fit smaller hardware.
Orchestration and gateways
Larger deployments route traffic through an AI gateway that enforces authentication, rate limits, and cost controls across served models, while MLOps pipelines automate rolling out model updates without downtime.
Important KPIs for model serving
Serving performance is measured across latency, cost, and reliability.
Operational efficiency metrics
- Time to first token: target under 300 milliseconds for interactive use cases
- Inter-token latency: target under 50 milliseconds during generation
- GPU utilization: target above 70 percent sustained load
- Throughput: tokens processed per second per GPU
Strategic business metrics
Utilization determines unit economics. Self-hosted serving typically breaks even against hosted API pricing only above roughly 70 percent sustained GPU utilization, with payback in 12 to 18 months according to recent German TCO analyses of self-hosted AI inference. KfW Research separately finds 36 percent of Mittelstand companies with 50 or more employees now use AI, up from 6 percent six years earlier, so the serving question is scaling with adoption.
Quality and reliability metrics
Uptime and request success rate matter as much as speed. A serving layer with 99.9 percent availability and stable latency under load earns the trust that lets teams depend on it for customer-facing work, not only internal experiments.
Risk factors and controls for model serving
Serving infrastructure introduces risks distinct from model quality itself.
Underutilized infrastructure
Provisioning GPU capacity for peak demand leaves expensive hardware idle most of the time, eroding any cost advantage over hosted APIs.
- Right-size GPU capacity against measured, not assumed, request volume
- Use autoscaling and spot capacity for bursty workloads
- Track cost per served token, not only total infrastructure spend
Data residency and sovereignty
Sending inference requests to a provider outside the EU can conflict with data protection rules. On-premise AI serving keeps requests inside company or EU infrastructure, often the deciding factor for regulated industries.
Latency and availability failures
A serving layer without redundancy is a single point of failure for every dependent application. Multi-region deployment, health checks, and automatic failover to a backup model are standard once serving becomes business-critical.
Practical example
A 140-employee precision tooling manufacturer in Baden-Württemberg runs camera-based quality inspection across twelve production lines. Each line used to run its own inference script on a local workstation with inconsistent uptime and no shared monitoring, so the company consolidated the models onto a shared serving layer on two GPU servers, with a hosted API as fallback during peak season.
- Centralized dashboard showing latency and error rate per production line
- Automatic failover to a hosted API when local GPU capacity is exceeded
- Shared model updates deployed to all lines simultaneously without downtime
- GPU utilization tracking that guided the decision to add a third server
Current developments and effects
Serving infrastructure is evolving quickly as inference volume grows faster than training volume.
Serverless and shared GPU inference
Providers increasingly bill inference per request instead of per hour of reserved capacity, lowering the entry barrier for mid-sized companies testing self-hosted serving.
- Pay-per-request serverless GPU inference from cloud providers
- Multi-tenant serving that shares GPU capacity across several models
- Spot and reserved capacity blending to reduce idle cost
Small models and edge serving
Smaller, distilled models increasingly run at the edge AI layer, cutting latency and connectivity dependence while escalating only complex queries to larger hosted models.
Consolidation of serving frameworks
Viable open-source serving engines are narrowing to vLLM and a small set of alternatives, simplifying vendor selection for enterprises building their own stack.
Conclusion
Model serving turns a trained model into a dependable production service, and the deployment pattern chosen shapes cost, latency, and compliance as much as the model itself. As inference spending overtakes training spending industry-wide, serving decisions move from a technical afterthought to a genuine cost and risk lever. Enterprises that measure utilization, latency, and cost per token from day one avoid overprovisioned infrastructure or unpredictable API bills. The direction is toward hybrid architectures that route each request to the cheapest infrastructure that still meets its latency and compliance requirements.
Frequently Asked Questions
What is model serving in simple terms?
Model serving is the software layer that runs a trained AI model in production so applications get reliable responses at scale, within a latency budget. It covers request handling, batching, scaling, and monitoring, not the model’s training.
Should a Mittelstand company self-host model serving or use a hosted API?
Most companies under 200 employees with irregular request volume are better served by a hosted API, since self-hosting only pays off above roughly 70 percent sustained GPU utilization. Sustained, high-volume, or data-sensitive workloads are where self-hosted serving becomes cost-effective.
What does model serving cost for a company with 100 to 200 employees?
Hosted API costs scale with token volume and typically start in the low four figures per month for moderate use. Self-hosted serving needs GPU hardware, setup time, and ongoing monitoring, which usually only pays off above a certain sustained request volume.
How does model serving relate to GDPR and the EU AI Act?
Model serving itself is not directly regulated, but where requests are routed determines which data protection rules apply. Serving requests through EU infrastructure, or fully on-premise, simplifies GDPR data transfer compliance and narrows the audit scope under the EU AI Act.
Do we need our own IT team to run model serving?
Not necessarily. Hosted APIs need no dedicated infrastructure team, and self-hosted serving benefits from basic infrastructure expertise, though many mid-sized companies use an external partner for setup and ongoing operation.
How does Superkind approach model serving for its AI agents?
Superkind routes each AI agent request to the model and infrastructure that fits the task, connecting to a company’s own systems rather than locking in a fixed serving stack, so the underlying model can change without disrupting the agents built on top of it.
