TL;DR
The Pragmatic Engineer has published a report on inference engineering: the practices software teams need when serving open-weight AI models through third-party providers or their own GPUs. It describes a path from API substitution to dedicated hardware and identifies cost, latency, rate limits and workload performance as key decision points.
The Pragmatic Engineer has published a report on inference engineering, the work involved in serving open-weight AI models reliably and at manageable cost. Drawing on interviews with Modal, Fireworks AI and Baseten, it outlines how software teams can move from frontier-lab APIs to third-party inference services or dedicated GPUs, while weighing latency, rate limits, caching and operational effort.
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The report describes two main ways to run open models. Companies can operate their own GPU infrastructure, which demands substantial setup and operational work, or use an inference provider that runs models on a customer’s behalf. These providers expose APIs resembling those from AI labs, but typically host open-weight models rather than training their own frontier models.
The article says an initial move can be as simple as replacing a frontier lab’s API endpoint with a provider’s API, an example of software engineering after code becomes cheap. That approach can speed adoption, but teams may encounter rate limits, latency variation and cache-hit issues as usage grows. The report identifies inconsistent performance, restricted capacity and costs that no longer make sense as possible signs that a team should investigate dedicated GPU deployments.
Dedicated deployments offer more control but require decisions about model size, throughput, interactive response times, KV-cache capacity and failover. The report also discusses speculative decoding as a possible way to improve performance when compute is available during token generation. It does not present that technique as a universal solution; the relevant setup depends on a workload’s characteristics.
When API Swaps Reach Their Limits
Inference choices can affect both the cost and reliability of AI features used by customers. The report points to examples previously covered by The Pragmatic Engineer in which companies moving to open models reported lower per-token costs: Uber by 50%, Pinterest by 90% and AT&T by 56%. These are figures reported by the publication, not independently verified comparisons in the material provided; the precise measurement periods and baselines are not specified here.
For software teams, the practical point is that changing models or vendors is not only an API decision. A service that works at low volume can behave differently under demand, and dedicated hardware can require a new level of capacity planning and operations. Teams need workload-specific evaluations before deciding whether lower unit prices offset engineering effort, latency trade-offs and infrastructure commitments.
The shift also expands the skills expected of engineers working on AI products. Familiarity with model evaluation, inference performance and deployment trade-offs can help teams decide when a managed shared service is sufficient and when greater control may justify dedicated resources.
Products Worth Considering

Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation

Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, White Smoke

Amazon Kindle Scribe Colorsoft 64GB with Premium Pen
As an affiliate, we earn on qualifying purchases.
From Frontier APIs to Open Models
Frontier AI labs train their own large models and offer them through APIs. The report distinguishes inference providers by their focus: they host and serve models whose weights are available, and compete on pricing, infrastructure efficiency and customization. They may offer post-training services, but the article says they do not carry out the costly frontier-scale pre-training runs themselves.
The report cites Revolut’s 56 NVIDIA B200 units as an example of a company running inference infrastructure itself. That route can reduce dependence on an external serving provider, but it requires staff and resources to configure and operate GPU systems. The alternative is outsourcing much of that work to a specialist provider.
The article’s interviews illustrate different provider approaches. Modal emphasizes programmable infrastructure and open-source tooling; Fireworks AI highlights specialized deployments and tuning; Baseten describes an inference platform with additional tools for agent workloads. The publication says it has no affiliation with or payment from the vendors mentioned and credits Richard Gong, Dmytro Dzhulgakov and Philip Kiely for contributing information.
“When you run code on Modal, you should be able to see all the code that’s running.”
— Richard Gong, inference lead at Modal
Costs and Performance Depend on Workloads
The source material does not establish that open-model hosting will lower costs for every company. The cited savings figures lack detailed baselines and measurement windows here, and a comparison would depend on model quality, usage volume, hardware, staffing and service terms. It is also unclear from the report summary how the named providers’ prices and performance compare under a common test.
Latency and cache behavior vary with workload and deployment configuration, so the report’s warning signs are decision prompts rather than fixed thresholds. It does not provide one model, provider or GPU configuration that suits all teams. The relative benefits of shared multi-tenant APIs and dedicated hardware must be assessed against each product’s traffic and reliability needs.
The source text ends partway through an example about Fireworks’ Ember-1 model, so no further details about that example can be confirmed from the supplied material.
Measure Before Committing GPUs
The report’s recommended next step is for teams to run evaluations on models suited to their own use case, then compare providers and deployment options. A team considering a switch can begin with a multi-tenant API and monitor response times, rate limits, cache behavior and spending as traffic changes.
If those measurements show persistent capacity or consistency problems, engineers can examine dedicated GPUs and plan for model sizing, throughput, caching and failover. The source does not identify a scheduled follow-up, a single recommended vendor or a formal industry standard; the next development for individual teams will depend on their evaluation results and operational requirements.
Key Questions
What does inference engineering mean?
Inference engineering covers the technical work of serving AI models, including model and provider selection, latency, caching, capacity and deployment configuration.
How is an inference provider different from an AI lab?
As described in the report, an inference provider generally hosts open-weight models and sells access to them. Frontier labs train their own foundational models and also provide model APIs.
When might a team move beyond a shared inference API?
The report points to rate limits, inconsistent performance and uneconomic usage costs as reasons to investigate dedicated deployments. Those signals do not by themselves establish that dedicated GPUs will be a better choice.
Does using open models guarantee lower costs?
No. The report cites cost reductions reported for several companies, but those figures do not establish savings for every workload. Teams need to compare their own model quality, usage, provider fees and operating costs.
What should teams evaluate before choosing a deployment?
Teams should test models against their use case and examine latency, throughput, rate limits, cache behavior and total cost. For dedicated GPUs, they also need to plan model size, KV-cache capacity and failover.
Source: newsletter.pragmaticengineer.com
