Compute Resource Management

Strategies for cost-efficient operation of GPU/NPU servers and cloud infrastructure

On-Premises vs. Cloud

    flowchart TD
    A["AI workload"] --> B{"Workload characteristics"}
    B -->|"Large-scale training<br/>Irregular load"| C["Cloud<br/>AWS / GCP / Azure"]
    B -->|"Continuous inference<br/>Data security critical"| D["On-premises<br/>GPU servers"]
    B -->|"Mixed strategy"| E["Hybrid<br/>Cloud bursting"]

    style A fill:#2563EB,stroke:#1D4ED8,color:#fff
    style C fill:#16A34A,stroke:#15803D,color:#fff
    style D fill:#EA580C,stroke:#C2410C,color:#fff
    style E fill:#7C3AED,stroke:#6D28D9,color:#fff
  

Cloud AI Service Comparison

Cost Optimization Strategies

1. Model Tiering

Complex tasks   → Large models (Claude Opus, GPT-4o)
General tasks   → Mid-size models (Claude Sonnet, GPT-4o-mini)
Simple tasks    → Small models (Claude Haiku, GPT-3.5)

2. Caching Strategy

For prompts or context that are reused repeatedly, prompt caching can cut costs by up to 90%.

3. Batch Processing

For workloads that don’t require a real-time response, using the Batch API cuts costs by 50%.

GPU Spec Guide

Managed Platforms vs. Self-Hosting

Before sizing a GPU, decide whether you should be holding one at all. There are three tiers, and they differ in what you own rather than in what the model can do:

The two AWS entries in the comparison above sit in different tiers, and conflating them is a common planning mistake:

• Amazon Bedrock is the managed model API tier — a single interface over foundation models from several providers, with no infrastructure to run. Its adjacent features are the ones you would otherwise assemble yourself: managed knowledge bases for RAG, agent orchestration, and Bedrock Guardrails for input and output filtering. Google Vertex AI and Azure AI occupy the same tier.
• Amazon SageMaker is the managed ML platform tier — an end-to-end environment for building, training, and deploying models you own, including classical ML on tabular data, not only foundation models. You choose the instance types and the endpoint stays up until you take it down.

A useful default: start on the managed API tier, because it turns a capacity problem into a line item. Move to a platform or to self-hosting only when a concrete requirement forces it — data residency, a model no provider hosts, a fine-tune you must own, or a sustained load where per-token pricing has become more expensive than the hardware.

Serving Open Models

Self-hosting a model means running an inference server, and the serving engine decides how much of the GPU you actually get to use. On identical hardware, throughput differs by an order of magnitude between a naive request loop and a batching engine.

The levers that decide throughput:

• Continuous batching — arriving requests join the running batch instead of waiting for it to drain
• KV cache management — paged attention lets many concurrent sequences share VRAM without pre-reserving worst-case space
• Tensor parallelism — split a model across GPUs once the weights exceed one card
• Quantization — buys VRAM headroom, and therefore concurrency, in exchange for some quality; see Model Selection & Tuning

Inside the serving loop — prefill, decode, and the memory wall

The levers above only make sense once you see that inference runs in two phases with completely different bottlenecks:

This asymmetry is why batching is the central optimization. During decode the GPU spends most of its time waiting on memory, so adding more sequences to the in-flight batch costs almost nothing in extra time — the weights were being read anyway. Throughput scales with batch size until VRAM or bandwidth runs out, which is the order-of-magnitude difference between a request loop and a batching engine.

Continuous (in-flight) batching is what makes this usable with real traffic: instead of forming a batch and waiting for every sequence in it to finish, the engine admits an arriving request into the running batch at the next step and evicts finished sequences immediately. Without it, one long generation holds the whole batch hostage and short requests inherit its latency.

KV cache — the dominant consumer of VRAM

The KV cache is what makes decoding tractable: keys and values for earlier tokens are stored instead of recomputed. Without it, every step would have to re-process the entire context to emit one more token, so the cost per token would grow with the text already written and the work to finish a response would grow quadratically with its length. With it, prefill writes the cache once and each decode step appends a single entry. The cost is memory, and it is not a rounding error:

KV bytes per token ≈ 2 (K and V) × layers × kv_heads × head_dim × bytes_per_element
total ≈ (KV bytes per token) × sequence_length × concurrent_sequences

Model weights are a fixed cost you pay once. The KV cache is a per-request cost that grows linearly with context length and with concurrency — so in long-context serving it, not the weights, is what decides how many users fit on the card. Three techniques manage it:

Admission control — refusing work on purpose

Because decode is memory-bound and VRAM is finite, an inference server that accepts everything handed to it does not degrade gracefully: it either hits OOM or collapses into a queue where every request misses its latency target. Admission control is the entry gate that prevents this, and it is the piece most teams discover they need only after an incident.

The mechanism is to estimate a request’s memory footprint before admitting it — prompt tokens plus the maximum new tokens it may generate, converted into KV blocks — and compare that against what is actually free:

Two rules follow. An unbounded queue is a latency bomb, not a safety net — it converts a capacity problem into a timeout problem further downstream. And the decision belongs as close to the edge as possible: shedding load at the gateway is cheaper than shedding it after the prefill has already consumed GPU time.

A deployment with zero rejections and a rising TTFT is not a healthy one — it is one that chose the timeout. The metrics that make this visible are in the table below.

Where inference runs — separate server or in-process

Serving an open model does not have to mean a separate service. Two topologies are in use, and the trade is infrastructure complexity against control over batching:

The embedded pattern has become practical outside Python. Kronk, from Ardan Labs, is a Go SDK that integrates llama.cpp, whisper.cpp, and stable-diffusion.cpp directly into a Go application for hardware-accelerated local inference, with an OpenAI-shaped API, and it also ships a model server for headless use. Its maintainers describe the SDK — rather than the bundled server — as the point, and a path toward one-binary deployment as the direction of travel; check the repository for where that currently stands.

The trade-off is worth being explicit about, because it is the same trade in both directions: once inference lives inside your process, continuous batching and admission control become your application’s problem, not an engine’s. For a single-user desktop tool that is a non-issue and the removed infrastructure is pure gain. For a shared multi-tenant service, those are exactly the mechanisms you would be re-implementing, and a serving engine already has them.

What to hold a serving deployment to: