LLM SERVING MASTERY

Lecture 1 · Foundations

Modern LLM
inference systems

Why one correct generation becomes a queueing, lifecycle, and reliability problem in production.

09:00 · one request was perfect

Serega at one GPU stove while a production queue and request tickets flood the kitchen

At 08:59 the demo passed. At 09:00 sixty-four clients arrived—and the system began completing work nobody still wanted.

One request → one production system

Six decisions

01Draw the boundaryrequest path and ownership 02Account for timeTTFT, decode, E2E, goodput 03Specify loadarrivals, lengths, saturation 04Control failure loopsqueues, deadlines, retries 05Choose an architecturestate, routing, lifecycle 06Measure, then decideexperiment contract and evidence

By the end, you can diagnose—not guess

Observable outcomes

01

Trace a request from arrival to final byte and name every waiting boundary.

02

Compute TTFT, decode time, goodput, concurrency, and retry amplification from raw events.

03

Separate demand, capacity, useful completions, attempts, and failures.

04

Write a reproducible workload/SLO/measurement contract before tuning an engine.

The 64-client outcome

Measured T4 experiment

192 attempts per path. Serialized pipeline: 38 completions and 154 timeouts. vLLM: 192 completions and zero timeouts. Same GPU and pinned model; three simultaneous waves. — Measured T4 experiment. Count every attempt before explaining latency. These are three burst waves, not a sustained-capacity measurement. Source / method.

Serving is a promise with a workload attached

Working definition

SYSTEM CONTRACT

A serving system accepts demand, schedules stateful generation, streams results, and sheds work so that declared quality, latency, throughput, cost, and availability objectives hold for a declared workload.

Remove the workload and the promise is unfalsifiable. Remove failure policy and the promise fails exactly when it matters.

Four questions before any optimization

The diagnostic frame

Question	Evidence	Typical mistake
Where?	trace and component boundary	blame “the model”
When?	queue/prefill/decode timestamps	report one latency
Under what load?	arrival process + token distributions	say “concurrency 32”
What happens on failure?	deadline, cancel, retry, admission policy	let queues grow

“

The model call is a stage. The product is the controlled flow around it.

Lecture thesis

01

Draw the system boundary

If you cannot point to the queue, you cannot explain the latency.

Inference is computation; serving is coordination

Do not collapse the layers

Inference

tokenize inputs
run model forward passes
sample next tokens
stop on a termination rule

Serving

authenticate, admit, queue, schedule
stream, cancel, retry, rate-limit
load, warm, drain, roll back
measure promises by tenant/workload

Where did 154 requests time out?

T4 case study · step with →

Client

Edge / gateway

Admission

Scheduler

Model worker

Telemetry

Demand

A measured T4 wave sent 64 simultaneous requests, repeated three times: 192 attempts per server. Follow one wave across the boundaries.

Edge

Both endpoints streamed chat completions. The client waited at most 60 seconds; a working HTTP route did not prove the service could absorb the wave.

Admission

The serialized baseline offered no effective overload gate: 154 of 192 requests timed out at 64 clients. An early, explicit rejection would be more honest than a silent wait.

Scheduler

The baseline serialized generation behind one lock. vLLM completed all 192 requests in the same 64-client, three-repeat workload.

Worker

Both paths served pinned Qwen2.5-1.5B-Instruct in FP16 on the same T4. The model and GPU alone cannot explain the difference.

Evidence

At 64 clients, p95 TTFT among successful requests was 53.251 s for the baseline versus 0.523 s for vLLM. Failures are counted separately.

Measured teaching demo · method and raw records

Four requests from one wave

Selected measured T4 requests

Serialized repeat 0 at 64 clients. Request 0: first content 0.225 s, complete 4.242 s. Request 1: first content 4.406 s, complete 13.140 s. Request 5: first content 13.394 s, complete 24.662 s. Request 2: timeout near 60.002 s; TTFT not retained on failure. — Selected measured T4 requests. Dot = first content; solid segment = first content to completion. Dashed = client interval with unresolved internal stages. Failed TTFT was discarded by the historical client. Source / method.

Scheduling at token-step boundaries

Illustrative schedule

Three requests across four scheduling steps. Request A advances twice then finishes; B joins and continues; C joins when A leaves. Request-level locking would keep B and C waiting for all of A. — Illustrative schedule. A slot represents a scheduling step, not equal elapsed time. Prefill eligibility and real engine limits still apply. Source / method.

Shared model, separate request state

Logical state map

One shared weights/tokenizer/runtime block feeds three requests. Each request has its own token history, KV state and stopping status. Finishing A releases A state, not the shared weights or B and C state. — Logical state map. Request completion releases its live state. A closed socket alone does not prove the worker released that state. Source / method.

Separate the three planes

Reason about change

Plane	Carries	Failure example
Data	prompts, tokens, streams	backpressure lost
Control	routing, rollout, scale, config	traffic sent before ready
Observability	metrics, traces, logs, profiles	wrong denominator hides failure

Same model, different operating problem

Online vs offline

Online

arrivals are externally timed
per-request deadlines matter
streaming and cancellation matter
capacity headroom buys reliability

Offline batch

backlog is controlled
completion horizon dominates
reordering is often acceptable
high utilization can be rational

Common misconception

“We put FastAPI around it”

What people think

“The endpoint is concurrent, therefore the model service is concurrent.”

Async HTTP can accept many sockets while all requests wait behind one serialized model path. Concurrency at the gateway says nothing about scheduling inside the engine.

Generation and delivery clocks

Illustrative event order

The worker emits token events before the server groups content into transport chunks. The client first sees chunk 1, then chunk 2. Chunk boundaries need not match token boundaries. — Illustrative event order. Record the first nonempty text chunk honestly when individual token events are unavailable.

Cancellation and release acknowledgement

Illustrative lifecycle

Client disconnect detected at 120 ms, gateway forwards cancellation at 125 ms, scheduler removes the sequence at 140 ms, worker releases request state at 145 ms. These constructed times imply a 25 ms release lag. — Illustrative lifecycle. Cancellation travels toward the worker; release acknowledgement can travel back. Already launched kernels need not stop instantly.

Every state needs an owner and a release event

Lifecycle table

State	Owner	Release event
socket / stream	gateway	final chunk or disconnect
queue entry	admission/scheduler	dispatch, expiry, cancellation
KV blocks	engine	finish, abort, eviction
trace	telemetry pipeline	export or sampled drop

Traffic begins after readiness

Illustrative deployment states

Load, warm, ready and drain are distinct states. The process may be alive throughout. Admission is off during load and warm, on when ready, and off again during drain. — Illustrative deployment states. Readiness permits new traffic. Draining stops new admissions while accepted requests finish or cancel under a deadline. Source / method.

Boundary checklist: can you point to each one?

Stop and audit

Boundary	Required event	Required policy
edge → admission	request received	quota/deadline check
admission → scheduler	accepted/enqueued	bounded capacity
scheduler → worker	first scheduled step	fairness/preemption
worker → stream	first/final token	backpressure/cancel

02

Account for time

One latency number erases the mechanism.

Put timestamps on the request clock

Five observable events

t0

ARRIVE

gateway receives bytes

t1

QUEUE

admitted

t2

RUN

prefill begins

t3

TOKEN

first token emitted

t4

DONE

final byte sent

Where did the first 323 ms go?

Interactive sequence · budget 300 ms

Client

Gateway

Queue

Worker

request + auth+8 ms

wait admitted+220 ms

prefill+90 ms

first chunk+5 ms

visible tokenTTFT

TTFT budget

edge8 ms

queue220 ms

prefill90 ms

transfer5 ms

total0

TTFT: account for every wait before first content

Client clock · illustrative serial decomposition

$$T_{\mathrm{TTFT}}=T_e+T_q+T_p+T_f$$

T_e + T_fmsEdge/auth and first visible content transfer

T_qmsAdmission-to-prefill queue residence

T_pmsPrompt processing + first-token sampling

8 + 220 + 90 + 5 = 323 ms TTFT; the 300 ms target is missed before streaming begins.

Use non-overlapping trace intervals. Real stages can overlap; client TTFT alone cannot separate queue, compute and buffering.

Prefill consumes the known prompt

Illustrative token sequence

Four known prompt token IDs enter one prompt-processing phase. That phase produces KV state for the prompt and the distribution from which the first output token is selected. — Illustrative token sequence. Positions can be processed in parallel subject to causal attention. Four boxes represent token IDs, not four words.

One output chain advances sequentially

Illustrative token sequence

Output y1 is fed back to compute y2; y2 is fed back to compute y3. The retained KV state grows at each step. Different requests may advance in the same iteration. — Illustrative token sequence. For n output tokens there are n−1 gaps after the first output. The first output comes from prompt processing.

Count the gaps after the first token

Illustrative trace · final content, not terminal acknowledgement

$$T_{\mathrm{last}}=T_{\mathrm{TTFT}}+(n-1)\,\overline{\Delta t}$$

T_TTFTmsRequest send → first visible token

ntokensOutput count; n = 128 gives 127 gaps

Δt̄ms / gapMean inter-token interval for this request

323 ms + 127 × 22 ms = 3,117 ms = 3.117 s to the last token.

A final protocol acknowledgement can arrive later: $\mathrm{E2E}=T_{\mathrm{last}}+T_{\mathrm{terminal}}$. The terminal delay follows the last content. Stream chunks need not be single tokens.

Name the metric and its interval

No overloaded “latency”

Metric	Clock interval	User question
TTFT	request → first visible token	When does it start?
ITL	token i → token i+1	Does it stutter?
TPOT	post-first-token duration / gaps	What is average cadence?
E2E	request → final byte	When is it complete?

Common metric trap

State the denominator

Ambiguous report

“TPOT = 22 ms.”

Say whether you divide decode duration by output tokens or by post-first-token gaps; say whether you average per request or pool tokens. Different tools use different conventions.

A percentile needs a population

p99 of what?

Valid statement

p99 TTFT among all admitted chat requests in region A during 10:00–10:15, with failures reported separately.

Invalid shortcut

“Latency p99 = 1.2 s” with no phase, route, tenant, window, sampling rule, or failure treatment.

Goodput counts answers that fulfil the promise

One observation window · illustrative counts

$$G=\frac{N_{\mathrm{useful}}}{\Delta t}$$

Guseful req/sRate of correct, successful, on-time answers

N_usefulrequestsUnique originals that satisfy all declared SLOs

ΔtsWall-clock observation window, including waits

120 complete; 110 correct; 90 also on time in 60 s → G = 90 / 60 = 1.50 req/s.

Raw throughput is 120 / 60 = 2.00 req/s. Fix window-edge treatment and count retries separately; this is request-goodput, not token-goodput.

Useful answers in one minute

Illustrative nested populations

120 completions, of which 110 are correct, of which 90 also meet all SLOs. In 60 seconds: 2.00, 1.83 and 1.50 requests per second respectively. — Illustrative nested populations. Use the same window and original-request population. Retries are attempts, not extra users.

Requests/s and tokens/s answer different questions

Keep both

Request throughput

Sensitive to output-length mix. Useful for product capacity when request classes are fixed.

Token throughput

Exposes generation work, but can rise while user-visible goodput falls.

One symptom, three possible causes

Constructed 2-second traces

Three hypothetical first-content delays of 2 seconds: queue-heavy, prefill-heavy, and proxy-buffer-heavy. The same client observation does not locate the wait. — Constructed 2-second traces. The component durations are invented for diagnosis. Measure owner spans before choosing the intervention.

Trace durations; do not subtract unrelated clocks

Measurement hygiene

Safe

monotonic clock within one process
span duration recorded by owner
synchronized clocks with known error

Unsafe

client wall clock minus server wall clock
mix milliseconds and seconds
derive queue time from rounded logs

A real request has more than one clock

Topic 01 · RTX 5070 Ti instructor run

Observation	Measured value	Valid inference
Model load	4.51 s	exclude from warm request latency
43 input → 63 output tokens	1.395 s non-streaming	one local request, no server queue
First nonempty text chunk	0.036 s	not token-level TTFT
Complete streamed response	1.293 s	not comparable as a p95

Pinned Qwen2.5-0.5B FP16 · raw teaching evidence · one request, not a capacity test.

03

Specify the load

A benchmark without an arrival process is a demo.

Who controls the next arrival?

Illustrative traffic clocks

An externally scheduled source sends at 0, 1, 2, 3 and 4 seconds regardless of responses. A single closed-loop user sends at 0 and again at 3 seconds only after the first response completes. — Illustrative traffic clocks. Closed loop is a valid user model. It cannot stand in for an independent arrival schedule when overload is the question.

A closed-loop client slows its own offered load

Fixed population · one outstanding request per user

$$X=\frac{N}{R+Z}$$

X · Nreq/s · usersCompletion rate; fixed client population

RsMean full response time, including waiting

ZsMean think time before the next request

N = 32, R = 0.5 s, Z = 0.5 s → 32 req/s. If R rises to 1.5 s, X falls to 16 req/s.

Steady cycle averages; one request per cycle, no parallel requests per user. A valid user model, but not an independent open-loop arrival schedule.

“Concurrency 32” is not a workload

Missing dimensions

The benchmark claim

“We tested at concurrency 32.”

Arrival model, prompt/output distributions, sampling parameters, streaming, deadline, cancellation, retry policy, warm/cold state, and duration all change the result.

Write the workload as a vector

Minimum specification

Dimension	Example	Why it matters
arrivals	Poisson, 8 req/s	burst and queue formation
input	p50 512, p95 4096 tokens	prefill work/state
output	empirical distribution, cap 256	decode residence
policy	deadline 5 s, no retry	useful demand

Little’s Law: rate × residence = in-flight requests

One boundary · the same request population · long-run means

$$L=\lambda W$$

LrequestsMean queued + running + streaming requests inside the boundary

λrequests / sEffective flow through that boundary; not rejected offered traffic

WsMean entry-to-exit time of a request inside the same boundary

8 req/s × 0.5 s = 4 requests in flight on average. At the same rate, 2 s residence implies 16 requests.

Stable averages and flow conservation; include all exits consistently. Not λ × p95; not a GPU capacity prediction. Little, 1961.

Rate, residence and average population

Stable illustrative averages

At eight requests per second and half a second average residence, average in-flight count is four. At the same rate and two seconds residence, it is sixteen. More resident work does not imply more throughput. — Stable illustrative averages. L = λW uses seconds for W and the same stable flow boundary. It does not determine a peak queue limit.

Little's Law is an identity—not a capacity model

What it omits

It can tell you

whether means are mutually consistent
how residence inflates in-flight population
which population definition you used

It cannot tell you

the service-time distribution
the saturation knee
GPU batching efficiency
tail latency or transient behavior

Why waiting explodes near saturation

M/M/1 teaching model · not a continuously batched GPU

$$W=\frac{1}{\mu-\lambda},\qquad \rho=\frac{\lambda}{\mu}<1$$

λ · μrequests / sArrival rate; service capacity of one server

WsMean time in queue + service for one request

ρdimensionlessUtilization; λ must be smaller than μ

μ = 10 req/s: λ = 8 → W = 0.5 s; λ = 9.5 → W = 2 s. Service mean stays 0.1 s.

Poisson arrivals; independent exponential service; one server; infinite buffer; steady state. Batching, KV limits and GPU kernels are omitted.

Waiting near saturation

M/M/1 analytical model

With service rate ten requests per second, mean response W = 1/(10−arrival rate). At rates 5, 8, 9 and 9.5, mean response is 0.2, 0.5, 1 and 2 seconds. Mean service stays 0.1 seconds. — M/M/1 analytical model. Poisson arrivals, exponential service, one server, unlimited queue, stable rate below 10/s. This curve is not a GPU measurement.

The last acceptable load point

Illustrative capacity sweep

At loads 2, 4, 6, 8, 10 requests per second, useful completions are 2, 4, 6, 6.5, 5.5 and p95 latency is 0.2, 0.25, 0.4, 0.9, 2 seconds. With p95 target 0.8 seconds, six requests per second is the last tested passing point, not proven exact capacity. — Illustrative capacity sweep. Constructed points, not the judge reference. A finer sweep is needed between the last passing and first failing points.

Service time and response time are different clocks

Queueing decomposition

Service time

Time the system actively needs to progress the request, under a precisely defined scheduling model.

Response time

Queue residence + service + transfers + buffering as observed at the chosen boundary.

High GPU utilization is not the objective

Resource metric ≠ product outcome

Dashboard comfort

“The GPU is at 99%, so the system is efficient.”

The GPU can be busy generating late, cancelled, retried, or low-priority tokens. Pair utilization with goodput, queue age, cancellation lag, and SLO attainment.

Scheduled requests disappear from a blocking test

Illustrative 4-second stall

A target schedule has requests at 0, 1, 2, 3, 4 seconds. A blocking client sends only at 0 and 4 while the service stalls. Requests intended at 1, 2, 3 are absent from its histogram, not fast responses. — Illustrative 4-second stall. Retain scheduled-send, actual-send and completion times. A histogram correction cannot reconstruct real queueing, rejection or cancellation outcomes. Source / method.

Aggregate percentiles hide workload shifts

Stratify before comparing

Slice	Primary stress	Compare
short in / short out	fixed overhead + TTFT	admission and batching delay
long in / short out	prefill	input-token bins
short in / long out	decode residence	ITL and cancellation
long in / long out	state + fairness	reject/OOM rates

Count attempts, not just original user requests

Retry amplification · illustrative stable rates

$$\lambda_a=\lambda_o\,\mathbb{E}[A]$$

λₐ · λₒattempts/s · originals/sServer attempt rate; original user demand

Aattempts / originalInitial attempt plus retries and hedged copies

E[A]mean multiplierAverage over the same original population

100 originals/s; 20% get one retry → E[A] = 1.2 → 120 attempts/s.

One retry maximum in this example. Retries bunch in time and add token work; a mean multiplier does not predict burst shape or cancellation.

Load checklist: enough to reproduce the pressure

Before plotting results

Declare	Do not substitute
offered and admitted rate	concurrency alone
token-length distributions	one mean prompt
deadline/cancel/retry	client defaults
warm state + duration	“after warmup”

04

Control the failure loops

Overload is a feedback system, not a traffic spike.

The system attacks itself

Serega closes an admission valve while duplicate request tickets loop back and overflow the serving queue

A timeout is not removed work. If the original keeps running and the client retries, one unhappy user becomes multiple attempts.

Original demand and retry traffic

Illustrative one-retry budget

100 originals produce 100 attempts. Twenty timed-out originals retry once, adding twenty attempts. Four of those retries also time out but the retry budget prevents another attempt. Total is 120, not 124. — Illustrative one-retry budget. The first request may still be running. Cancellation and idempotency semantics decide whether attempts duplicate useful work.

A retry is a budgeted policy decision

Not an exception handler

Retry only when

the operation is safe/idempotent
the remaining deadline is useful
the failure is plausibly transient
a shared retry budget permits it

Shape the retry

bounded attempts
exponential backoff + jitter
server hints where available
deduplication or idempotency key

How many of 64 can meet the deadline?

Interactive fluid burst model · not GPU evidence

Admission protects a bounded worker

Illustrative slot accounting

Eight service slots and sixteen waiting slots admit at most twenty-four of sixty-four simultaneous requests. Forty are rejected early. The model assumes equal jobs and FIFO waves, not GPU batching. — Illustrative slot accounting. Compare this fixed picture with the interactive burst model: increasing the queue can turn early rejects into late accepted failures.

A five-second deadline budget

Constructed admission decision

Transit and validation consume 0.2 seconds, predicted queue 3.9 seconds, predicted service 1.4 seconds: total 5.5 seconds versus a 5 second deadline. The optimistic finish misses by half a second. — Constructed admission decision. Reject or defer under this estimate. Real policies need prediction error margins and fairness, not only the component sum.

Twenty fast branches can still make a slow answer

All branches required · independent identical success probabilities

$$P(\mathrm{all\ on\ time})=p^k$$

pprobability [0, 1]One branch meets the same deadline

kbranchesNumber of required parallel subrequests

pᵏprobability [0, 1]Every required branch meets its deadline

p = 0.99 and k = 20 → 0.99²⁰ = 0.818: about 18.2% of parent requests miss it.

Independence is essential, not a production default. Shared queues/failures correlate branches; quorum and partial-result policies change the event.

The slowest required branch gates completion

Illustrative parallel dependency

Three required parallel branches take 100, 120 and 800 milliseconds. Joining all branches takes at least 800 milliseconds, not the mean 340 milliseconds. Shared failures invalidate independence assumptions. — Illustrative parallel dependency. Parallel critical path uses a maximum. Sequential stages add. The p^k probability example additionally assumes independent branches. Source / method.

Cached artifacts still need loading and warmup

Illustrative scale-out critical path

Illustrative scale-out critical path. A serial budget adds to 95 seconds. If stages overlap, measure the actual critical path instead.

A rollout is a state machine

No traffic before evidence

1

LOAD

revision + tokenizer

2

WARM

representative probe

3

READY

admit canary traffic

4

DRAIN

stop new work

5

STOP

release state

Shutdown order changes the failure wave

Illustrative release protocol

Illustrative release protocol. Drain is bounded; a permanently stuck request must not hold the deployment forever. Record finish versus partial response versus cancellation. Source / method.

Classify failure by the promise it breaks

Detection → action

Class	Signal	First action
overload	queue age + rejects	shed / reduce admission
resource exhaustion	allocation failures	protect state, narrow workload
dependency failure	downstream spans	budgeted fallback
bad revision	canary quality/SLO delta	stop rollout / rollback

“

Reject early enough that accepted work can still succeed.

Overload-control principle

05

Choose an architecture

The correct topology follows workload and failure semantics.

Three deployment shapes

Logical architecture comparison

Offline jobs read a backlog into a worker. A single interactive engine receives a client request directly. A gateway with a pool selects among replicas and must manage routing, health and state locality. — Logical architecture comparison. Boxes are logical responsibilities, not a requirement for separate processes. Add a boundary only when its isolation or scaling benefit is needed.

Start with one engine when one engine fits the promise

Complexity is a cost

Good fit

one model and policy
predictable load
failure domain acceptable
vertical capacity sufficient

Pressure to split

tenant/model routing
independent rollouts
horizontal capacity
different workload classes

Equal request counts can hide unequal work

Illustrative routing counterexample

Each worker has two requests. Worker A has two 128-token prompts, total 256 input tokens. Worker B has two 4096-token prompts, total 8192 input tokens. Counts match, token work differs by thirty-two times; runtime need not differ by that factor. — Illustrative routing counterexample. Token counts are a workload signal, not a linear execution-time predictor. Include outputs, queue age and cache state in the decision.

Cache savings can lose to queueing

Constructed routing decision

Cached worker A needs 300 ms queue plus 20 ms prefill, total 320 ms. Uncached worker B needs 20 ms queue plus 120 ms prefill, total 140 ms. Other costs are assumed equal. Cache affinity alone chooses the slower path. — Constructed routing decision. Illustrative estimates for the same request. Compare end-to-end predicted cost and uncertainty, not cache hit rate alone.

Admission policy changes who waits

Illustrative tenant queue

A shared FIFO places tenant B behind three tenant A requests. A per-tenant policy can select B earlier but must define fair shares and starvation protection. Queue slots do not encode equal token cost. — Illustrative tenant queue. The diagram shows order, not equal service time. Compare per-tenant deadline attainment before calling a policy fair.

Separating phases adds a state-transfer boundary

Conceptual architecture

A router sends prompt work to a prefill worker. The produced KV state crosses a transfer boundary to a decode worker, which streams output. Separate queues and transfer/recovery cost accompany independent scaling. — Conceptual architecture. No speedup is assumed. Specialization must outweigh state transfer, additional queueing and recovery costs. Source / method.

Scale on demand pressure, not only device activity

Signals have delay

Signal	Advantage	Trap
GPU utilization	available	late; includes waste
queue depth	direct pressure	ignores request size
queue age	deadline proximity	already reacting
predicted token work	closer to demand	prediction error

New capacity arrives after several separate delays

Illustrative serial scale-out critical path

$$T_{\mathrm{ready}}=T_d+T_v+T_l+T_w$$

T_d · T_vsDetect demand; provision and start the worker

T_l · T_wsLoad cached weights; warm up kernels and probes

T_readysTime until additional capacity is usable

15 s detect + 30 s provision + 40 s load + 10 s warmup = 95 s; a 20 s burst ends first.

Sum only serial stages; overlapping work follows the critical path. Persistent caches avoid downloads, not weight loading. Protect the gap with admission/headroom.

Architecture starts from product behavior

Four live archetypes

Product	Dominant promise	Architecture pressure
chat assistant	TTFT + smooth stream	burst admission, cancel
batch summarizer	cost / completion horizon	backlog scheduling
coding copilot	very short deadline	aggressive stale-work cancel
fan-out agent	end-to-end success tail	deadline propagation

There is no context-free “best serving architecture”

Optimization needs a contract

Architecture cargo cult

“Production systems use microservices, disaggregation, and autoscaling.”

Each boundary adds independent scaling and failure isolation—and also serialization, networking, state transfer, tail amplification, and operational work.

Choose the smallest topology that satisfies the contract

Decision record

Record	Question
workload	which demand and length distributions?
objective	which SLO/cost/quality constraint dominates?
state	where does per-request/model state live?
failure	what degrades, retries, drains, or rolls back?

06

Measure before optimizing

If the experiment cannot be repeated, the number cannot be trusted.

Freeze six things before a run

Experiment contract

Layer	Record exactly
model	artifact revision, tokenizer, precision, context limit
runtime	engine/container/driver versions and launch command
hardware	device, count, memory, topology, power mode
workload	arrivals, token distributions, sampling, seed, deadline
measurement	boundaries, warmup, duration, percentiles, failure rules
artifact	raw events, summary, commit, timestamp

The workload begins with an exact request

Concrete payload

Request

JSONPOST /v1/chat/completions

{
  "model": "served-model",
  "messages": [{"role":"user",
    "content":"Explain backpressure."}],
  "temperature": 0,
  "max_tokens": 128,
  "stream": true
}

Record beside it

rendered prompt token count
actual output tokens and stop reason
client deadline and disconnect
request and trace identifiers

A trace becomes a metric report

Constructed single-request record

Constructed single-request record. A worked example, not measured internal spans. The recorded terminal acknowledgement is later than the last content token.

A fast success percentile can hide failure

Constructed 100-attempt comparison

Service A completes twenty requests in 0.2 seconds and eighty time out. Service B completes all one hundred in one second. Successful-only p95 favors A, but deadline attainment favors B. — Constructed 100-attempt comparison. Report failure counts beside latency. Do not insert artificial completed 60-second responses for timeouts.

Six ways to produce a persuasive wrong answer

Audit before comparing

Trap	Bias	Repair
closed-loop only	hides overload	add scheduled open-loop sweep
drop failures	improves tails	report failure/reject rates
warm cache only	hides lifecycle cost	label warm and cold separately
different lengths	changes work	replay identical workload
client bottleneck	caps offered load	measure generator headroom
one short run	captures luck	repeat; retain raw events

A bounded experiment has a stopping rule

Illustrative measurement workflow

Freeze the manifest, warm the service, measure one bounded load point, check guardrails, and either stop or increase load. Always retain raw records and verify resources return to baseline. — Illustrative measurement workflow. Stop on resource growth, errors or missed safety limits. A successful one-request lab does not authorize a broad unattended load sweep.

Deliverable: a pinned single-request baseline

Topic 01 · no capacity claim

Artifact	Acceptance
baseline JSON	model revision, runtime, GPU, token counts, warm and cold intervals
boundary trace	owners, observable timestamps and cancellation path
reflection	explain why chunks are not tokens and why one request proves no capacity envelope
submission repo	private copy of the public example, judge access confirmed, contract checks + GPU smoke, then round-01 tag

The payoff

Serega calmly supervises a bounded request line, coordinated GPU workers, and stable observable flow

Production readiness is not infinite capacity. It is a declared envelope, controlled admission, cancellable work, and evidence that accepted requests meet the promise.

Primary reading and specifications

Continue in self-study

Dean, J.; Barroso, L. A. “The Tail at Scale.” Communications of the ACM, 2013.
HdrHistogram documentation: coordinated-omission correction and interval recording.
IETF RFC 9110, HTTP Semantics; WHATWG Server-Sent Events.
OpenTelemetry GenAI semantic conventions (development status): spans, metrics, events.
Kwon et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP 2023.
Kubernetes probes and Pod lifecycle.

Can you make the promise falsifiable?

Write one workload sentence, one SLO sentence, and the exact events needed to prove or disprove it. Next: why prefill and decode stress hardware differently.