Queue ordering by predicted output length on a real vLLM server

Summary

• Question: on a real vLLM 0.30 server, does ordering the waiting queue by a predicted output length (shortest predicted first) raise strict-SLO performance over vLLM's default FCFS, and how close is a decision model (Jev 1.13) to the true length? Does the calibrated simulator predict the real server?
• Setting: Qwen3-8B on one H200, KV cache capped at 84,000 tokens (about 60 thinking-on requests at once), 2,000 ShareGPT requests with Qwen3-8B's logged thinking-on output lengths, Poisson arrivals at 1.05-1.54 requests/s. SLO: TTFT <= 1 s and TPOT <= 50 ms; target: 90% of requests.
• Results (arrival draw 0, same rates for every policy):
  • FCFS falls below 90% attainment just above 1.10 req/s (92.0% at 1.10, 75.4% at 1.30, 41.5% at 1.42).
  • Every reordering keeps attainment high much longer:

    Policy	Attainment at 1.30 req/s	Attainment at 1.54 req/s
    Shortest prompt first	96.5%	88.4%
    LTR ranker	98.1%	95.5%
    Jev	98.6%	96.0%
    True length	99.5%	98.7%

  • From about 1.36-1.42 req/s, every reordering lets the backlog of deferred long requests grow (starvation), before attainment falls below 90%.
• Goodput (highest rate with attainment >= 90% and no growing backlog), arrival draw 0:

  Policy	Goodput (req/s)
  FCFS	1.127
  Shortest prompt first	1.30-1.42
  LTR	1.30-1.36
  Jev	1.30-1.36
  True length	1.36-1.42

  The shortest-first policies are bracketed because the backlog, not attainment, is what fails first. Real gain of Jev over FCFS: +15% to +21% (draw 0) and +16% to +26% (draw 1).

• Where predictors differ: attainment at high load (above), and the TTFT tail.
  • P99 TTFT at 1.30 req/s: shortest prompt first 52.9 s, LTR 22.0 s, Jev 2.7 s, true length 0.4 s.
  • Jev's P90 TTFT is the highest of the reorderings at low load (0.14 s vs 0.05-0.07 s) because its 92 ms decision latency is added to every request.
• Why reordering helps even without a model: vLLM admits the queue's head only if its whole prompt fits in the free KV blocks and stops at the first that does not. Under FCFS one long prompt at the head blocks every request behind it; serving short requests first keeps admission flowing. Output-length prediction adds to this at high load and in the tail.
• Simulator: after calibration, real and simulated attainment differ by at most 2.1 points in all 32 runs of rounds 1-2 (2.6 points over all 47 runs, all policies). The keep-up verdict agrees in 44 of 47. TPOT p50 is within 0-10%. The simulator underestimates the mean and P99 TTFT of the shortest-first policies (the deferred tail) by up to about 2x.

Setup

Policies

All but FCFS use --scheduling-policy priority with per-request priority = round(1000 x log2(predicted output length)), smaller first. Waiting requests never preempt running ones; under memory pressure vLLM preempts the lowest-priority running request.

Prediction quality on the 6,086 thinking-on test prompts (Kendall tau): Jev 0.559, LTR 0.545, prompt length 0.114. A second answer sampled for the same prompt reaches 0.725, the practical ceiling of any predictor that sees only the prompt.

Metrics

• TTFT = time of the first streamed chunk with non-empty content (thinking text counts) - scheduled arrival.
• TPOT = (stream end - first token) / (output tokens - 1).
• SLO attainment = share of the 2,000 requests with TTFT <= 1 s and TPOT <= 50 ms.
• Backlog grows / keeps up: over the middle 60% of the trace by arrival time, the server keeps up if its backlog of tokens still to process grows by at most 3% of the tokens that arrive. A waiting request counts its whole prompt and output; a decoding request counts the share of its output still to come.
• Goodput (per arrival draw) = the highest rate run at which attainment >= 90% and the server keeps up. It is interpolated linearly where attainment crosses 90% between two rates. Where the backlog starts to grow first, it is reported as the bracket [last passing rate, first failing rate).
• Server-side means (TTFT, inter-token latency) come from vLLM's /metrics histograms. Client and server TTFT agree except for the decision latency, which only the client counts (e.g. Jev at 1.10: 0.15 s client vs 0.05 s server).

Runs

Each run takes 25-33 minutes. Rounds 1-3 total 47 completed runs.

Results: every policy at the same rates (arrival draw 0)

SLO attainment (* = the backlog grows)

Mean TTFT (s) (* = the backlog grows)

P90 TTFT (s) (* = the backlog grows)

P99 TTFT (s) (* = the backlog grows)

Mean end-to-end latency (s) (* = the backlog grows)

Results: goodput

The calibrated simulator's goodput averaged over both draws: FCFS 1.188, shortest prompt first 1.394, LTR 1.425, Jev 1.460, true length 1.497 req/s (+17%, +20%, +23%, +26% over FCFS).

Results: real server vs calibrated simulator

• Rounds 1-2 (32 runs): attainment within 2.1 points in every run; keep-up verdict agrees in 29 of 32.
• All 47 runs: attainment within 2.6 points (largest: FCFS at 1.42 req/s, draw 0, 41.6% real vs 44.2% simulated); keep-up verdict agrees in 44 of 47. The three disagreements are Jev and LTR at 1.36 and the true length at 1.42 (all draw 0): the simulator says the server keeps up, the real server's backlog grows.
• TPOT p50 within -1% to +10%; mean end-to-end latency -2% to +23% (median +6%; the real server is slower).
• TTFT: FCFS matches well (e.g. 1.25 req/s draw 0: 1.46 s real, 1.47 s simulated). For the shortest-first policies at high load, the simulator's mean and P99 TTFT are up to about 2x lower: the requests they defer wait longer on the real server.
• Preemptions: same order of magnitude (real -28% to +43% relative to simulated).
• Calibration that made this agreement possible:
  1. Step times measured on the real H200: base 5.08 ms + 16.4 us x seqs + 41.4 us x max(0, seqs - 320) + 31.4 ns x context tokens; prefill 22.6 us x tokens + 1.1e-9 s x tokens x context attended.
  2. One client process per server (round 0's shared client lagged by 10-40 ms median).
  3. vLLM's default memory utilization (0.92).

Interpretation

1. Admission packing: in this memory-bound setting, the largest part of the gain over FCFS comes from not blocking admission behind a request that does not fit. Shortest prompt first, which knows nothing about output length, already moves attainment at 1.42 req/s from 41.5% (FCFS) to 92.5%.
2. Output-length information adds attainment at high load and shortens the TTFT tail. The ordering is clear: true length > Jev ~ LTR > shortest prompt first > FCFS. Jev and LTR are close; Jev has the shorter tail at 1.30-1.42 req/s, LTR the lower P90 at low load because of Jev's decision latency.
3. Starvation binds before the SLO for every shortest-first policy, from about 1.36-1.42 req/s. That is why their goodput is bracketed and nearly tied. Starvation control (aging) would be needed to push goodput further and to separate predictors by goodput.
4. The setting matters: these gains need a small KV pool (tens of requests at once). On default deployments with hundreds of requests in the cache, the simulator finds 0-3% goodput gains even with the true length (the simulated deployment sweep).

Caveats

• One model and GPU type; the small KV pool is imposed with --num-gpu-blocks-override, not a natural deployment.
• One workload (ShareGPT first turns, thinking on) and Poisson arrivals only.
• The same-rate comparison uses one arrival draw; draws differ by up to ~8% in FCFS goodput (1.127 vs 1.219).
• Rates are 6-12% apart, so backlog-limited goodput is only bracketed.
• LTR's 20 ms decision latency is assumed, not measured.
• Prefix caching is off, and TTFT counts the first thinking token.

Every run