I built a tool to benchmark any LLM in one command. Here are the numbers.

After 12 weeks of running individual benchmarks — concurrent latency, quantization, prefill vs decode, speculative decoding, PyTorch profiler, Triton kernels — I wanted one tool that wraps everything into a single command.

python dashboard.py --model llama3.2 --runs 3 --save

This is what came out.

The output

================================================================================
  LLM INFERENCE OPTIMIZATION DASHBOARD — llama3.2
================================================================================
  Config          Backend    P50 TTFT  P95 TTFT   tok/s    Recommendation
  ----------------------------------------------------------------------------
  short-output    ollama       3.04s    13.58s    11.5    Good — interactive chat
  medium-output   ollama       2.94s     2.94s    11.2    Good — interactive chat
  long-output     ollama       3.05s     3.16s    10.6    Good — interactive chat
================================================================================

  Best latency:    medium-output — P50 2.94s
  Best throughput: short-output  — 11.5 tok/s

What the numbers reveal

Run 1 took 13.58 seconds. Runs 2 and 3 took 3 seconds.

That is not noise. That is the cold start — the model loading into memory the first time. Every subsequent request reuses the warm model.

P95 TTFT captures this: 13.58s. P50 TTFT shows the warm experience: 3.04s. If you averaged all three runs you’d get ~6.5s — a number that represents neither the typical experience nor the worst case. P50 and P95 together tell the complete story.

This is why production inference servers keep models warm at all times. Pay the cold start cost once at startup, not on a real user’s first request.

Throughput drops as output length increases: 11.5 → 11.2 → 10.6 tok/s.

Small drop but the pattern is real. Longer outputs mean more decode iterations and more KV cache memory pressure. This matches the prefill vs decode analysis from benchmark 04.

What the tool does

Single Python file, no dependencies beyond requests. Four backends:

For each backend it runs three output-length configurations, measures P50/P95 TTFT and tokens/sec, calculates cost per 1K tokens, and gives a recommendation. Saves a JSON report for logging over time.

How to run it

git clone https://github.com/anirudhs16/llm-inference-benchmarks.git
cd llm-inference-benchmarks
pip install requests

ollama serve
python dashboard.py --model llama3.2

# OpenAI
python dashboard.py --model gpt-4o-mini --backend openai --api-key YOUR_KEY

# Save report
python dashboard.py --model llama3.2 --save

Why I built this

Every benchmark I ran over the last 12 weeks answered one specific question. What was missing was a single tool that surfaces latency, throughput, and cost in one shot.

The cold start number alone changed how I think about deployment. A 13.58s first request vs a 3.04s warm request is a 4.5x difference in user experience — invisible unless you measure P95 alongside P50.

Full benchmark series: anirudhs16.github.io/ai-systems

Dashboard code: github.com/anirudhs16/llm-inference-benchmarks