Inference · Architecture · Systems

AI infrastructure, written from the systems up.

How AI actually runs — inference engineering, GPU systems, architecture patterns, and projects. Written by a software engineer with a background in semiconductor memory design.

Sep 12, 2026

inference

I built a tool to benchmark any LLM in one command. Here are the numbers.

After 12 weeks of running individual benchmarks — concurrent latency, quantization, prefill vs decode, speculative decoding, PyTorch profiler, Trit...
Sep 05, 2026

inference

Writing a matrix multiplication kernel in Triton — the operation at the heart of every LLM

Last week I wrote a vector addition kernel in Triton — the simplest possible GPU program. This week: matrix multiplication. The operation that runs...
Aug 29, 2026

inference

I wrote my first GPU kernel in Triton. Here is what it taught me.

Everything I have measured so far — TTFT, throughput, kernel call overhead — happened at the PyTorch layer. This week I went one layer deeper and w...
Aug 02, 2026

architecture

The AI vendor landscape: a practical decision framework across 4 layers

One of the most common questions when building AI systems is not which model is best. It is which combination of vendors, at which layer of the sta...
Jul 05, 2026

inference

Speculative decoding: I ran it. The results were messier than expected.

The theory is clean. A small fast model drafts tokens. A large model verifies them in one pass. Fewer forward passes, faster responses.
Jun 12, 2026

inference

I profiled a single forward pass. 1460 GPU calls for 20 tokens.

Every benchmark so far has measured that something is slow — TTFT, tokens per second, memory usage. This week I looked inside a single forward pass...
Jun 10, 2026

inference

I built a CLI to benchmark any LLM. Here's what it outputs.

For the last few weeks I’ve been running one-off notebooks — copy a cell, change a number, run it, read the output. Useful for learning, but not re...
Jun 08, 2026

inference

LLM inference has two phases. They need completely different optimizations.

I ran a simple experiment this week. I asked the same model the same topic three times — once asking for one sentence, once for a paragraph, once f...
Jun 05, 2026

inference

I ran FP16, INT8, and INT4 on the same model. The results surprised me.

Quantization is one of the most talked-about inference optimizations. The pitch is simple: store model weights at lower precision, use less memory,...
May 24, 2026

inference

vLLM vs Ollama: I ran the same benchmark on both. 99× difference.

Last week I measured how Ollama handles concurrent requests. Per-request latency degraded 1.8× going from 1 to 4 simultaneous users. User 4 waited ...
May 22, 2026

inference

I measured what happens when 4 users hit a local LLM at the same time

Theory is one thing. Running it and watching the numbers is another.
May 15, 2026

architecture

Why enterprise AI projects fail — and it is never the model

Conversations with engineers building AI systems in enterprise settings reveal a consistent pattern: the projects that fail almost never fail becau...
May 08, 2026

inference

PagedAttention: the OS trick that made LLM serving practical

Last week I got vLLM running on a T4 GPU in Colab. First real number: 0.30s TTFT for a single request on facebook/opt-125m. The engine reserved 6.5...
May 01, 2026

inference

The roofline model: why your GPU is probably not doing what you think

In chip design, before you tape out a block, you ask one question: is this workload compute-bound or memory-bound? The answer determines everything...
Apr 18, 2026

inference memory

KV cache: why a long conversation can exhaust a 16 GB GPU

Last week I wrote about FlashAttention — how a memory layout insight cut attention’s HBM traffic from O(N²) to O(N). This week I want to show you w...
Apr 11, 2026

inference hardware

FlashAttention: the memory layout insight that changed AI inference

One of the most important AI inference breakthroughs of the last three years was not a better model. It was a memory layout insight — the same kind...