Internship study

Same GPU, three engines.

Software intern at Voxomos Systems, June to July 2026. I served one small model on vLLM, SGLang and TensorRT-LLM on a 6 GB laptop GPU to find what actually limits LLM serving, then wrote the GPU kernels behind vector search.

Decode floor
13.7 ms
per token at no load; vLLM and SGLang hit 15.1. The GPU sets the floor, not the engine.
Admission control
1.2 s
to first token with 32 requests in flight, against 25–76 s uncapped.
Prefix caching
2.8×
more vLLM requests per second when prompts share a 500-token prefix.
Energy per token
44×
TensorRT-LLM tokens per watt at batch 768 against batch 1.

Setup

GPU: RTX 3050 Laptop, 6 GB (5.67 GB usable)
Model: TinyLlama-1.1B-Chat, bf16
Parallelism: One GPU, no tensor or pipeline split
Load tools: vllm bench serve, sglang.bench_serving, trtllm-bench
Workloads: ShareGPT, shared-prefix and fixed-length prompts

LLM serving

How each engine behaved.

Each engine ran under its own load tool and workloads, so read these as how each one behaves under pressure, not as a head-to-head ranking.

vLLM

vllm bench serve

Throughput kept climbing as the arrival rate rose to 32 req/s, while time to first token stayed under 80 ms. Its best run came from automatic prefix caching, on prompts that shared a 500-token prefix.

Workload	Arrival	Requests	Req/s	Output tok/s	Mean TTFT	Mean TPOT
ShareGPT	All at once	100	7.47	659	2,248 ms	85.3 ms
ShareGPT	1 req/s	1,000	1.00	94	71 ms	15.1 ms
ShareGPT	4 req/s	250	3.42	348	49 ms	15.5 ms
ShareGPT	8 req/s	250	5.73	585	55 ms	16.9 ms
ShareGPT	32 req/s	250	11.02	1,112	80 ms	23.7 ms
Shared 500-token prefix	All at once	100	20.80	2,541	865 ms	31.5 ms

SGLang

sglang.bench_serving

With no cap on in-flight requests, throughput flattened near 1.4 req/s while the queue grew, and time to first token reached minutes. Capping it at 32 requests kept similar traffic at about a second.

Workload	Arrival	Requests	In-flight cap	Req/s	Output tok/s	Mean TTFT	Mean TPOT
Mixed prompts	All at once	100	None	0.31	161	156.8 s	31.1 ms
Shared prefix, 1 group	All at once	100	None	1.57	403	28.3 s	24.8 ms
Shared prefix, 10 groups	20 req/s	1,000	None	0.80	204	591.1 s	167.1 ms
ShareGPT	1 req/s	1,000	32	0.99	223	1.24 s	15.1 ms
ShareGPT	2 req/s	250	None	1.36	334	25.2 s	19.1 ms
ShareGPT	8 req/s	250	None	1.41	345	65.7 s	23.0 ms
ShareGPT	32 req/s	250	None	1.43	350	75.7 s	24.7 ms

TensorRT-LLM

trtllm-bench

Its offline bench sizes the batch to whatever fits in KV-cache memory: 768 short requests or 160 long ones. Batch size drove both throughput and tokens per watt. At batch 1 it gave the cleanest latency: 142 ms to first token and 13.7 ms per token.

Workload	Batch	Tokens in / out	Req/s	Output tok/s	GPU power	Tok/s per watt
Short sequences, auto batch	768	128 / 128	23.97	3,068	134.7 W	22.8
Long sequences, auto batch	160	1,024 / 256	4.39	1,125	74.9 W	15.0
Batch capped, 70% KV memory	32	1,024 / 256	3.31	849	139.1 W	6.1
Latency mode	1	1,024 / 256	0.28	71	136.3 W	0.5

Takeaways

What the numbers say.

Five things I'd tell anyone serving models on a small GPU.

Decode speed is set by the GPU.
At no load, all three engines landed between 13.7 and 15.1 ms per token. The engine decides how gracefully a system degrades, not how fast it can go.
Reused prefixes are the cheapest throughput.
vLLM's automatic prefix caching and SGLang's RadixAttention both posted their best numbers when prompts shared a long prefix.
Batch size is bounded by KV-cache memory.
TensorRT-LLM's throughput tracked the batch it could fit, and the same knob moved tokens per watt by 44×.
Cap what's in flight.
Uncapped, SGLang accepted more work than the GPU could serve and requests waited minutes for a first token. A cap of 32 held it near a second.
On 6 GB, memory is the bottleneck.
Every lever that mattered, from batch size to cache fraction to prefix hits, was a way of rationing the same KV-cache memory.

Vector search

Where the GPU wins, and where it doesn't.

Cosine similarity is the core operation behind every vector database, and at about 0.25 FLOP per byte it is memory-bound. I wrote the same search five ways and measured where a GPU actually helps.

Batch search, 100k
22×
CUDA against an AVX2 + OpenMP CPU baseline.
Batch search, 1M
16×
the same kernels on a corpus ten times larger.
Single-query crossover
~3,000
dimensions; below it, kernel launch overhead keeps the CPU ahead.

What I built

Code on GitHub

CPU: Scalar, AVX2, and AVX2 + OpenMP
CUDA, naive: One thread per vector pair
CUDA, optimized: Tiled shared memory, cuBLAS and async streams
HIP: A native port with 64-wide wavefronts, plus a HIPIFY-generated version
Profiling: Nsight Systems timelines and Nsight Compute kernel counters

Batch search

RTX 3050 Laptop GPU

Vectors searched	CPU	CUDA	Speedup
10,000	24.7 ms	1.14 ms	21.6×
100,000	228.5 ms	10.4 ms	22.0×
1,000,000	2,424 ms	150.6 ms	16.1×

Single query, by dimension

One vector at a time

With one query, kernel launch overhead costs more than the GPU saves until vectors pass about 3,000 dimensions.

Dimensions	CPU	CUDA, atomic	CUDA, reduction	Best GPU speedup
128	2.8 µs	52.1 µs	47.4 µs	0.06×
256	4.8 µs	30.8 µs	39.6 µs	0.16×
384	8.3 µs	35.9 µs	32.0 µs	0.26×
768	12.6 µs	32.7 µs	32.6 µs	0.39×
1,024	17.4 µs	32.4 µs	31.8 µs	0.55×
1,536	25.3 µs	37.2 µs	34.7 µs	0.73×
3,072	49.2 µs	30.8 µs	33.0 µs	1.60×
4,096	65.9 µs	31.6 µs	43.9 µs	2.09×

Reading

Also studied.

Topics I went deep on alongside the benchmarks.

FlashAttention v1–v3
FlashInfer
PagedAttention and KV-cache sizing
Continuous batching
Chunked prefill
Speculative decoding (EAGLE-3)
Prompt caching
Model routing
LLM observability
Serverless cold starts
Cache-aware routing
DeepSeek-V3 MoE and DualPipe
NCCL and RCCL
ROCm
SYCL and oneAPI
Roofline model

Hiring for backend, forward-deployed or GPU work? Let’s talk.

saxena.shikhar1808@gmail.com

GitHub LinkedIn LeetCode Résumé

Setup

vLLM

SGLang

TensorRT-LLM

Decode speed is set by the GPU.

Reused prefixes are the cheapest throughput.

Batch size is bounded by KV-cache memory.

Cap what's in flight.

On 6 GB, memory is the bottleneck.

What I built

Batch search

Single query, by dimension