Ollama, vLLM, and SGLang on a DGX Spark: A Structured-Inference Horserace

tl;dr: For one request, ordinary vLLM and SGLang were close to Ollama. At eight concurrent requests, standard vLLM was 6.1x faster and SGLang 5.8x faster; enabling Qwen3.6’s native MTP head in vLLM raised the speedup to 9.8x while reducing median latency to 16.9 seconds. Ollama stopped scaling beyond one request. All conditions returned valid JSON. Standard vLLM and SGLang were effectively tied; the decisive advantage came from speculative decoding.

The DGX Spark used for these measurements.

Why benchmark a real task?

Tokens per second are useful, but they are not the quantity I ultimately care about. I had a queue of short, structured classification requests: read a question and an answer from an earnings call, apply a fixed taxonomy, and return a small JSON object. This taxonomy comes from the work with Khaled Al Nuaimi and colleagues (Al Nuaimi et al., FinNLP 2025). The practical question was therefore:

How many valid classifications per second can one NVIDIA DGX Spark produce, and do vLLM or SGLang materially change that number relative to Ollama?

This workload is quite different from asking a model to write one long essay. The input is much longer than the output, every response must satisfy a JSON schema, and throughput should improve when independent requests are batched. It is precisely the kind of workload for which vLLM’s continuous batching may matter.

The comparison also tests speculative decoding. The elegant idea in Leviathan, Kalman, and Matias’s original paper is to propose future tokens cheaply, then verify several proposals together with the target model instead of decoding every token serially. Qwen3.6-27B packages a related self-speculative route as a native multi-token-prediction (MTP) head: vLLM uses that head to draft tokens and the main model verifies them. Whether this helps is an empirical question because short JSON answers, constrained decoding, and a modest batch size may behave differently from long-form generation.

Hardware and software

The experiment runs on one NVIDIA DGX Spark. Its GB10 Grace Blackwell Superchip combines an Arm CPU and Blackwell GPU with 128 GB of coherent LPDDR5x memory. NVIDIA reports 273 GB/s of memory bandwidth and a 140 W GB10 TDP in the official specifications.

The model is Qwen3.6-27B. The official model card describes a dense 27B language model, BF16 weights, a 262,144-token native context, and an MTP head trained for multiple prediction steps.

Component	Version or artifact
Machine	NVIDIA DGX Spark / GB10 / Arm64
System memory	128 GB coherent unified memory
Ollama	0.32.5
vLLM	0.26.0, PyTorch 2.11, CUDA 12.9 Arm64 wheel
SGLang	0.5.12.post1, PyTorch 2.11, CUDA 13.0
Ollama model	qwen3.6:27b-bf16, F16 GGUF, exact digest recorded
vLLM model	Qwen/Qwen3.6-27B, BF16 safetensors, pinned revision 6a9e13bd…
SGLang model	The same pinned BF16 safetensors as vLLM

The three setups use the same model family and parameter count, but they are not identical under the hood. Ollama loads an F16 GGUF file, whereas vLLM and SGLang load the same official BF16 files. Each server also comes with its own software stack. This is therefore an optimized-stack comparison: I am comparing complete setups that I could actually run, not changing only one component in a laboratory experiment. The results tell me which setup was faster here, but not exactly which internal difference caused the speedup. The The comparison between vLLM and SGLang is somewhat cleaner because they use the same model files and precision.

The test workload

Before benchmarking, I selected 60 exchanges from a larger research panel, balanced across the three classification labels. Half come from cases where earlier labeling methods disagreed, which makes them more likely to be unclear or borderline. The other half come from the remaining, more straightforward cases. This is not an objective measure of difficulty; it simply prevents the benchmark from containing only easy examples.

I used a fixed random selection that can be reproduced exactly, rather than choosing examples by hand. Most importantly, the same 60 exchanges were fixed before the benchmark and given to Ollama, vLLM, and SGLang. None of the three was allowed to influence which examples entered the test.

The rendered Qwen chat prompts contain, on average, 692 tokens (median 676, range 534 to 1,148). Outputs are deliberately short and capped at 256 tokens. The observed output-token totals are reported for every condition, because a backend that emits less text can otherwise appear artificially faster.

Every request uses:

• the same system rubric, question, and answer;
• temperature zero and seed 20260731;
• a 16,384-token context and 256-token output cap;
• thinking disabled;
• the same strict JSON schema;
• three fixed warm-up requests excluded from measurement.

The exact panel, selection code, and benchmark runner are retained with the research artifacts.

Conditions and metrics

I benchmark Ollama at concurrency 1, 2, and 4, and both vLLM and SGLang at concurrency 1, 2, 4, and 8. vLLM is run once with ordinary decoding and once with two-token MTP speculation. I disable prefix caching in both vLLM and SGLang because repeated prompts could otherwise make later tests artificially faster.

Ollama documents that parallel requests multiply the context-memory allocation by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH; this makes memory part of the concurrency trade-off, not a free setting (Ollama FAQ).

For every condition I record:

• completed requests per second;
• prompt and generated tokens per second;
• median and p95 end-to-end latency;
• time to first token;
• structured-JSON success;
• exact classification agreement between backends;
• GPU memory, utilization, and power samples;
• server/model startup time and failed configurations;
• MTP drafted-token acceptance when vLLM exposes it.

Results

The result is not that one server is always faster. Ordinary vLLM at concurrency one was 9% slower than Ollama, while SGLang was 3% slower. The advantage appears when independent requests can be continuously batched, and becomes much larger when the native MTP head is enabled.

Engine	Concurrent requests	Requests/s	Speedup vs Ollama c1	Median latency (s)	p95 (s)	Median TTFT (s)	Output tokens/s	GPU GiB
Ollama F16	1	0.0453	1.00x	22.1	26.2	1.26	3.97	51.2
Ollama F16	2	0.0455	1.00x	44.2	49.2	22.79	3.99	51.2
Ollama F16	4	0.0439	0.97x	90.7	101.5	68.93	3.85	51.2
vLLM BF16	1	0.0414	0.91x	23.8	27.5	1.12	3.66	79.5
vLLM BF16	2	0.0793	1.75x	25.4	28.5	1.37	6.95	79.8
vLLM BF16	4	0.1546	3.41x	25.5	29.4	1.29	13.59	79.8
vLLM BF16	8	0.2756	6.08x	27.2	32.9	1.50	24.24	79.8
vLLM BF16 + MTP	1	0.0911	2.01x	11.0	12.9	1.20	7.91	75.2
vLLM BF16 + MTP	2	0.1794	3.96x	11.1	13.0	1.38	15.77	75.2
vLLM BF16 + MTP	4	0.2751	6.07x	14.3	16.8	1.47	24.13	75.2
vLLM BF16 + MTP	8	0.4426	9.76x	16.9	21.1	1.93	39.09	75.2
SGLang BF16	1	0.0440	0.97x	22.6	27.4	0.92	3.90	71.9
SGLang BF16	2	0.0835	1.84x	23.8	27.8	1.11	7.38	72.4
SGLang BF16	4	0.1527	3.37x	25.7	31.3	0.85	13.58	72.4
SGLang BF16	8	0.2645	5.84x	28.6	35.6	1.20	23.46	73.3

The latency view makes the practical difference clearer. Ollama c2 and c4 do not increase throughput: requests simply wait behind one another. vLLM’s continuous batching keeps ordinary decoding near 24-27 seconds through c8. MTP is faster still.

What explains the difference?

The workload is decode-heavy despite having longer inputs than outputs. Ollama processed the 60-request panel at 3.85-3.99 output tokens/s regardless of its parallelism setting. Its c4 median time to first token reached 68.9 seconds, which looks like queueing rather than useful parallel work.

Ordinary vLLM behaves differently. Output throughput rises from 3.66 tokens/s at c1 to 24.24 at c8, while median time to first token remains below 1.6 seconds. Continuous batching therefore turns concurrency into throughput rather than a queue.

MTP changes the single-request result as well. Across concurrency levels, 84.9-85.2% of its drafted tokens were accepted. At c1, that more than doubled request throughput relative to ordinary vLLM and cut median latency from 23.8 to 11.0 seconds. At c8 it added another 61% throughput over standard vLLM. The MTP executor also used about 4.6 GB less GPU memory in this configuration, although memory allocation is an implementation detail rather than a general property of speculative decoding.

SGLang and standard vLLM are much closer than either is to vLLM+MTP. At matched concurrency, SGLang is 6.1% and 5.3% faster at c1 and c2, then 1.2% and 4.0% slower at c4 and c8. Its median latency stays between 22.6 and 28.6 seconds. I would call that an engineering tie on this panel, not evidence that one scheduler is universally faster. At c8, MTP vLLM is 67% faster than SGLang without speculation.

Output quality: did the engine change the answer?

Every one of the 900 measured responses across the 15 primary conditions passed the strict JSON parser. Completion-token totals were also close: 5,213-5,336 tokens per condition, so the fastest engine did not win by quietly emitting much shorter answers.

The class label was highly stable but not mathematically guaranteed to be identical. Relative to fresh Ollama c1:

• standard vLLM matched 60/60 labels at c1, c4, and c8, and 59/60 at c2;
• MTP matched 58/60 at c1, 59/60 at c2 and c4, and 60/60 at c8;
• SGLang matched 59/60 at every concurrency and produced identical class labels across all four of its conditions;
• the fastest condition, MTP c8, therefore matched Ollama exactly on this panel.

Some rationale wording and token counts differed, as expected from different precision, containers, kernels, and speculative paths. These checks establish serving consistency, not human-label accuracy.

A brief note on Arm64 setup

I ran vLLM and SGLang in separate environments rather than modify the research environment that produced the Ollama labels. Both engines work on the DGX Spark, although setup still had a few Arm64 rough edges: vLLM needed Python headers exposed to Triton and ninja on the server process’s PATH, while SGLang needed a compatible kernels pin. These were setup issues, not steady-state performance limitations.

I did not install vLLM’s experimental GGUF plugin. The results therefore compare the tested deployments as configured; they are not an exact-GGUF engine-only comparison.

What I would use

For throughput-oriented workloads, vLLM with Qwen’s trained MTP head is the clear winner on this DGX Spark. It completed nearly ten times as many classifications per second as Ollama c1 while also delivering lower median latency. For a queue of independent structured requests, this is the stack I would use.

The recommendation assumes enough memory for the BF16 executor, enough concurrent work to exploit batching, and tolerance for the Arm64/CUDA toolchain. For occasional one-off requests, Ollama remains attractive because it is simpler and uses less memory. But when throughput matters, leaving vLLM’s batching and the model’s MTP head unused leaves most of the machine’s serving capacity idle.

Without speculative decoding, there is no clear speed winner between standard vLLM and SGLang in these results; I would choose between them on operational fit.

Reproducibility and limitations

This benchmark covers one machine, one 27B model, one short-output workload, and a frozen 60-request panel; it is not a universal ranking of inference engines. Differences in precision and packaging also prevent a clean engine-only interpretation of the Ollama comparison.

The reference labels are provisional research annotations, not human ground truth. Agreement measures whether changing the serving stack changes the model output, not classification accuracy.