Qwen3 Embedding 0.6B on Thor

Test the official vllm container image on the thor machine with Qwen/Qwen3-Embedding-0.6B.

Test Configuration

• Machine: thor
• Container image: nvcr.io/nvidia/vllm:26.01-py3
• Container internal vllm version: 0.13.0+faa43dbf.nv26.01
• Model: Qwen/Qwen3-Embedding-0.6B
• Deployment mode: vllm serve --runner pooling
• Start parameters: --gpu-memory-utilization 0.65 --max-model-len 32768 --enforce-eager
• Dataset: datasets/qwen3_embedding_texts.jsonl
• Concurrency: 1,2,4,6,8,16
• Batch size: 1,8,32,128

Conclusion

• The official container image directly provides an OpenAI Embeddings-compatible interface on Thor.
• Both models and embeddings requests pass, with output vector dimension of 1024.
• In low-latency mode, batch=1 concurrency=16 reaches its peak of QPS=207.9151, with an average latency of 0.0688s, and p95=0.0851s.
• In throughput mode, batch=128 concurrency=16 reaches its peak of items/s=1371.0304, prompt tokens/s=23019.6541, average latency of 1.1980s, and p95=1.5949s.

Key Results

batch	concurrency	items/s	prompt tokens/s	avg latency (s)	p95 (s)
1	16	207.9151	3593.0330	0.0688	0.0851
8	16	584.2202	9794.8176	0.1909	0.2693
32	16	934.1475	15679.8117	0.4482	0.5799
128	16	1371.0304	23019.6541	1.1980	1.5949

Telemetry

• Sample count: 32
• Maximum temperature: 34.437 C
• Maximum power consumption: 8102 mW
• Maximum fan PWM: null
• Maximum GPU utilization: null

Output

Remote result directory:

/home/nvidia/bench-results/qwen3-embedding-vllm

Key files:

/home/nvidia/bench-results/qwen3-embedding-vllm/qwen3_embedding_bench.json
/home/nvidia/bench-results/qwen3-embedding-vllm/models.json
/home/nvidia/bench-results/qwen3-embedding-vllm/server.log
/home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.jsonl
/home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.summary.json