Qwen3 Embedding 0.6B on Orin

Tested vllm on orin machine using Qwen/Qwen3-Embedding-0.6B.

Test Configuration

• Machine: orin
• Container image: registry.lazycat.cloud/x/lzc-aipod-vllm:bffa39b-orin
• Model: Qwen/Qwen3-Embedding-0.6B
• Deployment mode: vllm serve --runner pooling
• GPU memory: --gpu-memory-utilization 0.65 --max-model-len 32768 --enforce-eager
• Dataset: datasets/qwen3_embedding_texts.jsonl
• Concurrency: 1,2,4,6,8,16
• Batch size: 1,8,32,128

Conclusion

• OpenAI Embeddings-compatible API is available; both /v1/models and /v1/embeddings pass.
• Output vector dimension is 1024.
• Under small batch, low-latency mode, batch=1 concurrency=16 achieves peak QPS=99.7079, average latency of 0.1465s, with p95=0.1781s.
• In large batch throughput mode, batch=128 concurrency=8 reaches its peak of items/s=706.4871, prompt tokens/s=11868.5, with an average latency of 1.2431s, p95=1.5489s.

Key Results

batch	concurrency	items/s	prompt tokens/s	avg latency (s)	p95 (s)
1	16	99.7079	1723.0776	0.1465	0.1781
8	8	170.4528	2883.0500	0.3544	0.3940
32	8	522.1799	8784.2497	0.4447	0.5071
128	8	706.4871	11868.5000	1.2431	1.5489

Telemetry

• Sample count: 73
• Maximum temperature: 58.937 C
• Maximum power consumption: 41189 mW
• Maximum fan PWM: 20.7843
• Maximum GPU utilization: 99.9%

Outputs

Remote results directory:

/home/nvidia/bench-results/qwen3-embedding-vllm

Key files:

/home/nvidia/bench-results/qwen3-embedding-vllm/qwen3_embedding_bench.json
/home/nvidia/bench-results/qwen3-embedding-vllm/models.json
/home/nvidia/bench-results/qwen3-embedding-vllm/server.log
/home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.jsonl
/home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.summary.json