vLLM Distributed Inference on Jetson Orin AGX

Official Documentation

Test Container Images

1. mitakad/vllm:0.11.2-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated
2. mitakad/vllm:0.12.0-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated
3. mitakad/vllm:0.13.0-r36.4.tegra-aarch64-cp312-cu129-24.04-pre-release

Steps

Note: The model must be downloaded separately and placed in the current directory. Execute the command on the master node.

pip install modelscope
~/.local/bin/modelscope download --model openai-mirror/gpt-oss-120b --local_dir ./openai-mirror/gpt-oss-120b

Connect the 10G network port on the master node to the 10G port of the AI Pod on another node using a network cable.

1. Master Node

sudo ip addr add 192.168.38.1/24 dev eno1

VLLM_HOST_IP=192.168.38.1 && \
VLLM_IFNAME=eno1 && \
docker run -d --rm --runtime nvidia -it \
    --network host \
    --shm-size=8g \
    --privileged \
    --ipc=host \
    -v /dev/shm:/dev/shm \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e VLLM_HOST_IP=${VLLM_HOST_IP} \
    -e UCX_NET_DEVICES=${VLLM_IFNAME} \
    -e NCCL_SOCKET_IFNAME=${VLLM_IFNAME} \
    -e GLOO_SOCKET_IFNAME=${VLLM_IFNAME} \
    -e TP_SOCKET_IFNAME=${VLLM_IFNAME} \
    -v `pwd`/:/models \
    -v `pwd`/cache:/root/.cache/huggingface \
    --name vllm_xray \
    mitakad/vllm:0.11.2-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated

docker exec -ti vllm_xray bash

ray start -v  --head --port 6379 --node-ip-address 192.168.38.1 --num-gpus 1
sysctl vm.drop_caches=3

2. Other Nodes

sudo ip addr add 192.168.38.2/24 dev eno1

VLLM_HOST_IP=192.168.38.2 && \
VLLM_IFNAME=eno1 && \
docker run -d --rm --runtime nvidia -it \
    --network host \
    --shm-size=8g \
    --privileged \
    --ipc=host \
    -v /dev/shm:/dev/shm \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    -e VLLM_HOST_IP=${VLLM_HOST_IP} \
    -e UCX_NET_DEVICES=${VLLM_IFNAME} \
    -e NCCL_SOCKET_IFNAME=${VLLM_IFNAME} \
    -e GLOO_SOCKET_IFNAME=${VLLM_IFNAME} \
    -e TP_SOCKET_IFNAME=${VLLM_IFNAME} \
    -v `pwd`/:/models \
    -v `pwd`/cache:/root/.cache/huggingface \
    --name vllm_xray \
    mitakad/vllm:0.11.2-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated

docker exec -ti vllm_xray bash

ray start -v  --address 192.168.38.1:6379 --node-ip-address 192.168.38.2 --num-gpus 1
sysctl vm.drop_caches=3

3. Copy the model to another worker node in the same directory. The directory I placed is ~.

rsync -aP ~/openai-mirror nvidia@192.168.38.2:~

4. Execute the following command on the primary node machine

docker exec -ti vllm_xray bash

# http://192.168.1.128:1085 是我本地的代理，您可以去掉或者更改成您的
curl -x http://192.168.1.128:1085 -L  https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken -o fb374d419588a4632f3f557e76b4b70aebbca790

# 测试是否正常
TIKTOKEN_RS_CACHE_DIR=$(pwd) python -c 'from openai_harmony import load_harmony_encoding; load_harmony_encoding("HarmonyGptOss")'

# 启动模型
sysctl vm.drop_caches=3
TIKTOKEN_RS_CACHE_DIR=$(pwd) TRANSFORMERS_OFFLINE=1 vllm serve /models/openai-mirror/gpt-oss-120b --port 8000 \
    --served-model-name gpt-oss-120b \
    --max-model-len 2048 \
    --max-num-batched-tokens 2048 \
    --kv-cache-memory-bytes 3000M \
    --max-num-seqs 10 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \
    --gpu-memory-utilization 0.8 \
    --distributed-executor-backend ray

5. Access

Access via curl on the terminal. 192.168.1.175 is the IP address in my local network; you need to replace it with your machine's IP address.

curl 'http://192.168.1.175:8000/v1/chat/completions' \
    --header 'Content-Type: application/json' \
    --data '{
        "model": "gpt-oss-120b",
        "messages": [
            {
                "role": "user",
                "content": "who are you?"
            }
        ],
        "stream": false
    }'

Test Data

model	Tokens/s
Qwen3-4B-Instruct-2507-FP8	10.5
openai/gpt-oss-120b	10.1

Question

1. If you encounter errors like "repo_id is incorrect" when loading a local model with vLLM, the simplest solution is to manually modify the verification function in the HuggingFace Hub validation script.