AI Pod Manual

English

简体中文

English

简体中文

Appearance

VLLM ​

By default, the AI Pod uses ollama to provide large model services, as it allows easy switching among various models. If your requirements are fixed and you do not frequently switch or release models, you can deploy your service using vllm.

We recommend selecting the appropriate vLLM container image based on your hardware.

For Orin, you can use the vLLM container image packaged for the AI Pod (vllm 0.16.0 + cu126):

bash docker run --rm -it registry.lazycat.cloud/x/lzc-aipod-vllm:d59c2ca bash

For Jetson Thor, prioritize testing the official NVIDIA vLLM container image:

bash docker run --rm --runtime=nvidia --ipc=host --network=host nvcr.io/nvidia/vllm:26.04-py3 python3 -c "import vllm, torch; print(vllm.__version__, torch.__version__, torch.cuda.get_device_name(), torch.cuda.get_device_capability())"

The container image has been verified on Thor: it recognizes NVIDIA Thor and supports the OpenAI-compatible /v1/models and /v1/chat/completions endpoints. If the model configuration includes multimodal fields but only text inference is required, add --language-model-only during startup.

Thor can also use the Jetson AI-IOT container image:

docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor

This container image has successfully run Qwen3.5-27B-Text-NVFP4-MTP. However, Qwen3.5/3.6 27B FP8 is currently not recommended as the default configuration for the GDN linear-attention path on Thor.

If you wish to use the official Jetson container image, refer to dustynv/vllm, for example: docker pull dustynv/vllm:r36.4-cu129-24.04.

Additional Resources ​

  1. https://hub.docker.com/r/mitakad/vllm
  2. Resource List
  3. https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html
  4. https://docs.nvidia.com/deeplearning/frameworks/vllm-release-notes/

Example ​

The following example uses the translation model pre-installed in the AI Pod (Qwen/Qwen3-4B-Instruct-2507 + vLLM):

  1. Run the following command directly in the AI Pod:

docker run --rm -p 9999:3000 -ti registry.lazycat.cloud/x/lzc-aipod-trans:3480c9d

  1. Access it via the local network.
curl http://192.168.1.167:9999/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "aipod-trans", "messages": [{"role": "user", "content": "hello!"}]}'

Note: Remember to change the IP address (192.168.1.167) to your local device's IP address.