Single Usage

You can deploy services independently within an AI Pod using docker-compose. Please select the corresponding configuration based on the AI Pod model.

AGX Orin

services:
  ollama:
    image: registry.lazycat.cloud/catdogai/jetson-ollama:0.17.5
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - ./data:/root/.ollama
    environment:
      - LANGUAGE=en_US:en
      - LANG=en_US.UTF-8
      - LC_ALL=en_US.UTF-8
      - OLLAMA_DEBUG=1
      - OLLAMA_ORIGINS=*
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_FLASH_ATTENTION=1
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_CONTEXT_LENGTH=8192
      - OLLAMA_KV_CACHE_TYPE=q8_0
      - OLLAMA_LLM_LIBRARY=cuda_jetpack6
      - LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_jetpack6:/usr/local/cuda/compat:/usr/local/cuda/lib64
    ports:
      - 11434:11434
    mem_limit: 60G
    memswap_limit: 60G

Thor

services:
  ollama:
    image: registry.lazycat.cloud/catdogai/jetson-ollama:0.17.5
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - ./data:/root/.ollama
    environment:
      - LANGUAGE=en_US:en
      - LANG=en_US.UTF-8
      - LC_ALL=en_US.UTF-8
      - OLLAMA_DEBUG=1
      - OLLAMA_ORIGINS=*
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_FLASH_ATTENTION=1
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_KV_CACHE_TYPE=q8_0
      - LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/cuda/lib64:/usr/local/cuda-13.0/lib64:/usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu
    ports:
      - 11434:11434
    mem_limit: 60G
    memswap_limit: 60G

Save the configuration for the corresponding model to docker-compose.yml, then start it using docker-compose up -d in the same directory.

After successful startup, access it via the 11434 port on the local network.

The latest version of ollama automatically sets the context length based on available memory. For the AGX Orin AI Pod, with 64G of memory, the default context length may be large. For example, use OLLAMA_CONTEXT_LENGTH=8192 to limit the context length to 8k.

Info

If you cannot access registry.lazycat.cloud on your AI Pod, you can switch to dockerhub and access it via docker pull catdogai/jetson-ollama:0.17.5.

Ollama frequently updates, and the documentation may not be updated in time. The latest Ollama version tags can be found on the https://hub.docker.com/r/catdogai/jetson-ollama/tags page.