# Configure Browser Flags ## Configure Local Network Access Checks In the latest version of `Chrome (142+)`, the following issues may occur: - The "Page Summary" page shows an error of `Failed to fetch` - The developer tools console displays `...has been blocked by CORS policy: Permission was denied for this request to access the 'unknown' address space.` 1. Open `Chrome` in `chrome://flags/#local-network-access-check` 2. Change the setting from `Default` to `Disabled`; the default option is `Blocking` ![local network access checks](../../ai-browser/images/config-local-access-network.png) Related links: - https://chromestatus.com/feature/5152728072060928 - https://developer.chrome.com/blog/local-network-access ## Configure Network 1. Open the address `chrome://proxy` ![config network](../../ai-browser/images/config-network.png) 2. In the input field at the bottom, enter `*.heiyu.space;*.lazycat.cloud;` to gain access to the addresses `*.heiyu.space` and `*.lazycat.cloud`. 3. Click the `Apply` button to apply. --- # Image Translation ![image result](../../ai-browser/images/aipod-image-trans-result.png) In the AI Plugin Assistant, image translation is supported. You can select your preferred languages in the AI Pod configuration options. ![image config](../../ai-browser/images/aipod-config-image-trans-options.png) After configuration, when browsing a website, place your mouse over an image. A translation icon will appear in the bottom-right corner of the image. Clicking this icon will initiate translation of the current image. ![image trans icon](../../ai-browser/images/image-trans-tips.png) After translation is complete, the result will be overlaid directly on the original image. You can also click the restore icon in the bottom-right corner to revert to the original format. --- # Self-Signed Certificate Before the certificate is officially issued, you can manually import the certificate into your browser. 1. Open `chrome://certificate-manager/localcerts` in the browser. ![./images/localcerts.jpg](../../ai-browser/images/localcerts.jpg) 2. Download the certificate LazyCatCloud-CA.crt and save it to your local machine. 3. Return to `chrome://certificate-manager/localcerts/usercerts`, and import the certificate in `Trusted certificates`. ![./images/usercerts.jpg](../../ai-browser/images/usercerts.jpg) 4. The result after import ![./images/usercerts-result.jpg](../../ai-browser/images/usercerts-result.jpg) --- # API This document summarizes the backend APIs related to LC-X3. ## Check if a AI Pod is available ### `GET /backend/device/hasDefault` Purpose: - Determine whether the current LCMD device has an available default AI Pod. - Can be used for pre-checking before AI application startup, or to determine during page loading whether to guide users to first add an AI Pod. Request: - Method: `GET` - Path: `/backend/device/hasDefault` - Full URL example: `https://ai./backend/device/hasDefault` - No request body required Response: - `200 OK`: A default AI Pod is available currently. - `503 Service Unavailable`: No available default AI Pod exists currently. Notes: - The response does not return JSON; it uses HTTP status codes directly. - The backend first checks for a default AI Pod. If no explicit default is set, it automatically selects an active device from authorized devices as the default AI Pod. - Even if a default AI Pod exists, if the device has not sent a heartbeat in more than `1` minutes, it will return `503`. Example: ```bash curl -i "https://ai./backend/device/hasDefault" ``` Return `200` as an example: ```http HTTP/1.1 200 OK ``` Return `503` as an example: ```http HTTP/1.1 503 Service Unavailable ``` --- # LazyCat AI Assistant Plugin ::: info Info The LazyCat AI Assistant plugin is distributed via `AI POD`. Ensure that the LazyCat LCMF client is open and online. The plugin version will remain consistent with the current `AI POD` version. ::: ## Plugin Updates In various AI browser interfaces, the plugin will automatically download and install the latest version when you restart the browser. You can check the current plugin version in the `chrome://extensions` interface. ![](../../aipod/images/chrome-extension-version.png) You can also view the current plugin version in the settings of `AI POD`. ![](../../aipod/images/ai-extenstion-version.png) ## Install AI Plugin in Chrome 0. Open `AI POD` → Plugins → Select `cloud.lazycat.app.ai.zip` and click Download. ![](../../aipod/images/extenstions-shortcut.png) 1. Open `Chrome` browser and ensure you are using the latest version of `Chrome` browser. 2. Go to the extensions page by entering `chrome://extensions` in the address bar and pressing Enter. 3. Enable developer mode. Find the "Developer Mode" switch in the top-right corner of the page and ensure it is turned on. 4. Drag and drop the `cloud.lazycat.app.ai.zip` file directly into the `Chrome` extension page. 5. Open a new tab to see that the plugin has been installed (the homepage now shows the `AI POD` page). ## LazyCat Assistant Plugin New Tab The LazyCat Assistant plugin will by default take over new browser tabs. You can switch back to your previous tab page in the plugin settings. ![](../../aipod/images/chrome-extension-newtab.jpg) --- # AI Pod Firmware The AI Pod contains a firmware component responsible for communication with the LCMD. The firmware version must match the `AI POD` version in the LCMD. Firmware updates are available through two methods: - In the AI Browser, the system automatically detects whether an update is available. If an update is available, a popup window indicating an upgrade is shown. Clicking "Upgrade" will trigger the firmware update. ![update tips](../../aipod/images/firemare-update.png) - In the AI Pod management list, when an update is available, a notification prompt will appear. ![device list firemare update](../../aipod/images/device-list-firemare.png) --- # LCMD Port Forwarding The services deployed within the AI Pod can be accessed via `https://xxx-ai.$微服名称.heiyu.space`. However, this method requires the LCMD client. To address the scenario where users need to access the service within a local network, or on machines that cannot install the LCMD client, or in specific environments where access to `heiyu.space` is unavailable, alternative access methods are needed. `AIPOD` The port forwarding in the application only exposes the `xxxx-ai`'s `http` request to the *LCMD* network. You can access it via `http://微服名称.heiyu.space:$端口号`, or directly via `http://微服ip:端口`. 1. Open the settings interface of `AIPOD`, and click Port Forwarding ![port-forward-page](../../aipod/images/port-forward-page.jpg) 2. Add the services you need to forward, such as `ollama` ![port-forward-create-ollama](../../aipod/images/port-forward-create-ollama.jpg) 3. Testing Open `http://微服名称.heiyu.space:端口`, for example, the ports I mentioned above such as `ollama` and `8888`. ![port-forward-ollama.jpg](../../aipod/images/port-forward-ollama.jpg) Or access directly via the local network using the IP address of the LCMD. ``` > curl http://192.168.1.246:8888 Ollama is running⏎ ``` ::: info Info If you need to expose your service to other LCMD devices, use `局域网转发工具` from the [App Store](https://appstore.lazycat.cloud/#/shop/detail/cloud.lazycat.app.forward). ![local-port-forward-tools.jpg](../../aipod/images/local-port-forward-tools.jpg) ::: # AI Pod Port Forwarding Please refer to [AI Pod Port Forwarding](../os/port-forward.md) --- # Related Resources ## NVIDIA Jetson Container Images Some commonly used base container images supported by NVIDIA Jetson, with frequent updates available. 1. https://hub.docker.com/u/dustynv 2. https://github.com/dusty-nv/jetson-containers 3. https://pypi.jetson-ai-lab.io/ 4. https://pypi.jetson-ai-lab.io/jp6/cu129 5. https://pypi.jetson-ai-lab.io/jp6/cu128 6. https://pypi.jetson-ai-lab.io/jp6/cu126 7. Jetson Thor / CUDA 13.0 Python package source: https://pypi.jetson-ai-lab.io/sbsa/cu130 8. Jetson Thor vLLM container image: `ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor` Additional use cases: 1. https://www.jetson-ai-lab.com/tutorial-intro.html ## NVIDIA vLLM Official Resources 1. NGC vLLM container image: `nvcr.io/nvidia/vllm:-py3` 2. Current version verified to boot on Jetson Thor: `nvcr.io/nvidia/vllm:26.04-py3` 3. vLLM version information: `0.19.0+6bc3197f.nv26.04.48680843` 4. Container CUDA / PyTorch: `CUDA 13.2` / `torch 2.12.0a0` 5. Thor Runtime Status: Detects `NVIDIA Thor`, compute capability `(11, 0)`, and can launch OpenAI-compatible `/v1/models` and `/v1/chat/completions` 6. Note: Thor currently supports CUDA 13.0; when running `26.04-py3`, it enters CUDA minor compatibility mode Minimum validation command for Thor: ``` docker run --rm --runtime=nvidia --ipc=host --network=host \ nvcr.io/nvidia/vllm:26.04-py3 \ python3 -c "import vllm, torch; print(vllm.__version__, torch.__version__, torch.cuda.get_device_name(), torch.cuda.get_device_capability())" ``` If the model configuration includes multimodal fields but only text inference is required, when starting `vllm serve`, it is recommended to add `--language-model-only` to avoid vLLM loading a non-existent `preprocessor_config.json`. ## Packaged Container Images Officially packaged container images are generally older and may not receive timely version updates. We also update software versions in these base container images. 1. ollama https://hub.docker.com/r/catdogai/jetson-ollama/tags 2. vLLM https://hub.docker.com/r/catdogai/lzc-aipod-vllm/tags 3. Thor vLLM latest officially verified baseline `nvcr.io/nvidia/vllm:26.04-py3` 4. Thor vLLM Jetson AI-IOT baseline `ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor` ## Common Container Images 1. CUDA container image `dustynv/cuda:12.8-samples-r36.4.0-cu128-24.04` 2. PyTorch container image `dustynv/pytorch:2.6-r36.4.0-cu128-24.04` --- # SenseVoice ASR (Speech-to-Text) `SenseVoice ASR` is an ASR (speech-to-text) service packaged based on `SenseVoice.cpp`, running within the AI Pod. The service domain name inside the AI Pod is `asr-ai`. It provides an OpenAI Audio API-style interface for speech-to-text transcription, enabling backend services, frontend applications, or scripts to upload audio files via HTTP and retrieve transcribed text. ## Service Endpoint - API Documentation: - Service Address: - Health Check: - Transcription Endpoint: Access format after deployment in the AI Pod: - Service Address:`https://asr-ai.${LAZYCAT_BOX_DOMAIN}` - API Documentation: `https://asr-ai.${LAZYCAT_BOX_DOMAIN}/docs` - Health Check: `https://asr-ai.${LAZYCAT_BOX_DOMAIN}/health` - Transcription Interface: `https://asr-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/transcriptions` ## Health Check ### `GET /health` This endpoint is used to verify the availability of the ASR HTTP service. Request: ```bash curl https://asr-ai.${LAZYCAT_BOX_DOMAIN}/health ``` Response example: ```json { "status": "ok" } ``` ## Transcription Interface ### `POST /v1/audio/transcriptions` This interface accepts an audio file and returns transcription results. The request must use `multipart/form-data`. The service first uses `ffmpeg` to convert the uploaded audio into a `16 kHz`, single-channel `wav`, then performs SenseVoice inference. Therefore, the actual supported audio formats depend on the container image's `ffmpeg` decoding capabilities. Common formats such as `wav`, `mp3`, `m4a`, `mp4`, `webm`, `flac`, and `ogg` can typically be uploaded directly. | Parameters | Type | Required | Description | |---|---|---|---| | `file` | file | Yes | Pending audio file to be transcribed | | `language` | string | no | Voice language, default `auto`. Supports `auto` / `zh` / `en` / `ja` / `ko` / `yue`, invalid values are handled as `auto`. | | `response_format` | string | no | Default `json`. Pass `text` returns plain text; pass any other value returns JSON | | `token_timestamps` | string | no | When transmitting `true`, the segmented result contains `words` timestamp information. | | `split_on_word` | string | no | When transmitting `true`, it also includes `words` timestamp information. | The current service does not provide `srt` / `vtt` subtitle output, nor does it offer `progress` / `progress_format` progress event stream. When subtitles are required, it is recommended to generate them using the JSON field `segments`. | `response_format` | Content-Type | Description | |---|---|---| | `json` | `application/json` | Default format, including `text` and `segments` | | `verbose_json` | `application/json` | Compatible with common invocation patterns, returns the same type of JSON structure in practice. | | `text` | `text/plain; charset=utf-8` | Only return the transcribed text. | JSON response example: ```json { "text": "欢迎使用算力舱语音转写服务。", "segments": [ { "id": 0, "start": 0.0, "end": 2.4, "text": "欢迎使用算力舱语音转写服务。" } ] } ``` After enabling `token_timestamps=true` or `split_on_word=true`, `segments` will additionally include `words`: ```json { "id": 0, "start": 0.0, "end": 2.4, "text": "欢迎使用算力舱语音转写服务。", "words": [ { "word": "欢迎", "text": "欢迎", "start": 0.12, "end": 0.42, "probability": 0.98 } ] } ``` Error response example: ```json { "error": "multipart field 'file' is required" } ``` Common Errors: | HTTP status codes | Scenario | |---|---| | `400` | Not used `multipart/form-data`, missing `file` field, or `ffmpeg` cannot convert uploaded audio | | `500` | Failed to write temporary file | ## Call Examples ### JSON Transcription ```bash curl -fsS -X POST "https://asr-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/transcriptions" \ -F "file=@/path/to/audio.wav" \ -F "language=auto" \ -F "response_format=json" ``` ### Plain text transcription ```bash curl -fsS -X POST "https://asr-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/transcriptions" \ -F "file=@/path/to/audio.wav" \ -F "response_format=text" ``` ### Timestamped Transcription ```bash curl -fsS -X POST "https://asr-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/transcriptions" \ -F "file=@/path/to/audio.wav" \ -F "language=zh" \ -F "token_timestamps=true" ``` ## Recommended Calling Flow 1. Prepare audio files. For video files, it is recommended to extract the audio first before uploading. 2. Upload `multipart/form-data` to `/v1/audio/transcriptions`. 3. `language` Unknown transmission `auto`; known language transmission `zh` / `en` / `ja` / `ko` / `yue`. 4. When processing segmented data, use default JSON; when only displaying text, use `response_format=text`. 5. When word-level timestamps are required, pass `token_timestamps=true` or `split_on_word=true`. ## Backend Environment Variables In business services dependent on ASR, the ASR base URL can be configured as: ```yaml services: convert: environment: - ASR_URL=https://asr-ai.${LAZYCAT_BOX_DOMAIN} - HOMEDIR=/lzcapp/run/mnt/home ``` - `ASR_URL`: ASR service base address, business code concatenated with `/v1/audio/transcriptions`. - `HOMEDIR`: root path for business file read/write, configured according to application requirements. Frontend matches `asr-ai` domain. ```ts const asrUrl = (path = "/v1/audio/transcriptions") => { const currentHost = window.location.host; const protocol = window.location.protocol; const host = currentHost.replace(/^([^.]+)/, "asr-ai"); return `${protocol}//${host}${path}`; }; ``` ## Independent Deployment Usage Select the corresponding image based on the device. The AGX Orin uses image `-agxorin`, and the Thor T5000 uses image `-thor`. ### AGX Orin ```yml services: asr: image: registry.lazycat.cloud/catdogai/lzc-aipod-sensevoiceasr:0.2.0-sensevoice-small-q4k-agxorin ports: - 3000:3000 environment: - PORT=3000 - SENSEVOICE_MODEL=/opt/sensevoice-models/sense-voice-small-q4_k.gguf - SENSEVOICE_LANGUAGE=auto - SENSEVOICE_THREADS=4 - SENSEVOICE_USE_ITN=true - OMP_NUM_THREADS=1 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 10s timeout: 30s retries: 100 start_period: 30s ``` ### Thor T5000 ```yml services: asr: image: registry.lazycat.cloud/catdogai/lzc-aipod-sensevoiceasr:0.2.0-sensevoice-small-q4k-thor ports: - 3000:3000 environment: - PORT=3000 - SENSEVOICE_MODEL=/opt/sensevoice-models/sense-voice-small-q4_k.gguf - SENSEVOICE_LANGUAGE=auto - SENSEVOICE_THREADS=4 - SENSEVOICE_USE_ITN=true - OMP_NUM_THREADS=1 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 10s timeout: 30s retries: 100 start_period: 30s ``` Boot: ```bash docker compose up -d curl http://127.0.0.1:3000/health ``` After startup, you can access `http://127.0.0.1:3000/docs` to view the built-in API documentation of the service. ## Service Environment Variables | Variable | Default value | Description | |---|---|---| | `PORT` | `3000` | HTTP listening port | | `SENSEVOICE_MODEL` | `/opt/sensevoice-models/sense-voice-small-q4_k.gguf` | SenseVoice GGUF model path | | `SENSEVOICE_LANGUAGE` | `auto` | Default recognized language | | `SENSEVOICE_THREADS` | `4` | Inference thread count | | `SENSEVOICE_USE_GPU` | `true` | Whether to enable GPU | | `SENSEVOICE_USE_ITN` | `true` | Whether to enable reverse text regularization, including punctuation processing | | `SENSEVOICE_USE_PREFIX` | `false` | Whether to output SenseVoice prefix information | | `OMP_NUM_THREADS` | `1` | OpenMP thread count | ## Related Services - SenseVoice ASR is enabled by default with `SENSEVOICE_USE_ITN=true`, which performs text normalization and punctuation processing. --- # bge-m3 ### vllm ``` bge-m3-vllm-1 | INFO: Application startup complete. bge-m3-vllm-1 | INFO 08-20 03:40:32 [metrics.py:481] Avg prompt throughput: 3698.0 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 953 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. bge-m3-vllm-1 | INFO 08-20 03:40:38 [metrics.py:481] Avg prompt throughput: 28348.0 tokens/s, Avg generation throughput: 44.4 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 698 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. bge-m3-vllm-1 | INFO 08-20 03:40:43 [metrics.py:481] Avg prompt throughput: 29275.2 tokens/s, Avg generation throughput: 46.4 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 441 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. bge-m3-vllm-1 | INFO 08-20 03:40:49 [metrics.py:481] Avg prompt throughput: 29540.3 tokens/s, Avg generation throughput: 46.1 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 188 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. bge-m3-vllm-1 | INFO: 172.20.0.4:45788 - "POST /v1/embeddings HTTP/1.1" 200 OK ``` 250+ concurrent requests, output token rate reaches over 29,540 tokens/second ``` python3 ollama_perf.py --file GuiMiZhiZhuu.txt --url https://bgem3vllm-ai.13gxg.heiyu.space/v1/embeddings --model bge-m3 --- 测试参数 --- 文件路径: GuiMiZhiZhuu.txt 模型名称: bge-m3 API URL: https://bgem3vllm-ai.13gxg.heiyu.space/v1/embeddings 切片长度: 1024 字符 总字数: 4775359 总切片数: 4664 注意: 所有切片将在一个 API 请求中发送。 ------------------ 开始发送单个大请求，请稍候... 请求完成！ --- 性能测试结果 --- 总耗时: 132.80 秒 总字数: 4775359 总处理 Tokens: 3384888 请求状态: 成功 (HTTP 200) --- 核心指标 --- 平均 Token 速度: 25489.00 tokens/秒 平均吞吐量: 35959.57 字/秒 -------------------- ``` ### ollama ``` $ python3 ollama_perf.py --file GuiMiZhiZhuu.txt --url https://ollama-ai.13gxg.heiyu.space/v1/embeddings --model bge-m3:latest --- 测试参数 --- 文件路径: GuiMiZhiZhuu.txt 模型名称: bge-m3:latest API URL: https://ollama-ai.13gxg.heiyu.space/v1/embeddings 切片长度: 1024 字符 总字数: 4775359 总切片数: 4664 注意: 所有切片将在一个 API 请求中发送。 ------------------ 开始发送单个大请求，请稍候... 请求完成！ --- 性能测试结果 --- 总耗时: 529.10 秒 总字数: 4775359 总处理 Tokens: 3375250 请求状态: 成功 (HTTP 200) --- 核心指标 --- 平均 Token 速度: 6379.19 tokens/秒 平均吞吐量: 9025.38 字/秒 -------------------- ``` Small file ``` python3 ollama_perf.py --file xce --url https://ollama-ai.13gxg.heiyu.space/v1/embeddings --model bge-m3:latest --- 测试参数 --- 文件路径: xce 模型名称: bge-m3:latest API URL: https://ollama-ai.13gxg.heiyu.space/v1/embeddings 切片长度: 1024 字符 总字数: 41648 总切片数: 41 注意: 所有切片将在一个 API 请求中发送。 ------------------ 开始发送单个大请求，请稍候... 请求完成！ --- 性能测试结果 --- 总耗时: 4.95 秒 总字数: 41648 总处理 Tokens: 29539 请求状态: 成功 (HTTP 200) --- 核心指标 --- 平均 Token 速度: 5968.35 tokens/秒 平均吞吐量: 8414.97 字/秒 -------------------- ``` --- # DeepSeek R1 ## Ollama deepseek-r1:70b The `deepseek-r1:70b` model image in Ollama is `43GB`, and it occupies approximately `46GB` of memory after loading into RAM. ```text deepseek-r1:70b d37b54d01a76 46 GB 100% GPU 8192 4 minutes from now total duration: 7m10.425334424s load duration: 56.463007ms prompt eval count: 25 token(s) prompt eval duration: 44.639105ms prompt eval rate: 560.05 tokens/s eval count: 1480 token(s) eval duration: 7m10.322503313s eval rate: 3.44 tokens/s ``` ## llama.cpp Benchmark The following data comes from `~/lzc-aipod-benchmark/deepseek-r1-distill-70b/cumulative-report.md`, corresponding to the model `DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf`, with runtime `llama.cpp`. This section is the `Distill 70B` `GGUF` benchmark, not the above `ollama deepseek-r1:70b` pull log. | Device | Runtime | Weight | Single Stream | Peak Aggregate | Power | GPU Temp | |:-------|:--------|:-------|:--------------|:---------------|:------|:---------| | Jetson AGX Orin | `llama.cpp` | `GGUF Q4_K_M` | `3.346 tok/s` (`g64 / c1`) | `33.851 tok/s` (`g64 / c16`) | `50.32~68.787 W` | `89.5 C` | | Thor T4000 | `llama.cpp` | `GGUF Q4_K_M` | `2.909~2.912 tok/s` (`g64~g1024 / c1`) | `46.165 tok/s` (`g64 / c48`) | `22.066~22.295 W` for `c1`; `37.798 W` at peak | `52~53 C` for `c1`; `48.562 C` at peak | | Thor | `llama.cpp` | `GGUF Q4_K_M` | `4.392~4.415 tok/s` (`g64~g1024 / c1`) | `-` | `37.443~39.254 W` | `68~75 C` | | Thor T5000 | `llama.cpp` | `GGUF Q4_K_M` | `4.458~4.552 tok/s` (`g64~g1024 / c1`) | `-` | `37.699~38.937 W` | `61~63 C` | ## Notes - `Jetson AGX Orin` corresponds to the test machine `lzc-pod-juyIZt` in the benchmark repository. - `THOR` and `T5000` have completed the single-stream steady-state retest of `g64,128,256,512,1024 / c1 / 300s` in this round. - High-concurrency peak performance results are currently only available for `T4000` and `Jetson AGX Orin`; therefore, peak aggregate throughput for `THOR` and `T5000` is pending. - Data source: `/home/catdog/lzc-aipod-benchmark/deepseek-r1-distill-70b/cumulative-report.md` --- # GPT-OSS 120B ## Benchmark The following data comes from `~/lzc-aipod-benchmark/gpt-oss-120b-moe/cumulative-report.md`, corresponding to model `gpt-oss-120b-mxfp4`, runtime `llama.cpp`, and weight format `GGUF`. | Device | Runtime | Weight | Single Stream | Peak Aggregate | Power | GPU Temp | |:-------|:--------|:-------|:--------------|:---------------|:------|:---------| | Thor | `llama.cpp` | `GGUF mxfp4` | `34.615 tok/s` (`g512 / c1`) | `178.536 tok/s` (`g64 / c16`) | `31.793 W` | `67 C` | | Thor T5000 | `llama.cpp` | `GGUF mxfp4` | `35.088 tok/s` (`g1024 / c1`) | `330.602 tok/s` (`g64 / c48`) | `31.261 W` for latest `c1`; `54.419 W` at `c48` | `56 C` for latest `c1`; `57.593 C` at `c48` | | Jetson AGX Orin | `-` | `-` | `no data` | `no data` | `-` | `-` | | Thor T4000 | `-` | `-` | `no data` | `no data` | `-` | `-` | ## Notes - `THOR` The current best single-stream result comes from the generation length matrix of `g128 / g256 / g512 / g1024`, where `g512 / c1` is `34.615 tok/s`. - `T5000` The current default single-path approach is fixed to `g1024 / c1 / server_parallel=1 / 300s`, with the latest retesting result being `35.088 tok/s`. - `T5000`'s high-concurrency peak comes from `c40 / c48` extended scanning, where `g64 / c48` reaches `330.602 tok/s`. - In the current benchmark repository, `Orin` and `T4000` lack the official `gpt-oss-120b` results, so only `no data` is retained. - Data source: `/home/catdog/lzc-aipod-benchmark/gpt-oss-120b-moe/cumulative-report.md` --- # gpt-oss | Device | Name | Ctx | Memory | Token | |--------|---------------------|------|--------|-------| | Thor | gpt-oss-120b-Q4_K_M | 128k | 64.6G | | | | | | | | | | | | | | --- # Benchmark - [Whisper.cpp](./whispercpp.md) - [Qwen3.5](./qwen3.5.md) - [SD.CPP](./sdcpp.md) - [Qwen3 Coder Next](./qwen3-coder-next.md) - [Qwen3 8B Embedding](./qwen3-8b-embedding.md) - [Qwen3 Embedding 0.6B on Orin](./qwen3-embedding-0.6b-vllm-orin.md) - [Qwen3 Embedding 0.6B on Thor](./qwen3-embedding-0.6b-vllm-thor.md) - [Qwen3 VL Embedding 2B on Orin](./qwen3-vl-embedding-2b-vllm-orin.md) - [Qwen3 VL Embedding 2B on Thor](./qwen3-vl-embedding-2b-vllm-thor.md) - [PaddleOCR-VL-1.5](./paddleocr-vl-1.5.md) - [BGE-M3](./bgem3.md) - [Llama 3.1 70B](./llama-3.1-70b.md) - [DeepSeek R1](./deepseek-r1.md) - [GPT-OSS 120B](./gpt-oss-120b.md) ## Embedding Model ![embedding model](../../benchmark/embedding-model.png) --- # llama.cpp build: 716301d1 (5757) ## Meta-Llama-3.1-70B-Instruct-IQ4_XS.gguf ``` ./build/bin/llama-batched-bench -ngl 999 -m ../Meta-Llama-3.1-70B-Instruct-IQ4_XS.gguf -fa -npl 1,2,4,6,8,10,12 -npp 512 -ntg 128 -c 16384 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: Orin, compute capability 8.7, VMM: yes build: 5757 (716301d1) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for aarch64-linux-gnu llama_model_load_from_file_impl: using device CUDA0 (Orin) - 58814 MiB free llama_model_loader: loaded meta data with 33 key-value pairs and 724 tensors from ../Meta-Llama-3.1-70B-Instruct-IQ4_XS.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Meta Llama 3.1 70B Instruct llama_model_loader: - kv 3: general.finetune str = Instruct llama_model_loader: - kv 4: general.basename str = Meta-Llama-3.1 llama_model_loader: - kv 5: general.size_label str = 70B llama_model_loader: - kv 6: general.license str = llama3.1 llama_model_loader: - kv 7: general.tags arr[str,6] = ["facebook", "meta", "pytorch", "llam... llama_model_loader: - kv 8: general.languages arr[str,8] = ["en", "de", "fr", "it", "pt", "hi", ... llama_model_loader: - kv 9: llama.block_count u32 = 80 llama_model_loader: - kv 10: llama.context_length u32 = 131072 llama_model_loader: - kv 11: llama.embedding_length u32 = 8192 llama_model_loader: - kv 12: llama.feed_forward_length u32 = 28672 llama_model_loader: - kv 13: llama.attention.head_count u32 = 64 llama_model_loader: - kv 14: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 15: llama.rope.freq_base f32 = 500000.000000 llama_model_loader: - kv 16: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 17: general.file_type u32 = 30 llama_model_loader: - kv 18: llama.vocab_size u32 = 128256 llama_model_loader: - kv 19: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 20: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 21: tokenizer.ggml.pre str = llama-bpe llama_model_loader: - kv 22: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 23: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 24: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 25: tokenizer.ggml.bos_token_id u32 = 128000 llama_model_loader: - kv 26: tokenizer.ggml.eos_token_id u32 = 128009 llama_model_loader: - kv 27: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ... llama_model_loader: - kv 28: general.quantization_version u32 = 2 llama_model_loader: - kv 29: quantize.imatrix.file str = /models_out/Meta-Llama-3.1-70B-Instru... llama_model_loader: - kv 30: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt llama_model_loader: - kv 31: quantize.imatrix.entries_count i32 = 560 llama_model_loader: - kv 32: quantize.imatrix.chunks_count i32 = 125 llama_model_loader: - type f32: 162 tensors llama_model_loader: - type q5_K: 80 tensors llama_model_loader: - type q6_K: 1 tensors llama_model_loader: - type iq4_xs: 481 tensors print_info: file format = GGUF V3 (latest) print_info: file type = IQ4_XS - 4.25 bpw print_info: file size = 35.29 GiB (4.30 BPW) load: special tokens cache size = 256 load: token to piece cache size = 0.7999 MB print_info: arch = llama print_info: vocab_only = 0 print_info: n_ctx_train = 131072 print_info: n_embd = 8192 print_info: n_layer = 80 print_info: n_head = 64 print_info: n_head_kv = 8 print_info: n_rot = 128 print_info: n_swa = 0 print_info: is_swa_any = 0 print_info: n_embd_head_k = 128 print_info: n_embd_head_v = 128 print_info: n_gqa = 8 print_info: n_embd_k_gqa = 1024 print_info: n_embd_v_gqa = 1024 print_info: f_norm_eps = 0.0e+00 print_info: f_norm_rms_eps = 1.0e-05 print_info: f_clamp_kqv = 0.0e+00 print_info: f_max_alibi_bias = 0.0e+00 print_info: f_logit_scale = 0.0e+00 print_info: f_attn_scale = 0.0e+00 print_info: n_ff = 28672 print_info: n_expert = 0 print_info: n_expert_used = 0 print_info: causal attn = 1 print_info: pooling type = 0 print_info: rope type = 0 print_info: rope scaling = linear print_info: freq_base_train = 500000.0 print_info: freq_scale_train = 1 print_info: n_ctx_orig_yarn = 131072 print_info: rope_finetuned = unknown print_info: ssm_d_conv = 0 print_info: ssm_d_inner = 0 print_info: ssm_d_state = 0 print_info: ssm_dt_rank = 0 print_info: ssm_dt_b_c_rms = 0 print_info: model type = 70B print_info: model params = 70.55 B print_info: general.name = Meta Llama 3.1 70B Instruct print_info: vocab type = BPE print_info: n_vocab = 128256 print_info: n_merges = 280147 print_info: BOS token = 128000 '<|begin_of_text|>' print_info: EOS token = 128009 '<|eot_id|>' print_info: EOT token = 128009 '<|eot_id|>' print_info: EOM token = 128008 '<|eom_id|>' print_info: LF token = 198 'Ċ' print_info: EOG token = 128001 '<|end_of_text|>' print_info: EOG token = 128008 '<|eom_id|>' print_info: EOG token = 128009 '<|eot_id|>' print_info: max token length = 256 load_tensors: loading model tensors, this can take a while... (mmap = true) load_tensors: offloading 80 repeating layers to GPU load_tensors: offloading output layer to GPU load_tensors: offloaded 81/81 layers to GPU load_tensors: CUDA0 model buffer size = 35606.98 MiB load_tensors: CPU_Mapped model buffer size = 532.31 MiB .................................................................................................. llama_context: constructing llama_context llama_context: n_seq_max = 12 llama_context: n_ctx = 16384 llama_context: n_ctx_per_seq = 1365 llama_context: n_batch = 2048 llama_context: n_ubatch = 512 llama_context: causal_attn = 1 llama_context: flash_attn = 1 llama_context: freq_base = 500000.0 llama_context: freq_scale = 1 llama_context: n_ctx_per_seq (1365) < n_ctx_train (131072) -- the full capacity of the model will not be utilized llama_context: CUDA_Host output buffer size = 5.87 MiB llama_kv_cache_unified: CUDA0 KV buffer size = 5120.00 MiB llama_kv_cache_unified: size = 5120.00 MiB ( 16384 cells, 80 layers, 12 seqs), K (f16): 2560.00 MiB, V (f16): 2560.00 MiB llama_context: CUDA0 compute buffer size = 266.50 MiB llama_context: CUDA_Host compute buffer size = 52.13 MiB llama_context: graph nodes = 2567 llama_context: graph splits = 2 main: n_kv_max = 16384, n_batch = 2048, n_ubatch = 512, flash_attn = 1, is_pp_shared = 0, n_gpu_layers = 999, n_threads = 12, n_threads_batch = 12 | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 14.184 | 36.10 | 42.285 | 3.03 | 56.468 | 11.33 | | 512 | 128 | 2 | 1280 | 8.941 | 114.53 | 50.120 | 5.11 | 59.060 | 21.67 | | 512 | 128 | 4 | 2560 | 17.983 | 113.88 | 71.604 | 7.15 | 89.587 | 28.58 | | 512 | 128 | 6 | 3840 | 27.168 | 113.07 | 81.013 | 9.48 | 108.181 | 35.50 | | 512 | 128 | 8 | 5120 | 36.480 | 112.28 | 93.166 | 10.99 | 129.646 | 39.49 | | 512 | 128 | 10 | 6400 | 45.901 | 111.54 | 95.976 | 13.34 | 141.877 | 45.11 | | 512 | 128 | 12 | 7680 | 55.496 | 110.71 | 78.255 | 19.63 | 133.751 | 57.42 | llama_perf_context_print: load time = 148042.28 ms llama_perf_context_print: prompt eval time = 718462.74 ms / 27408 tokens ( 26.21 ms per token, 38.15 tokens per second) llama_perf_context_print: eval time = 42280.79 ms / 128 runs ( 330.32 ms per token, 3.03 tokens per second) llama_perf_context_print: total time = 866615.25 ms / 27536 tokens ``` ``` ./build/bin/llama-bench -m ../Meta-Llama-3.1-70B-Instruct-IQ4_XS.gguf -ngl 999 -fa 1 -n 0 -p 64,128,256,512 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: Orin, compute capability 8.7, VMM: yes | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: | | llama 70B IQ4_XS - 4.25 bpw | 35.29 GiB | 70.55 B | CUDA | 999 | 1 | pp64 | 71.54 ± 0.06 | | llama 70B IQ4_XS - 4.25 bpw | 35.29 GiB | 70.55 B | CUDA | 999 | 1 | pp128 | 109.25 ± 0.26 | | llama 70B IQ4_XS - 4.25 bpw | 35.29 GiB | 70.55 B | CUDA | 999 | 1 | pp256 | 91.70 ± 46.86 | | llama 70B IQ4_XS - 4.25 bpw | 35.29 GiB | 70.55 B | CUDA | 999 | 1 | pp512 | 113.74 ± 0.19 | build: 716301d1 (5757) ``` --- # PaddleOCR-VL-1.5 ## Benchmark | Model | Framework | Scene | TTFT | Token/s | Sample | Notes | |:------|:----------|:------|:-----|:--------|:-------|:------| | PaddleOCR-VL-1.5 (f16) | llama.cpp | OCR / ReadingGeneration | 310 ms | 94 | - | Can recognize complex layouts, but may exhibit hallucinations when encountering large blocks of repeated text. | | PaddleOCR-VL-1.5-GGUF-Q4_K_M.gguf | llama.cpp | ReadingGeneration | - | 199.66 | 307 tokens / 1.5 s | More prone to hallucinations compared to f16 | | PaddleOCR-VL-1.5-GGUF-Q8_0.gguf | llama.cpp | ReadingGeneration | - | 176.82 | 2012 tokens / 11 s | Balanced speed and stability; recommended for use with this precision. | ## Notes - This model primarily provides pure OCR capabilities. - To achieve layout analysis and Markdown output, it must be used in conjunction with other models. - Container image: `registry.lazycat.cloud/x/lzc-aipod-paddleocr-vl:2017cd2` --- # qwen3-embedding-8b ### ollama ``` python3 ollama_perf.py --file GuiMiZhiZhuu.txt --url https://ollama-ai.13gxg.heiyu.space/v1/embeddings --model dengcao/Qwen3-Embedding-8B:Q8_0 --- 测试参数 --- 文件路径: GuiMiZhiZhuu.txt 模型名称: dengcao/Qwen3-Embedding-8B:Q8_0 API URL: https://ollama-ai.13gxg.heiyu.space/v1/embeddings 切片长度: 1024 字符 总字数: 4775359 总切片数: 4664 注意: 所有切片将在一个 API 请求中发送。 ------------------ 开始发送单个大请求，请稍候... 请求完成！ --- 性能测试结果 --- 总耗时: 3763.11 秒 总字数: 4775359 总处理 Tokens: 3325333 请求状态: 成功 (HTTP 200) --- 核心指标 --- 平均 Token 速度: 883.67 tokens/秒 平均吞吐量: 1268.99 字/秒 -------------------- ``` Small file ``` $ python3 ollama_perf.py --file xce --url https://ollama-ai.13gxg.heiyu.space/v1/embeddings --model dengcao/Qwen3-Embedding-8B:Q5_K_M --- 测试参数 --- 文件路径: xce 模型名称: dengcao/Qwen3-Embedding-8B:Q5_K_M API URL: https://ollama-ai.13gxg.heiyu.space/v1/embeddings 切片长度: 1024 字符 总字数: 41648 总切片数: 41 注意: 所有切片将在一个 API 请求中发送。 ------------------ 开始发送单个大请求，请稍候... 请求完成！ --- 性能测试结果 --- 总耗时: 34.52 秒 总字数: 41648 总处理 Tokens: 29006 请求状态: 成功 (HTTP 200) --- 核心指标 --- 平均 Token 速度: 840.33 tokens/秒 平均吞吐量: 1206.59 字/秒 -------------------- ``` ### vllm Default is the official Qwen/Qwen3-Embedding-8B bfloat16 model type ``` $ python3 ollama_perf.py --file GuiMiZhiZhuu.txt --url http://192.168.1.209:8999/v1/embeddings --model qwen3-embedding-8b --- 测试参数 --- 文件路径: GuiMiZhiZhuu.txt 模型名称: qwen3-embedding-8b API URL: http://192.168.1.209:8999/v1/embeddings 切片长度: 1024 字符 总字数: 4775359 总切片数: 4664 注意: 所有切片将在一个 API 请求中发送。 ------------------ 开始发送单个大请求，请稍候... 请求完成！ --- 性能测试结果 --- 总耗时: 2312.80 秒 总字数: 4775359 总处理 Tokens: 3329997 请求状态: 成功 (HTTP 200) --- 核心指标 --- 平均 Token 速度: 1439.81 tokens/秒 平均吞吐量: 2064.75 字/秒 -------------------- ``` Memory usage required: 25 GB --- # Qwen3 Coder Next | Model | Name | Ctx | Memory | Token/s | |:----------------|:---------------------------------|:-----|--------|:--------| | Thor | Qwen3-Coder-Next-UD-Q4_K_XL.gguf | 256k | 55G | 22.28 | | Jetson agx orin | Qwen3-Coder-Next-UD-Q4_K_XL.gguf | 256k | 58.3G | 17.88 | | | | | | | ::: info Info On Jetson AGX Orin 64GB, the lzc-earlyoom service must be stopped first; otherwise, it will trigger service termination. `sudo systemctl stop lzc-earlyoom` --- # Qwen3 Embedding 0.6B on Orin Tested `vllm` on `orin` machine using `Qwen/Qwen3-Embedding-0.6B`. ## Test Configuration - Machine: `orin` - Container image: `registry.lazycat.cloud/x/lzc-aipod-vllm:bffa39b-orin` - Model: `Qwen/Qwen3-Embedding-0.6B` - Deployment mode: `vllm serve --runner pooling` - GPU memory: `--gpu-memory-utilization 0.65 --max-model-len 32768 --enforce-eager` - Dataset: `datasets/qwen3_embedding_texts.jsonl` - Concurrency: `1,2,4,6,8,16` - Batch size: `1,8,32,128` ## Conclusion - OpenAI Embeddings-compatible API is available; both `/v1/models` and `/v1/embeddings` pass. - Output vector dimension is `1024`. - Under small batch, low-latency mode, `batch=1 concurrency=16` achieves peak `QPS=99.7079`, average latency of `0.1465s`, with `p95=0.1781s`. - In large batch throughput mode, `batch=128 concurrency=8` reaches its peak of `items/s=706.4871`, `prompt tokens/s=11868.5`, with an average latency of `1.2431s`, `p95=1.5489s`. ## Key Results | batch | concurrency | items/s | prompt tokens/s | avg latency (s) | p95 (s) | | --- | --- | ---: | ---: | ---: | ---: | | 1 | 16 | 99.7079 | 1723.0776 | 0.1465 | 0.1781 | | 8 | 8 | 170.4528 | 2883.0500 | 0.3544 | 0.3940 | | 32 | 8 | 522.1799 | 8784.2497 | 0.4447 | 0.5071 | | 128 | 8 | 706.4871 | 11868.5000 | 1.2431 | 1.5489 | ## Telemetry - Sample count: `73` - Maximum temperature: `58.937 C` - Maximum power consumption: `41189 mW` - Maximum fan PWM: `20.7843` - Maximum GPU utilization: `99.9%` ## Outputs Remote results directory: ```bash /home/nvidia/bench-results/qwen3-embedding-vllm ``` Key files: ```bash /home/nvidia/bench-results/qwen3-embedding-vllm/qwen3_embedding_bench.json /home/nvidia/bench-results/qwen3-embedding-vllm/models.json /home/nvidia/bench-results/qwen3-embedding-vllm/server.log /home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.jsonl /home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.summary.json ``` --- # Qwen3 Embedding 0.6B on Thor Test the official `vllm` container image on the `thor` machine with `Qwen/Qwen3-Embedding-0.6B`. ## Test Configuration - Machine: `thor` - Container image: `nvcr.io/nvidia/vllm:26.01-py3` - Container internal `vllm` version: `0.13.0+faa43dbf.nv26.01` - Model: `Qwen/Qwen3-Embedding-0.6B` - Deployment mode: `vllm serve --runner pooling` - Start parameters: `--gpu-memory-utilization 0.65 --max-model-len 32768 --enforce-eager` - Dataset: `datasets/qwen3_embedding_texts.jsonl` - Concurrency: `1,2,4,6,8,16` - Batch size: `1,8,32,128` ## Conclusion - The official container image directly provides an OpenAI Embeddings-compatible interface on Thor. - Both `models` and `embeddings` requests pass, with output vector dimension of `1024`. - In low-latency mode, `batch=1 concurrency=16` reaches its peak of `QPS=207.9151`, with an average latency of `0.0688s`, and `p95=0.0851s`. - In throughput mode, `batch=128 concurrency=16` reaches its peak of `items/s=1371.0304`, `prompt tokens/s=23019.6541`, average latency of `1.1980s`, and `p95=1.5949s`. ## Key Results | batch | concurrency | items/s | prompt tokens/s | avg latency (s) | p95 (s) | | --- | --- | ---: | ---: | ---: | ---: | | 1 | 16 | 207.9151 | 3593.0330 | 0.0688 | 0.0851 | | 8 | 16 | 584.2202 | 9794.8176 | 0.1909 | 0.2693 | | 32 | 16 | 934.1475 | 15679.8117 | 0.4482 | 0.5799 | | 128 | 16 | 1371.0304 | 23019.6541 | 1.1980 | 1.5949 | ## Telemetry - Sample count: `32` - Maximum temperature: `34.437 C` - Maximum power consumption: `8102 mW` - Maximum fan PWM: `null` - Maximum GPU utilization: `null` ## Output Remote result directory: ```bash /home/nvidia/bench-results/qwen3-embedding-vllm ``` Key files: ```bash /home/nvidia/bench-results/qwen3-embedding-vllm/qwen3_embedding_bench.json /home/nvidia/bench-results/qwen3-embedding-vllm/models.json /home/nvidia/bench-results/qwen3-embedding-vllm/server.log /home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.jsonl /home/nvidia/bench-results/qwen3-embedding-vllm/telemetry.summary.json ``` --- # Qwen3 VL Embedding 2B on Orin Use `vllm` on the `orin` machine to test `Qwen/Qwen3-VL-Embedding-2B`, verifying both offline embedding and HTTP service startup. ## Test Configuration - Device: `orin` - Container image: `registry.lazycat.cloud/x/lzc-aipod-vllm:bffa39b-orin` - Model: `Qwen/Qwen3-VL-Embedding-2B` - Offline mode: `LLM(..., runner="pooling")` - HTTP mode: `vllm serve --runner pooling` - VRAM specifications: `--gpu-memory-utilization 0.65 --max-model-len 8192 --enforce-eager` - Dataset: `datasets/qwen3_vl_embedding.jsonl` - Synthesized resources: `scripts/prepare_qwen3_vl_assets.py` ## Conclusion - Offline `embed()` path passed, `16` image-text samples completed embedding. - Output vector dimension is `2048`. - Offline throughput is `2.4299 items/s`, total duration `6.5845s`. - Image-text retrieval example `4/4` top1 match correct. - HTTP service started successfully, `/v1/models` ready. ## Offline verification result | case | top1 | | --- | --- | | `case-jtop` | `positive-doc` | | `case-openwebui` | `positive-doc` | | `case-network` | `positive-doc` | | `case-embedding` | `positive-doc` | ## Telemetry - Sample count: `150` - Maximum temperature: `67.156 C` - Maximum power consumption: `56761 mW` - Maximum fan PWM: `35.2941` - Maximum GPU utilization: `99.8%` ## Output Remote result directory: ```bash /home/nvidia/bench-results/qwen3-vl-embedding-vllm ``` Key Files: ```bash /home/nvidia/bench-results/qwen3-vl-embedding-vllm/qwen3_vl_embedding_offline.json /home/nvidia/bench-results/qwen3-vl-embedding-vllm/qwen3_vl_http_probe.json /home/nvidia/bench-results/qwen3-vl-embedding-vllm/qwen3_vl_http.log /home/nvidia/bench-results/qwen3-vl-embedding-vllm/qwen3_vl_models.json /home/nvidia/bench-results/qwen3-vl-embedding-vllm/telemetry.jsonl /home/nvidia/bench-results/qwen3-vl-embedding-vllm/telemetry.summary.json ``` --- # Qwen3 VL Embedding 2B on Thor Use the official `vllm` container image on the `thor` machine to test `Qwen/Qwen3-VL-Embedding-2B`. The current container image combination cannot run successfully. ## Test Configuration - Device: `thor` - Container image: `nvcr.io/nvidia/vllm:26.01-py3` - Container `vllm` version: `0.13.0+faa43dbf.nv26.01` - Model: `Qwen/Qwen3-VL-Embedding-2B` - Try path: - Offline `LLM(..., runner="pooling")` - HTTP `vllm serve --runner pooling` - Try parameters: - `--gpu-memory-utilization 0.65 --max-model-len 8192 --enforce-eager` - `--gpu-memory-utilization 0.5 --max-model-len 8192 --enforce-eager` - `dtype=float16` also performed additional validation - Dataset: `datasets/qwen3_vl_embedding.jsonl` - Synthesized resource: `scripts/prepare_qwen3_vl_assets.py` ## Conclusion - The official container image can recognize the model and enter the initialization process of `runner=pooling`. - When `gpu-memory-utilization=0.65` is encountered, startup fails immediately due to insufficient idle GPU memory. - Reduced GPU memory usage to `0.5` and proceeded to the visual encoding profile stage, but initialization still failed. - The root cause is not in the script parameters, but in the current official container image's `vllm + torch` combination: the visual path in `Qwen3-VL-Embedding-2B` reaches `torch.nn.functional.conv3d` and triggers `RuntimeError: GET was unable to find an engine to execute this computation`. - `float16` further validation yielded the same result, confirming this is not an isolated issue with the `bfloat16` path. ## Failure Summary - `0.65` configuration error: - `Free memory on device (32.52/59.9 GiB) on startup is less than desired GPU memory utilization (0.65, 38.94 GiB).` - `0.5` and `float16` configuration errors: - `RuntimeError: GET was unable to find an engine to execute this computation` - Triggered at the `qwen3_vl.py` visual encoding path's `F.conv3d(...)` stage ## Telemetry - Sample count: `22` - Maximum temperature: `32.593 C` - Maximum power consumption: `6887 mW` - Maximum fan PWM: `null` - Maximum GPU utilization: `null` ## Artifacts Remote result directory: ```bash /home/nvidia/bench-results/qwen3-vl-embedding-vllm ``` Only telemetry files have been generated; offline results and HTTP probe are not available. ```bash /home/nvidia/bench-results/qwen3-vl-embedding-vllm/telemetry.jsonl /home/nvidia/bench-results/qwen3-vl-embedding-vllm/telemetry.summary.json ``` --- # Qwen3.5 ## Benchmark The following three-machine comparison is based on the full formal matrix of `lzc_thor` running on `2026-03-20T05-47-33Z`, retaining only the complete results from correct usage, without including experimental single-request data such as `p1`, `cublas`, or `mmq`. | Model | Best Parallel | Orin | Thor T4000 | Thor T5000 | |:------|:--------------|:-----|:-----------|:-----------| | `Qwen3.5-0.8B-Q4_K_M` | `16` | `1326.86 tok/s` / `TTFT 1.94 s` / `42.54 W` | `1311.50 tok/s` / `TTFT 1.88 s` / `21.72 W` | `1783.26 tok/s` / `TTFT 1.37 s` / `30.35 W` | | `Qwen3.5-2B-Q4_K_M` | `16` | `989.57 tok/s` / `TTFT 2.91 s` / `46.43 W` | `960.84 tok/s` / `TTFT 3.01 s` / `22.46 W` | `1291.47 tok/s` / `TTFT 2.17 s` / `32.93 W` | | `Qwen3.5-4B-Q4_K_M` | `16` | `431.96 tok/s` / `TTFT 7.16 s` / `48.48 W` | `442.65 tok/s` / `TTFT 7.07 s` / `23.95 W` | `622.09 tok/s` / `TTFT 4.87 s` / `35.87 W` | | `Qwen3.5-9B-Q4_K_M` | `16` | `318.65 tok/s` / `TTFT 10.85 s` / `56.49 W` | `314.55 tok/s` / `TTFT 11.23 s` / `24.60 W` | `450.86 tok/s` / `TTFT 7.71 s` / `35.85 W` | | `Qwen3.5-27B-Q4_K_M` | `16` | `95.80 tok/s` / `TTFT 44.38 s` / `56.82 W` | `90.82 tok/s` / `TTFT 48.44 s` / `25.32 W` | `150.03 tok/s` / `TTFT 25.49 s` / `36.91 W` | | `Qwen3.5-35B-A3B-Q4_K_M` | `16` | `192.59 tok/s` / `TTFT 27.41 s` / `43.12 W` | `181.57 tok/s` / `TTFT 30.34 s` / `18.74 W` | `271.35 tok/s` / `TTFT 19.09 s` / `31.05 W` | | `Qwen3.5-122B-A10B-IQ2_S` | `16` | `65.14 tok/s` / `TTFT 46.02 s` / `42.48 W` | `60.92 tok/s` / `TTFT 50.95 s` / `19.49 W` | `76.55 tok/s` / `TTFT 43.03 s` / `26.25 W` | ## Latest T4000 SM110 Retest `2026-03-20T17-10-00Z`'s latest `sm110 maxn retest` has now completed `Thor T4000`, and is therefore listed separately to avoid mixing in incomplete T5000 / Orin data into the same comparison table. | Model | Best Parallel | Thor T4000 | |:------|:--------------|:-----------| | `Qwen3.5-0.8B-Q4_K_M` | `16` | `1313.47 tok/s` / `TTFT 1.88 s` / `21.73 W` | | `Qwen3.5-2B-Q4_K_M` | `16` | `960.84 tok/s` / `TTFT 3.01 s` / `22.84 W` | | `Qwen3.5-4B-Q4_K_M` | `16` | `442.74 tok/s` / `TTFT 7.07 s` / `23.95 W` | | `Qwen3.5-9B-Q4_K_M` | `16` | `314.46 tok/s` / `TTFT 11.23 s` / `24.60 W` | | `Qwen3.5-27B-Q4_K_M` | `16` | `91.00 tok/s` / `TTFT 48.40 s` / `25.33 W` | | `Qwen3.5-35B-A3B-Q4_K_M` | `16` | `181.36 tok/s` / `TTFT 30.35 s` / `18.74 W` | | `Qwen3.5-122B-A10B-IQ2_S` | `16` | `60.67 tok/s` / `TTFT 51.23 s` / `19.49 W` | ## Notes - Comparison data from three machines: `/home/catdog/lzc_thor/docs/runs/2026-03-20T05-47-33Z` - Source of the latest `sm110` retest: `/home/catdog/lzc_thor/docs/runs/2026-03-20T17-10-00Z` - `Qwen3.5-397B-A17B-IQ1_S` did not form valid benchmarks with `T4000` and `Orin` in the most recent test, thus it is not included in the official benchmark table. - Power consumption column records the maximum power observed during testing. --- # SD.CPP ## Benchmark The data is based on the official `lzc_thor` result from `2026-03-21T03-10-55Z`'s `speed_matrix`; only models with valid results from all three `Orin / Thor T4000 / Thor T5000` instances are retained.The boundary detection, single-service/double-service comparison, and smoke test cases are not included in this table. | Model | Best Case | Orin | Thor T4000 | Thor T5000 | |:------|:----------|:-----|:-----------|:-----------| | `FLUX.1-dev` | Based on the best case for each machine | `0.033 img/s` / `30.72 s` / `44.75 W` | `0.047 img/s` / `42.73 s` / `20.59 W` | `0.068 img/s` / `29.41 s` / `38.41 W` | | `SD2.1` | Based on the best case for each machine | `0.158 img/s` / `28.58 s` / `48.54 W` | `0.183 img/s` / `46.49 s` / `20.16 W` | `0.252 img/s` / `9.94 s` / `32.42 W` | | `SD3 Medium` | Based on the best case for each machine | `0.072 img/s` / `20.83 s` / `49.16 W` | `0.148 img/s` / `13.48 s` / `20.55 W` | `0.237 img/s` / `8.44 s` / `38.28 W` | | `SDXL` | Based on the best case for each machine | `0.030 img/s` / `67.45 s` / `48.28 W` | `0.040 img/s` / `50.10 s` / `20.93 W` | `0.048 img/s` / `41.34 s` / `34.31 W` | | `SSD-1B` | Based on the best case for each machine | `0.217 img/s` / `11.56 s` / `47.82 W` | `0.243 img/s` / `16.44 s` / `20.93 W` | `0.357 img/s` / `11.21 s` / `34.69 W` | ## Best Case | Model | Orin Best Case | Thor T4000 Best Case | Thor T5000 Best Case | |:------|:---------------|:---------------------|:---------------------| | `FLUX.1-dev` | `p1` | `batch_2` | `batch_2` | | `SD2.1` | `p8` | `p16` | `p4` | | `SD3 Medium` | `p2` | `batch_2` | `batch_2` | | `SDXL` | `batch_2` | `batch_2` | `batch_2` | | `SSD-1B` | `p4` | `batch_4` | `batch_4` | ## Notes - Data source: `/home/catdog/lzc_thor/docs/runs/2026-03-21T03-10-55Z` - The second number in the table represents the average latency of the best case. - The power consumption column records the maximum power during the test. - `FLUX.2-dev` and `FLUX.2-klein` did not produce stable and usable results in the latest test, and thus are not included in the official benchmark table. --- # Whisper.cpp ## Benchmark The following comparison uses the latest official matrix of `lzc_thor` from `2026-03-21`. The main table retains only models that have comparable results on all three platforms: `Orin / Thor T4000 / Thor T5000`. | Model | Scene | Orin | Thor T4000 | Thor T5000 | Notes | |:------|:------|:-----|:-----------|:-----------|:------| | `base.en` | `jfk`, concurrency=`16` | `3.39 req/s` / `P95 4.55 s` / `56.82 W` | `7.39 req/s` / `P95 2.10 s` / `11.65 W` | `9.44 req/s` / `P95 1.64 s` / `17.68 W` | All three devices can stably complete stress testing and are suitable as baseline for lightweight English language models. | | `small.en` | `jfk`, concurrency=`16` | `2.46 req/s` / `P95 6.28 s` / `26.88 W` | `4.07 req/s` / `P95 3.80 s` / `14.25 W` | `5.36 req/s` / `P95 2.89 s` / `21.76 W` | More balanced between speed and recognition quality compared to `base.en` | | `medium.en` | `jfk`, concurrency=`16` | `1.34 req/s` / `P95 11.52 s` / `35.50 W` | `1.77 req/s` / `P95 8.71 s` / `16.85 W` | `2.53 req/s` / `P95 6.11 s` / `26.94 W` | Advantages of the medium-sized model are more evident on Thor T5000 | ## More | Model | Scene | Thor T4000 | Thor T5000 | Notes | |:------|:------|:-----------|:-----------|:------| | `large-v3-turbo` | `jfk`, concurrency=`16` | `1.59 req/s` / `P95 9.70 s` / `20.93 W` | `2.52 req/s` / `P95 6.13 s` / `36.15 W` | - `2026-03-21` The Orin results in the latest matrix are still incomplete and have not fully been recorded. Retaining only the Thor-to-Thor comparison for now. | ## Notes - Data source: `/home/catdog/lzc_thor/docs/runs/2026-03-21T08-45-00Z` - The main table uses the official stress matrix results and does not include interrupted or unsynchronized cases. - The power consumption column records the maximum power during testing, not the average power. --- # BLIP Image Understanding Service `BLIP` is a service for image content understanding and description generation, suitable for image summarization, image-text retrieval preprocessing, and other scenarios. ## Service Information - Service address: - Health check address: - Container image: `registry.lazycat.cloud/x/videorag/blip:c218d42` ## Standalone Deployment 1. Save the following content into a `docker-compose.yml` file. ```yml services: blip: image: registry.lazycat.cloud/x/videorag/blip:c218d42 ports: - 3000:3000 environment: - AUTO_UNLOAD_INTERVAL=600 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 30s timeout: 30s retries: 30 start_period: 30s ``` 2. Run `docker-compose up -d` in the current directory to start 3. After startup, access `http://127.0.0.1:3000/health` to verify service health --- # CLIP Image-Text Encoding Service `CLIP` Service is used for unified vector encoding of text and images, suitable for cross-modal retrieval, image-text matching, and similarity computation. ## Service Information - Service address: - Health check address: - Container image: `registry.lazycat.cloud/x/videorag/clip:e5202e3` ## Standalone Deployment 1. Save the following content to a `docker-compose.yml` file: ```yml services: clip: image: registry.lazycat.cloud/x/videorag/clip:e5202e3 ports: - 3000:3000 environment: - AUTO_UNLOAD_INTERVAL=300 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 30s timeout: 30s retries: 30 start_period: 30s ``` 2. Run `docker-compose up -d` in the current directory to start 3. After startup, access `http://127.0.0.1:3000/health` to verify service health --- # ComfyUI After the `comfyui` 2.0 version, an overlay feature was added to enable environment persistence, allowing users to rollback to a previous version in case the environment is corrupted during upgrades or when installing certain nodes. ## Starting ComfyUI with Docker Compose The following example is suitable for Jetson AGX Orin (AI Pod X3), using the `catdogai/comfyui:2.0.3-agxorin` container image to start ComfyUI directly. Please ensure Docker and Docker Compose are installed on the device. Create a new directory and inside it, create a `docker-compose.yml` file, writing the following configuration: ```yaml services: app: image: catdogai/comfyui:2.0.3-agxorin privileged: true volumes: - /etc/timezone:/etc/timezone:ro - /run/jtop.sock:/run/jtop.sock - /sys/firmware/devicetree:/sys/firmware/devicetree:ro - /proc/device-tree:/proc/device-tree:ro - ./lzc-comfy-data:/lzc-comfy-data ports: - 8188:8188 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8188/api/system_stats"] interval: 5s timeout: 5s retries: 100 start_period: 30s ``` Execute in the directory of `docker-compose.yml`: ```bash docker compose up -d ``` After startup, access via the device's built-in browser. If accessing from another computer on the local network, open `http://算力舱IP:8188`. The models, input/output files, and runtime environment will be persisted in the `lzc-comfy-data` directory under the current working directory. Please preserve this directory. On first startup, the runtime environment needs to be prepared. You can check the startup logs and service status using the following commands: ```bash docker compose logs -f app docker compose ps ``` ## Overlay Runtime Usage The container image starts ComfyUI via `lzc-comfy-overlay run`. It synchronizes the immutable code and venv from within the container image to the persistent runtime layer, then launches the service from that runtime layer: - Container image code directory: `/opt/ComfyUI` - Container image venv: `/opt/venv` - Runtime code link: `/opt/ComfyUI-user` - Runtime venv link: `/opt/venv-user` - Default data root directory: `/lzc-comfy-data` - Overlay status directory: `/lzc-comfy-data/overlay` The system prioritizes OverlayFS and mounts: ```text /lzc-comfy-data/overlay/merged/app /lzc-comfy-data/overlay/merged/venv ``` If the current container or data disk does not support OverlayFS, the script will automatically downgrade to a regular persistent rsync operation mode. The downgrade mode allows ComfyUI to start up, but does not provide snapshot and rollback capabilities. The user data directory will be migrated and linked to: ```text /lzc-comfy-data/overlay/userdata/user /lzc-comfy-data/overlay/userdata/output /lzc-comfy-data/overlay/userdata/input /lzc-comfy-data/overlay/userdata/models ``` These directories will not be covered by container image upgrades. `lzc-comfyui.ignore` should only exclude the root directory's `/user/***`, `/output/***`, `/input/***`, `/models/***`, `/temp/***`. Do not use unanchored rules like `input/`, `models/`, as this may accidentally exclude subdirectories with the same name in ComfyUI's source code. Common commands: ```bash # 准备 overlay 挂载和升级同步，不启动 ComfyUI lzc-comfy-overlay boot # 完整启动流程，镜像 CMD 默认使用这个命令 lzc-comfy-overlay run # 查看数据根目录、当前镜像版本、挂载状态和待处理回滚 lzc-comfy-overlay status # 手动创建快照 lzc-comfy-overlay snapshot --reason before-manager-update # 查看快照列表 lzc-comfy-overlay list # 标记回滚目标，重启 app 容器后生效 lzc-comfy-overlay rollback # 清理过期自动快照 lzc-comfy-overlay cleanup ``` The upgrade process is controlled by `lzc-version`. The previously recorded container image version on the device end is at: ```text /lzc-comfy-data/overlay/control/last-image-version ``` When `/opt/ComfyUI/lzc-version` in the new container image does not match the recorded value, or when `COMFYUI_FORCE_UPDATE=1` is set, ComfyUI and the venv are synchronized during startup. Before synchronization, a `pre-image-upgrade-*` snapshot is created. After the upgrade, the service must pass the `http://localhost:8188/api/manager/version` health check within 300 seconds. If it fails, the script automatically rolls back to the pre-upgrade snapshot and restarts. Available environment variables: - `LZC_COMFY_DATA_DIR`: Overrides the default data root directory `/lzc-comfy-data`. - `COMFYUI_FORCE_UPDATE=1`: Forces re-synchronization of the runtime layer from the container image. - `COMFYUI_DISABLE_UPDATE=1`: Skips image synchronization if both runtime layer and venv are already available. Automatic snapshots include startup snapshots, daily snapshots, and pre-upgrade snapshots. Automatic snapshots retain at most the last 5; snapshots older than 7 days are cleaned up at `cleanup`. Manual snapshots are not automatically cleaned. Q: What if ksampler does not support sm87 CUDA instructions? A: This is because the segattention library has not been adapted to support sm87 instructions. Simply changing `/opt/venv/lib/python3.12/site-packages/sageattention/core.py` to match the sm89 logic will resolve it. # Add Proxy Create a proxy in the `/ssd/lzc-ai-agent/services/cloud.lazycat.aipod.comfyui` directory of the AI Pod. `compose.override.yml` ```yml services: app: environment: - https_proxy=http://代理ip:代理端口 ``` Note: If the proxy IP is not set to `127.0.0.1`, the address will be inaccessible within the container. --- # deepseed ``` docker run --gpus all --network=host --shm-size 10.24g --name deepseed -d -ti -v `pwd`:/models -v `pwd`/cache:/root/.cache/huggingface dustynv/pytorch:2.7-r36.4.0-cu128-24.04 bash docker exec -ti deepseed bash ``` --- # FFmpeg ## Jetson Officially Provided FFmpeg 4.4 ```sh echo "deb https://repo.download.nvidia.com/jetson/ffmpeg main main" | sudo tee -a /etc/apt/sources.list echo "deb-src https://repo.download.nvidia.com/jetson/ffmpeg main main" | sudo tee -a /etc/apt/sources.list sudo apt update sudo apt install ffmpeg ``` ## compile ffmpeg ``` LD_LIBRARY_PATH=$HOME/ffmpeg_build/lib ./ffmpeg --version LD_LIBRARY_PATH=$HOME/ffmpeg_build/lib ./ffmpeg -decoders LD_LIBRARY_PATH=$HOME/ffmpeg_build/lib ./ffmpeg -decoders | grep av1 LD_LIBRARY_PATH=$HOME/ffmpeg_build/lib ./ffmpeg -decoders | grep h264_nvv4l2dec ``` - https://github.com/HandBrake/HandBrake/commit/75f9c84c140c8841cfe1324ef59452025899ad8b.patch?full_index=1 --- # Text-to-Image The following API calls require that you have installed the `文生图` application. [Click to install](https://appstore.lazycat.cloud/#/shop/detail/cloud.lazycat.aipod.imagen) The text-to-image API follows the OpenAI API format. 1. Get model endpoint (currently only one default model: sd-cpp-local corresponding to z-image-turbo) `curl https://imagen-ai.$微服名称.heiyu.space/v1/models` 2. Image generation endpoint (https://platform.openai.com/docs/api-reference/images/create) ``` curl https://imagen-ai.微服.heiyu.space/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "model": "sd-cpp-local", "prompt": "A lovely cat", "n": 1, "size": "512x512", "response_format": "b64_json" }' \ -o >(jq -r '.data[0].b64_json' | base64 --decode > output.png) ``` Parameter usage similar to OpenAI: - prompt: prompt text - n: number of generated images - size: image dimensions; both width and height must be integer multiples of 16, otherwise the input is invalid - response_format: currently only supports a fixed format Model-specific parameters: Since the OpenAI format does not reserve additional parameter interfaces, the advanced parameters of Z-Image-Turbo can be placed within the prompt for current version. ``` "prompt": "A lovely cat{\"seed\": 357925}", ``` Currently supported: - **steps**: iteration step count. The default value set by the backend in the current version is 7, with the model-recommended value of 9. The valid range is 0–9. Fewer steps result in faster generation; for demonstration or testing, you can set it to 4 to quickly generate results without waiting too long. Note that "turbo" refers to a distilled model, and the step count must not exceed 20, otherwise the generated image will be improperly rendered and unusable. - **seed**: generation seed. The default is a random value. A user can manually specify a fixed seed to achieve consistent and reproducible results, which is essential for specific use cases requiring stability. --- # LC-X3 The LCMD device and the LC-X3 AI Pod are independent devices. The LCMD device cannot directly access the AI Pod's computing resources. First deploy services on the AI Pod, then call their APIs from the LCMD device. There are no limits on the number of LCMs or AI Pods. Multiple LCMs can connect to a single AI Pod, and a single AI Pod can serve multiple LCMs. This document supports [llms.txt](../llms.en.txt) or [llms-full.txt](../llms-full.en.txt) ## Common Services - [Kokoro TTS API](./tts/tts_api.md) - [Qwen3TTS OpenAI API](./tts/qwen3tts_api.md) --- # DHCP The device shows intermittent connectivity, frequently switching between "connected" and "no connected" states, with no `ip` address assigned via wired network. 1. Check if DHCP broadcast packets are being received. ``` sudo apt install nmap sudo nmap --script broadcast-dhcp-discover ``` If DHCP packets can be received, it indicates that the router and device are connected; the issue lies in the system configuration within the AI Pod. Investigation should be directed toward NetworkManager. - Check NetworkManager logs bash sudo journalctl -u NetworkManager -b 0 -r # -r for reverse order, latest entries at the top - Alternatively, try manually assigning an IP address and use ping to verify if the network interface is properly connected. 2. Check whether data packets are received on a specific network interface. ``` sudo apt install arp-scan sudo arp-scan -I eno1 -l # sudo /usr/sbin/arp-scan -l ``` Here, use `-I` to specify the wired network interface. Otherwise, you might receive packets from Wi-Fi. If packets can be received, it indicates that the issue is not with the Ethernet cable or the device, but possibly with the router. The easiest approach is: - Unplug the Ethernet cable and restart the router - Reconnect the Ethernet cable 3. Ping the specified network interface ``` ping -I eno1 192.168.1.1 ping -I eno1 www.baidu.com ``` --- # Docker startup error for Traefik? Detailed error message below: ``` May 16 06:06:57 lzc-pod-juyIZt dockerd[1391]: time="2025-05-16T06:06:57.617540160Z" level=error msg="failed to start container" container=675c06f1d672ee0c36441d4e0fcd69d48e38355a40a4e85b7a42d0019467ffc4 error="failed to set up container networking: driver failed programming external connectivity on endpoint monitoring-grafana (02c852b6417b0a591b3b22838f3fd71452fd32bf386bf1c186cb0ba5f256b398): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.20.0.6 --dport 3000 ! -i br-8321df9162d9 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table `raw': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n (exit status 3))" ``` Resolve: https://forums.developer.nvidia.com/t/problems-with-docker-version-28-0-1-on-jetson-orin-nx/325541/26 Inside `/etc/systemd/system/docker.service.d/`, add an `override.conf` file containing the following content: ```override.conf [Service] Environment="DOCKER_INSECURE_NO_IPTABLES_RAW=1" ``` After adding, execute the following command to restart the relevant services ``` sudo systemctl daemon-reload sudo systemctl restart docker sudo systemctl restart lzc-ai-agent ``` After Docker restart, you can use `docker info` to access. ``` > docker info | grep DOCKER_INSECURE_NO_IPTABLES_RAW WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set ``` --- # Jetson Suspend After HDMI Black Screen ## Phenomenon After the AI Pod enters suspend mode, the device can be restored over the network. SSH and system services function normally, but the Ubuntu desktop on the HDMI display remains black. The screen does not recover. Typical symptoms: - The device can be accessed using `ssh nvidia@<设备 IP>`. - `gdm` or `display-manager` remains `active`. - The X11 session is still active. - The HDMI display is detected, for example, `xrandr` shows `HDMI-0 connected`. - The local HDMI output is black; the GNOME desktop fails to render properly. ## Diagnostics Commands Log in to the device via SSH: ```bash ssh nvidia@192.168.1.221 ``` Check system version: ```bash cat /etc/nv_tegra_release uname -a ``` Check graphics service status: ```bash systemctl status gdm --no-pager systemctl status display-manager --no-pager loginctl list-sessions ``` Check HDMI output status: ```bash export DISPLAY=:0 export XAUTHORITY=/run/user/1000/gdm/Xauthority xrandr --current ``` Check suspend/resume and graphics stack logs: ```bash journalctl -b --no-pager | egrep -i "suspend|resume|PM:|gnome-shell|mutter|Xorg|nvgpu|semaphore|error notifier" | tail -n 200 ``` If logs show content similar to this, it typically indicates that the system has recovered, but the NVIDIA graphics stack or GNOME Shell has not properly resumed. ```text nvgpu ... semaphore acquire timeout nvgpu ... error notifier set to 24 ... owned by gnome-shell gnome-shell / mutter DisplayConfig timeout ``` ## Cause This issue is not about the system failing to wake up from suspend; it is a graphics stack recovery failure. In the Jetson Linux R36.4.3 / JetPack 6.2 environment, after suspend/resume, NVIDIA `nvgpu`, Xorg, GNOME Shell / Mutter may enter an inconsistent state. The system, network, and services recover correctly, but the HDMI desktop fails to re-render, resulting in a black screen. There are known issues reported on the NVIDIA Developer Forum regarding HDMI black screen after resume on Jetson Linux R36.4.3 / JetPack 6.2. The long-standing recommendation is to upgrade to Jetson Linux R36.5 / JetPack 6.2.2 or later versions. References: - https://forums.developer.nvidia.com/t/orin-nano-8g-hdmi-blank-when-resume/337317 - https://forums.developer.nvidia.com/t/after-suspend-wakeup-hdmi-can-not-display/353817 - https://developer.nvidia.com/embedded/jetson-linux-r365 ## Temporary Solution One: Manually restart gdm After black screen, if SSH login is still possible, manually restart the graphical login service: ```bash sudo systemctl restart gdm ``` A general service name can also be used: ```bash sudo systemctl restart display-manager ``` This operation will restart the local graphical session over HDMI, interrupting the current desktop applications. ## Temporary Solution Two: Automatically restart gdm after resume If system upgrades are not currently possible but you want the desktop to automatically restart after resuming from suspend, you can add a systemd sleep hook: ```bash sudo tee /lib/systemd/system-sleep/restart-gdm-after-resume >/dev/null <<'EOF' #!/bin/sh case "$1" in post) systemctl restart gdm ;; esac EOF sudo chmod +x /lib/systemd/system-sleep/restart-gdm-after-resume ``` Delete this temporary plan: ```bash sudo rm -f /lib/systemd/system-sleep/restart-gdm-after-resume ``` Note: This solution only temporarily works around the issue. Each resume will restart the local graphical session. ## Temporary Solution Three: Disable Sleep If the device is primarily used as a service device and does not rely on the local HDMI desktop, the most stable temporary fix is to disable sleep/suspend to avoid re-triggering the black screen: ```bash sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target ``` Verification: ```bash systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target ``` Sleep recovery capability: ```bash sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target ``` ## Long-term Plan Upgrade to Jetson Linux R36.5 / JetPack 6.2.2 or later versions, and re-verify whether the HDMI desktop recovers after suspend/resume. --- # AI Application Dependency on AI Pod Service Status Monitoring Starting with LCMD `1.3.8`, use `upstreams` to access the `health check` endpoint on the AI Pod, as shown below: ```yml name: ComfyUI package: cloud.lazycat.aipod.comfyui version: 0.2.0 description: ComfyUI is a web UI for AI Art. license: https://choosealicense.com/licenses/mit/ homepage: author: application: subdomain: comfyui gpu_accel: true routes: - /=file:///lzcapp/pkg/content/dist health_check: test_url: http://127.0.0.1/version start_period: 5m upstreams: - location: /version backend: https://comfyui-ai.{{ .S.BoxDomain }}/api/manager/version trim_url_suffix: / use_backend_host: true dump_http_headers_when_5xx: true ``` --- # Routing to AI Services in LZC A simple AI application with a single AI Pod hosting the service, including the UI components. After the service is deployed within the AI Pod, it can be accessed via `https://${服务名称}-ai.${微服名称}.heiyu.space`. In the LPK application routing, you might write something like this: ```yml routers: - /=https://${服务名称}-ai.${微服名称}.heiyu.space ``` ## LCMD 1.3.8+ System This feature uses the LCMD 1.3.8 [upstreams](https://developer.lazycat.cloud/spec/manifest.html#%E5%8D%81%E3%80%81upstreamconfig-%E9%85%8D%E7%BD%AE) system. ```yml application: upstreams: - location: / backend: https://comfyui-ai.{{ .S.BoxDomain }} use_backend_host: true dump_http_headers_when_5xx: true ``` Using template [manifest.yml](https://developer.lazycat.cloud/advanced-manifest-render.html#%E6%A8%A1%E6%9D%BF%E5%8F%82%E6%95%B0) ## System before Microservice 1.3.8 In this case, the router cannot know the current accessed `${微服名称}`, so a forwarding tool is required to handle it, such as `nginx` or `caddy`. ```yml application: routes: - /=http://caddy:80 services: caddy: image: registry.lazycat.cloud/caddy:2.9.1-alpine setup_script: | cat <<'EOF' > /etc/caddy/Caddyfile { auto_https off http_port 80 https_port 0 } :80 { redir https://${服务名称}-ai.{$LAZYCAT_BOX_DOMAIN} permanent } EOF cat /etc/caddy/Caddyfile ``` --- # MDNS Cannot Discover Device Scenario: The AI Pod is connected via Ethernet, but the LCMD fails to discover the AI Pod via MDNS. Solution: 1. Use `hc ustc` to modify the `apt` source (skip this step if the environment can directly update via apt). 2. In the LCMD terminal, run `apt install avahi-utils`. 3. After installation, use `avahi-browser -a` to query the local network broadcast. If the `_ai_jetson` broadcast is received, MDNS is functioning properly and MDNS-related issues can be ruled out. 4. Use `avahi-daemon -r` to restart `daemon`; after successful restart, verify Step 3 again. --- # View the Service Forwarding Rules in the AI Pod The AI Pod uses `traefik` for forwarding, and matching rules are defined by `Host`. When you access via , the request is automatically forwarded based on `ollama-ai` to the container specified by `ollama`, with the port being the one you configured in your container's `expose`. ::: info Info If your service exposes multiple ports, forwarding may fail. You can check using `docker inspect`, or verify whether the `Dockerfile` already defines the `expose` field. ::: You can open to view the `traefik` forwarding configuration in the AI Pod. ![traefik http router screenshot](../../issues/images/traefik-dashboard-http-router.png) --- # Timeout when accessing service via HTTPS When you use `https://${服务名称}-ai.${微服名称}.heiyu.space` to prompt `Gateway timeout`, this may indicate that the service defined in `docker-compose.yml` has not added `traefik` for the network interface. You can add the default network interface in `docker-compose.yml`, or specify the network interface only for the services exposed. 1. Default network interface ```yml networks: default: external: true name: traefik-shared-network ``` 2. Specify network interface ```yml services: xxx: networks: - traefik-shared-network ``` --- # If encountering an Ubuntu system that won't boot, but can access via HDMI TTY switching ## Step 1: Check current system information 1. **Check Ubuntu version**: In TTY, run the following command: bash $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.5 LTS Release: 22.04 Codename: jammy This will display your Ubuntu distribution version and codename (e.g., Ubuntu 22.04.5 LTS). 2. **Check Jetson L4T version**: Jetson L4T is NVIDIA's custom underlying driver and system software for Jetson platforms. Run the following command: bash $ cat /etc/nv_tegra_release # R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025 # KERNEL_VARIANT: oot TARGET_USERSPACE_LIB_DIR=nvidia TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia This will display your L4T version information, which is crucial for determining compatible software packages. 3. **Check JetPack version (if installed)**: JetPack is a bundle containing L4T and various NVIDIA SDKs (such as CUDA, cuDNN). You can check it using the following command: bash sudo apt-cache show nvidia-jetpack | grep "Version" Version: 6.2.1+b38 Version: 6.2+b77 Version: 6.1+b123 ## Step 2: Verify completeness of desktop environment dependencies The most common cause for a graphical interface failing to start is a corrupted or incomplete meta-package for the desktop environment or its dependencies. Ubuntu's default desktop environment is GNOME, and the corresponding meta-package is typically `ubuntu-desktop`. 1. **Check the status of the `ubuntu-desktop` package**: bash dpkg -l | grep ubuntu-desktop If the software package is already installed, you will see a line of information starting with `ii`. If it is not installed or its status is abnormal, there may be a warning message. 2. **Check for corrupted dependencies**: Run the apt dependency checker to identify any unsatisfied dependencies. bash sudo apt-get check If this command produces any output, it indicates the presence of dependency issues. You can attempt to fix them using the following command: bash sudo apt-get -f install 3. **Try reinstalling the desktop environment**: This is the most effective way to resolve dependency-related problems. It will reinstall all missing desktop components. bash sudo apt-get update sudo apt-get install --reinstall ubuntu-desktop sudo systemctl status gdm3 sudo systemctl restart gdm3 The installation process may take some time. After completion, reboot your Jetson device: bash sudo reboot --- 1. In the Micro Service search, the video format must be webm (VP9 encoded). To verify, drag the video into the AI browser to check if it plays normally. 2. You can download a webm video from the internet, or follow the instructions below to convert your video to webm format. 3. Method 1: Use online conversion tools — https://cloudconvert.com/mp4-to-webm/, https://video.online-convert.com/convert-to-webm, https://convertio.co/mp4-webm/ 4. Method 2: Use local tools for conversion. On Windows, see https://videoconvert.minitool.com/video-converter/mp4-to-webm.html for several tools. On macOS, see https://www.winxdvd.com/video-transcoder/convert-webm-to-mp4-vlc.htm 5. Method 3: Use the `vp9convert` tool in LazyCat Micro Service. Search for "vp9convert" in the app store and install it. After installation, you can upload your video files to the cloud disk. Once uploaded, you will see the file in vp9convert. Click `转换` to wait for the video to be transcoded. (After successful transcoding, please delete any non-webm videos from the cloud disk.) --- # Installation Method ## Store Installation Search for "Knowledge Base" in the store to search for installation After installation, on the homepage of your `AI POD` application or AI browser, you will see a `Knowledge` icon. ![Knowledge Base Detail](../../knowledge/images/knowledge-shortcuts-2025-11-04_11-42.png) After clicking Open, the system will check whether the relevant services in the AI Pod are running. You may see the following startup progress page. Please wait a moment (model downloading and startup are required). ![prepare-running](../../knowledge/images/prepare-running.png) If you need to directly invoke knowledge base capabilities within `Codex`, `Claude Code`, and `Cherry Studio`, you can refer to [Knowledge Base MCP](/en/knowledge/mcp.md). --- # Knowledge Base MCP `Knowledge` provides MCP interface support, enabling `Codex`, `Claude Code`, and `Cherry Studio` clients with MCP capability to directly access files, web pages, and retrieval capabilities within the knowledge base. ## Prerequisites - Install the latest version of the knowledge base; installation instructions are available at [Installation Methods](/en/knowledge/install.md). - This document is based on the current knowledge base repository version `1.3.40`. - If using the desktop MCP client, it is recommended to generate an access token for the corresponding user. ## Service Endpoints - Knowledge Application URL: - MCP Endpoint: - Protocol: `Streamable HTTP` ## Authentication Method 1. `ssh` Login to LCMD. 2. Execute the following command to generate a token for users who need to access the knowledge base: ```bash hc api_auth_token gen --uid ``` 3. In the MCP client configuration, pass this token via `Lzc-Api-Auth-Token`. ## Tool Description The knowledge base MCP organizes tools under three namespaces: `rag`, `fs`, and `ws`. Many MCP clients will display them as `rag_search`, `fs_ls`, `ws_ls` respectively. ### `rag` - `rag_search`: Search for relevant fragments in the knowledge base. - `rag_summarize`: Perform summary analysis on a specified file or web page. - `rag_read`: Paginated reading of file or web page content. ### `fs` - `fs_ls`: List the directory structure of knowledge base files. - `fs_search`: Search files or directories by keyword. - `fs_find`: Find a file by filename. - `fs_traverse`: Recursively traverse directories. - `fs_get_info`: View details of a file or directory. - `fs_get_stats`：Check file system statistics. - `fs_sync`：Sync knowledge base file status. - `fs_mkdir`、`fs_rm`、`fs_mv`、`fs_update_meta`：Create, delete, move, update metadata. ### `ws` - `ws_ls`：List web knowledge directory. - `ws_search`：Search saved web pages. - `ws_get_stats`：View web knowledge statistics. - `ws_mkdir`、`ws_delete`、`ws_update_meta`：Manage web directory and metadata. > `fs_mkdir`、`fs_rm`、`fs_mv`、`fs_update_meta`、`ws_mkdir`、`ws_delete`、`ws_update_meta` are write operations that modify knowledge base content. These should be used only after confirming the correct scope. ## MCP Example > Note: Replace `xxx` with your LCMD name in the configuration below. ### Claude Code Configuration Edit the Claude Code configuration file `~/.claude.json`, and add the following MCP server configuration: ```json { "mcpServers": { "knowledge": { "type": "http", "url": "https://knowledge.xxx.heiyu.space/mcp/", "headers": { "Lzc-Api-Auth-Token": "" } } } } ``` ### Codex Configuration Edit the Codex configuration file `~/.codex/config.toml`, and add the following MCP server configuration: ```toml [mcp_servers] [mcp_servers.knowledge] type = "http" url = "https://knowledge.xxx.heiyu.space/mcp/" [mcp_servers.knowledge.http_headers] Lzc-Api-Auth-Token = "" ``` ### AI Dialogue Client Configuration Using Cherry Studio as an example, configure the following settings under *Settings -> MCP Server*: - Name: `knowledge` - Type: Streamable HTTP (`streamableHttp`) - URL: `https://knowledge.xxx.heiyu.space/mcp/` - Header: `Lzc-Api-Auth-Token: ` After saving, enable the MCP server. The knowledge base tools will then appear in the tool list. ## Common Usage Scenarios - Query "the document conclusions in the knowledge base regarding a specific topic" within `Codex`. - First list `/` or files under a specific directory, then perform retrieval within that directory scope. - Generate a summary of a document directory, then further examine the matched segments. - Search for previously saved web content within the knowledge base, rather than re-scraping live web pages. ## Parameter Descriptions - ``: User ID required to access the knowledge base. - ``: Access token generated via `hc api_auth_token gen`. - `<微服名称>`: Current LCMD name, corresponding to `knowledge.<微服名称>.heiyu.space`. --- # LC-X5 Configuration and `lzc-thor` Usage After flashing the Lc-X5, you can use `lzc-thor` to complete system initialization, MGBE network testing, VNC configuration, and disk management. Except for viewing help, all commands in this document are recommended to be executed using `sudo`. First, use the following command to verify the current system's supported commands: ```bash sudo lzc-thor help ``` ## Command Quick Reference | Use scenarios | Command | | --- | --- | | Complete the routine initialization of the new device | `sudo lzc-thor all` | | Install JetPack and monitoring tools separately | `sudo lzc-thor packages` | | Fix the machine and network card identities of the cloned system | `sudo lzc-thor clone-identity` | | Install Docker and NVIDIA Container Runtime | `sudo lzc-thor docker` | | Install the standalone `docker-compose` command | `sudo lzc-thor docker-compose` | | Disable Ubuntu automatic upgrades | `sudo lzc-thor disable-upgrades` | | Configure MGBE performance parameters | `sudo lzc-thor mgbe` | | Configure or execute MGBE bandwidth test | `sudo lzc-thor mgbe serve ...`、`sudo lzc-thor mgbe client ...` | | Configure timezone, time synchronization, maximum frequency, and Firefox | `sudo lzc-thor startup` | | Install and start `lzc-ai-agent` | `sudo lzc-thor agent` | | Configure desktop VNC | `sudo lzc-thor vnc ...` | | Expand the root filesystem on the current system disk | `sudo lzc-thor resizefs` | | Establish a read-only APP recovery system and LVM runtime system | `sudo lzc-thor provision ...` | | Add a new NVMe whole disk to the root filesystem | `sudo lzc-thor expand-lvm ...` | | Recover factory operating system from APP recovery | `sudo lzc-thor factory-reset ...` | `vnc`- MGBE test, `resizefs`, `provision`, `expand-lvm`, and `factory-reset` are not included in `all`; they must be executed separately as needed. ## Rapid Initialization for New Devices ### Using Default User ```bash sudo lzc-thor all ``` The command executes in the following order: 1. Install JetPack, compilation tools, and GPU monitoring tools. 2. Regenerate the machine identity of the cloned system and configure a stable, unique Ethernet MAC address for each device. 3. Install Docker CE, Buildx, Compose plugin, and NVIDIA Container Toolkit. 4. Install the independent `/usr/bin/docker-compose`. 5. Disable Ubuntu automatic upgrades. 6. Configure threaded NAPI and IOMMU DMA-FQ for all discovered MGBE interfaces, and run `jetson_clocks`. 7. Configure the timezone, network time synchronization, enable `jetson_clocks` and Firefox to start at boot. 8. Install and start `lzc-ai-agent`. By default, the `nvidia` user is added to the `docker` group. After full initialization, it is recommended to restart the device to ensure that machine identity, network interface identity, and user group configurations take effect. After re-login, you can execute `docker ps` to verify Docker permissions. ### Specify Docker User If the regular login user on the system is not `nvidia`, use `--docker-user` to specify the user: ```bash sudo lzc-thor all --docker-user ubuntu ``` Also supports the following equivalent formulations: ```bash sudo lzc-thor all --user ubuntu sudo lzc-thor all ubuntu ``` `lzc-thor-init-all` The equivalent entry point for `lzc-thor all`: ```bash sudo lzc-thor-init-all --docker-user ubuntu ``` ## General Configuration Commands ### `help`: View command help ```bash sudo lzc-thor help sudo lzc-thor --help ``` When viewing the detailed usage of a command with parameters, place `--help` after the corresponding command: ```bash sudo lzc-thor clone-identity --help sudo lzc-thor mgbe serve --help sudo lzc-thor provision --help sudo lzc-thor expand-lvm --help sudo lzc-thor factory-reset --help ``` ### `packages`: Install Jetson Software Package ```bash sudo lzc-thor packages ``` This command installs the following packages via apt: - `nvidia-jetpack`: Jetson platform software stack. - `build-essential`: Common compilation tools. - `nvitop`, `gpustat`, `nvtop`: GPU status monitoring tools. This command is suitable for scenarios where only these packages need to be reinstalled or repaired, without performing a full initialization. The command has no additional parameters. ### `clone-identity`: Repair cloned device identity After cloning multiple LC-X5 devices from the same source disk, devices may share identical `/etc/machine-id`, NetworkManager keys, or wired network identities, leading to DHCP and local network device recognition issues. Preview the operations that will be performed: ```bash sudo lzc-thor clone-identity --dry-run ``` Confirm and execute the repair, then restart after completion: ```bash sudo lzc-thor clone-identity sudo reboot ``` This command will regenerate `/etc/machine-id`, delete the NetworkManager host key introduced by cloning, and configure the existing Ethernet connection to use a stable local management MAC generated per-device. By default, it only reloads the NetworkManager configuration and does not restart active network connections. If operating directly from the local console and acceptable to have the network temporarily interrupted, this can be applied directly: ```bash sudo lzc-thor clone-identity --apply-now ``` > `--apply-now` will restart NetworkManager, and the SSH session may disconnect immediately. When operating via SSH, prefer commands without this parameter, and then restart the device. Available parameters: | Parameters | Description | | --- | --- | | `--apply-now` | Restart NetworkManager immediately after writing the configuration. | | `--no-machine-id` | Do not regenerate `/etc/machine-id`. | | `--no-profile-update` | Do not modify the existing NetworkManager connection configuration. | | `--dry-run` | Show only the schedule, do not modify the system. | When maintaining a cloned system that is mounted but not running, use `LZC_THOR_ROOT` to specify its root directory. In offline mode, only files under this directory are modified, and NetworkManager of the current system is not affected. ```bash sudo LZC_THOR_ROOT=/mnt/thor-root lzc-thor clone-identity ``` ### `docker`: Install Docker and NVIDIA Runtime Install only Docker without modifying any user groups: ```bash sudo lzc-thor docker ``` Allow specifying users to use Docker without `sudo`: ```bash sudo lzc-thor docker --docker-user ubuntu ``` `--user ubuntu` And writing `ubuntu` directly is an equivalent approach. This command does the following: - Configures the Docker official apt software repository. - Installs Docker CE, Docker CLI, containerd, Buildx, and Compose plugins. - Installs NVIDIA Container Toolkit and configures the NVIDIA runtime as the default Docker runtime. - Enables and restarts `docker.service`. - When specified, adds the user to the `docker` group. After modifying the user group, existing login sessions typically require exiting and relogging in to obtain the new permissions. ### `docker-compose`: Install standalone Compose command ```bash sudo lzc-thor docker-compose docker-compose version ``` This command downloads the Linux aarch64 binary from the latest GitHub release of Docker Compose, verifies `.sha256`, and installs it into `/usr/bin/docker-compose`. It coexists with the `docker compose` plugin provided by the Docker package. The device must have access to GitHub. This command always retrieves the latest version available at runtime and does not support version selection parameters. ### `disable-upgrades`: Disable automatic upgrades ```bash sudo lzc-thor disable-upgrades ``` This command shuts down and disables apt scheduled updates, `unattended-upgrades`, and update-notifier services, while uninstalling the related packages. It is suitable for devices requiring a fixed JetPack, kernel, and driver environment. After disabling automatic upgrades, the system will not automatically install security updates; compatibility validation must be arranged by administrators and updates must be performed manually. ### `startup`: Configure the boot-up environment ```bash sudo lzc-thor startup ``` This command will: - Query the time zone using the current public IP; if the query fails, use `Asia/Shanghai`. - Enable `systemd-timesyncd`, and configure NTP services such as Cloudflare and Alibaba Cloud. - Create and enable `lzc-jetson-clocks.service`, locking the Jetson to maximum frequency at boot time. - Install Firefox via Snap. This command requires access to the time zone query service, the Ubuntu software repository, and Snap services. It has no additional parameters. `agent`: Install AI Pod proxy ```bash sudo lzc-thor agent sudo systemctl status lzc-ai-agent.service ``` This command installs the ARM64 client from the package included in `lzc-ai-agent`, then enables and restarts `lzc-ai-agent.service`. It is used for standalone reinstallation or repair of the AI Pod agent. The command has no additional parameters. ## MGBE Configuration and Bandwidth Testing ### `mgbe`: Configure MGBE Performance Parameters ```bash sudo lzc-thor mgbe ``` This command automatically discovers all physical MGBE interfaces with names similar to `mgbe0_0` and performs the following configuration: - Enables threaded NAPI. - Runs `jetson_clocks`. - Sets the IOMMU group of each interface to `DMA-FQ`. This command only handles performance parameters; it does not assign test IPs or start `iperf3`. It is included in `lzc-thor all`. ### How MGBE Testing Works MGBE testing requires two devices: one running the server command and the other running the client command. Before testing, both devices must execute: ```bash sudo lzc-thor mgbe sudo apt install -y iperf3 ``` The test assistance commands are divided into three steps: 1. Assign static IPs to each link via `serve config` and `client config`, write the configuration to `/etc/netplan/99-mgbe-test.yaml`, and save the link allocation to `/var/lib/lzc-thor/mgbe-test`. 2. `serve run` starts `iperf3` from the stored server configuration; `client run` initiates the test from the stored client configuration. 3. Both ends execute `clean` to stop the test process and delete temporary network configurations and states. `config` and `run` must be used together on the same end, in accordance with their respective roles. After modifying the IP address or the number of test links, the corresponding `config` must be re-executed. The parameters and purposes of each test command are as follows: | Command | Parameters | Usage | | --- | --- | --- | | `mgbe serve config` | Must provide `--base-ip IP`; optional `--mode 3x\|4x`, `--dry-run` | Assign IP to server interface, apply netplan, and save server state. | | `mgbe serve run` | Optional `--dry-run` | Read the server status and start `iperf3` services on 8 different ports for each link. | | `mgbe client config` | Must provide `--server-base-ip IP` and `--local-suffix 1-254`; optional `--mode 3x\|4x`, `--dry-run` | Calculate the endpoint addresses based on the server-side baseline IP, apply the client's netplan, and save the state. | | `mgbe client run` | Optional `--duration SEC`, `--parallel N`, `--udp`, `--dry-run` | Check if each link is reachable, and simultaneously run all `iperf3` clients, and aggregate the sender bandwidth. | | `mgbe clean` | Optional `--dry-run` | Stop recorded service processes, delete test netplan and both end states. | ### Three-Link Test Example The following example applies to a scenario where T4000 and T5000 are connected using three 25GbE links. Server-side execution: ```bash sudo lzc-thor mgbe clean sudo lzc-thor mgbe serve config --base-ip 192.168.100.10 --mode 3x sudo lzc-thor mgbe serve run ``` Client executes: ```bash sudo lzc-thor mgbe clean sudo lzc-thor mgbe client config \ --server-base-ip 192.168.100.10 \ --local-suffix 20 \ --mode 3x sudo lzc-thor mgbe client run ``` The IP address will increment by one in the third octet along the chain: | link | Server-side IP | Client IP | | --- | --- | --- | | Article 1 | `192.168.100.10` | `192.168.100.20` | | Article 2 | `192.168.101.10` | `192.168.101.20` | | Article 3 | `192.168.102.10` | `192.168.102.20` | `--local-suffix` The last segment of the client IP is in the range from `1` to `254`, and must not match the last segment of the server IP. ### Choose 3x, 4x, or auto mode `--mode` is only for `serve config` and `client config`: | Writing method | Used interfaces | | --- | --- | | Do not write `--mode` | `auto` mode, uses all automatically discovered MGBE interfaces. | | `--mode 3x` | Use the first 3 interfaces sorted by interface name. | | `--mode 4x` | Use the first 4 interfaces after sorting by interface name. | Both ends must use the same mode. Example of four-link test: ```bash # 服务端 sudo lzc-thor mgbe serve config --base-ip 192.168.100.10 --mode 4x sudo lzc-thor mgbe serve run # 客户端 sudo lzc-thor mgbe client config \ --server-base-ip 192.168.100.10 \ --local-suffix 20 \ --mode 4x sudo lzc-thor mgbe client run ``` `run` Will load the mode saved by `config`, and does not accept `--mode`. The old `lzc-thor mgbe test` has been removed. ### Adjust Test Parameters The client defaults to 10 seconds of testing, with each `iperf3` client process using 24 parallel streams: ```bash sudo lzc-thor mgbe client run --duration 30 --parallel 32 ``` UDP Test: ```bash sudo lzc-thor mgbe client run --udp ``` Each MGBE link uses 8 TOS values to start 8 client processes. Thus, 3x mode launches 24 client processes concurrently, and 4x mode launches 32; `--parallel` is the number of parallel streams within each process, not the total number of processes. When executing `run`, the tool checks the saved interface's MTU 9000, threaded NAPI, IOMMU DMA-FQ, and the corresponding ODMDATA 25GbE configuration; a warning is issued if `jetson_clocks` or ODMDATA cannot be confirmed. If the check fails, re-run `sudo lzc-thor mgbe` and verify the device's link mode before using incomplete test results. After the test completes, `Concurrent aggregate sender bandwidth` is output. This value only summarizes the send-side results resolvable within a single concurrent test window and should not be calculated by summing up sequential test results across multiple rounds and treated as link bandwidth. ### Preview and Cleanup of Test Configuration The test auxiliary command supports `--dry-run`, which allows checking the planned configuration without modifying the host. ```bash sudo lzc-thor mgbe serve config --base-ip 192.168.100.10 --mode 3x --dry-run sudo lzc-thor mgbe client run --dry-run sudo lzc-thor mgbe clean --dry-run ``` After official beta testing, perform the following on both the server and client sides: ```bash sudo lzc-thor mgbe clean ``` `clean` Will stop the `iperf3` service process, delete the states from `/etc/netplan/99-mgbe-test.yaml` and `/var/lib/lzc-thor/mgbe-test`, and reapply network configuration. ## VNC Configuration `vnc` is not included in `all`. Configure VNC for the default `nvidia` desktop user: ```bash sudo lzc-thor vnc --password '' ``` Configure for other desktop users: ```bash sudo lzc-thor vnc --user ubuntu --password '' ``` The user and password can also be passed in via environment variables: ```bash sudo LZC_THOR_VNC_USER=ubuntu \ LZC_THOR_VNC_PASSWORD='' \ lzc-thor vnc ``` | Parameters or variables | Description | | --- | --- | | `--user USER` | Target desktop user; defaults to read `LZC_THOR_VNC_USER`, defaults to `nvidia` when not set. | | `--password PASSWORD` | The required VNC password, which can also be provided via `LZC_THOR_VNC_PASSWORD`. | This command installs Vino on demand, enables its desktop startup item, and writes VNC configuration to the user's settings. A device restart is required afterward. When desktop automatic login is not configured, the target user typically needs to log in locally to the graphical desktop before VNC can connect. ## Storage Commands `resizefs`, `provision`, `expand-lvm`, and `factory-reset` manipulate partitions, filesystems, LVM, or boot configurations. Before execution, important data should be backed up, and the device name and current root filesystem should be verified. ```bash findmnt / lsblk -o NAME,PATH,SIZE,TYPE,FSTYPE,PARTLABEL,MOUNTPOINTS sudo pvs sudo vgs sudo lvs ``` ### `resizefs`: Use remaining space on current system disk After cloning a smaller system disk to a larger disk, you can execute: ```bash sudo lzc-thor resizefs ``` This command automatically selects the appropriate handling method based on the current `/` layout: - For a standard ext4 root partition: extend the partition to occupy the remaining disk space, then online-extend the ext4 filesystem. - For a standard `aipod_vg/lv_root` with a single partitioned PV: extend the partition containing the PV and the PV itself, allocate all free space in the VG to `lv_root`, then online-extend the ext4 filesystem. This command does not support multi-PV, encrypted storage, full-disk filesystems, non-ext4 root filesystems, or non-standard LVM layouts. For multi-disk LVM systems, when adding a new disk, use `expand-lvm` instead of `resizefs`. ### `provision`: Set up APP recovery and LVM runtime `provision` is used to convert a LC-X5 with `APP` and `DATA_PV` partitions into the following layout: - `APP`: a read-only factory system and recovery environment, preserving the L4TLauncher boot files. - `aipod_vg/lv_root`: a writable ext4 root filesystem for daily use. This is a two-stage process, with each stage automatically restarting the system. #### Stage 1: Prepare in the writable APP First, check the current state and plan: ```bash sudo lzc-thor provision status sudo lzc-thor provision --dry-run ``` Confirmed. Proceeding to prepare. ```bash sudo lzc-thor provision --yes ``` Stage 1: Installs storage and initramfs tools, expands `DATA_PV`, checks LVM support in the initrd, writes managed boot entries, sets the APP configuration to read-only, and sets the next boot target to `LZC Thor APP recovery`, then reboots. #### Stage 2: Creates the running system in APP recovery After the device restarts and logs into `LZC Thor APP recovery`, execute: ```bash sudo lzc-thor provision --maintenance --yes ``` Stage 2 will verify that the current root directory is indeed a read-only APP, create `aipod_vg/lv_root`, clone the APP block-wise to the logical volume, extend the ext4 filesystem, generate a new filesystem UUID and machine identity, and then switch to the LVM runtime before restarting. After entering the runtime system, you can check: ```bash sudo lzc-thor provision status findmnt / findmnt /factory ``` Under normal circumstances, `/` comes from `/dev/aipod_vg/lv_root`, and the APP is mounted in read-only mode on `/factory`. The software package corresponding to the current boot file is held, so online updates to the kernel, DTB, and initrd are not supported. > `provision` will modify the partition and boot configuration. Do not skip `--dry-run`, and do not forcibly execute this on devices that do not meet the layout requirements of `APP` and `DATA_PV`. ### `expand-lvm`: Adding an NVMe Drive to the Root File System This command is only applicable when the root directory is already writable (`/dev/aipod_vg/lv_root`), and is used to add a complete NVMe disk to the root file system. First, verify the target disk based on capacity, model, and serial number: ```bash lsblk -d -o NAME,PATH,SIZE,MODEL,SERIAL ``` Perform a read-only check and preview on the empty disk first: ```bash sudo lzc-thor expand-lvm --disk /dev/nvme1n1 --dry-run ``` Confirm the target is correct, then proceed with scaling. ```bash sudo lzc-thor expand-lvm --disk /dev/nvme1n1 ``` An empty disk will create a GPT and establish a partition marked as `LZC_THOR_PV`. Then, the commands `pvcreate`, `vgextend`, `lvextend`, and `resize2fs` will be executed to expand the root file system online. If partitions or filesystems already exist on the disk, the operation is rejected by default. To proceed, explicitly confirm that all data can be completely erased, and then use: ```bash # 交互式确认时，按提示完整输入目标磁盘路径 sudo lzc-thor expand-lvm --disk /dev/nvme1n1 --force-wipe # 无人值守确认；会直接清空目标盘 sudo lzc-thor expand-lvm --disk /dev/nvme1n1 --force-wipe --yes ``` | Parameters | Description | | --- | --- | | `--disk /dev/nvmeXnY` | Required; must be a complete NVMe device, not a partition. | | `--dry-run` | Execute runtime environment and disk security checks, display the plan, do not write to disk. | | `--force-wipe` | Authorize clearing non-empty target disk; prompt for interaction when `--yes` is not provided. | | `--yes` | Can only be used together with `--force-wipe`, skip interactive confirmation. | After expansion, each member disk becomes a permanent part of the root file system. Removing any member disk may cause the system to fail to boot. When performing a factory reset later, all `DATA_PV` and `LZC_THOR_PV` member disks will be cleared and rebuilt. The command supports resuming from completed `pvcreate`, `vgextend`, or expansion steps. Member disks that have already been fully joined and allocated space will not be expanded again during subsequent executions. ### `factory-reset`: Restore LVM Runtime System `factory-reset` can only be executed in a manually selected read-only `LZC Thor APP recovery` and cannot be run during normal LVM runtime. 1. Reboot the device and manually select `LZC Thor APP recovery` in the extlinux menu. 2. If the system uses multiple LVM member disks, ensure all expanded disks are connected. 3. Enter recovery mode, check the status, and perform a dry run first: ```bash sudo lzc-thor factory-reset status sudo lzc-thor factory-reset --dry-run ``` 4. Confirm and execute recovery: ```bash sudo lzc-thor factory-reset --yes ``` This command checks whether the system was booted from a read-only APP, whether the LVM topology matches expectations, whether all member disks are online, and whether each PV supports discard. It then clears all `DATA_PV` and `LZC_THOR_PV`, rebuilds `aipod_vg/lv_root` with the original total capacity, clones a fresh runtime system from the APP, and restarts. The APP, the `/etc/fstab` within the APP, and the extlinux default boot entry remain unchanged. > `factory-reset --yes` permanently deletes the runtime system, Docker data, models, applications, and user files. `--dry-run` does not write to disk, but still performs environment recovery, member disk, and discard capability checks. ## Common Issues ### Why is there still a permission denied error after running Docker? After joining the `docker` group, the current session does not automatically refresh group information. Exit the SSH or desktop session and log back in, then run: ```bash docker ps ``` ### Why do `mgbe serve run` or `client run` show no status? `run` Do not receive IP addresses and `--mode`; instead, read the most recent state saved by the same role during `config`. First re-execute `serve config` or `client config`, then proceed to execute the corresponding `run`. ### When to use `resizefs` and `expand-lvm`? - The original system disk has grown larger; wish to use the remaining space at the end of the same disk: use `resizefs`. - Already running on `aipod_vg/lv_root`, want to add another complete NVMe to the root file system: using `expand-lvm`. ### How to view only the current storage layout? ```bash sudo lzc-thor provision status ``` `provision status` Both `factory-reset status` and `/factory` output only the current root directory, `/boot`, block devices, LVM, and extlinux default boot entries, without modifying the system. --- # LC-X5 The LC-X5 is a device in the AI Pod series designed to work with LCMD to provide computing power for AI applications and model services. Detailed system usage, flashing, and service deployment documentation for the LC-X5 will be provided in subsequent updates. --- # OCR A `paddleocr`-based `ocr` service has been deployed in the AI Pod. ## Documentation The API of the `ocr` service can be accessed via . ```bash curl -X 'POST' \ 'https://ocr-ai.13gxg.heiyu.space/ocr' \ -H 'accept: application/json' \ -H 'Content-Type: multipart/form-data' \ -F 'file=@ocr-4.png;type=image/png' \ -F 'request={"max_size":1920,"overlap":100,"memory_limit_mb":1024,"det":true,"rec":true,"cls":true}' ``` ## Deploying as a Standalone 1. Save the following content to a `docker-compose.yml` file ```yml services: ocr: image: registry.lazycat.cloud/x/lzc-aipod-ocr:d81102a ports: - 3000:3000 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 10s timeout: 30s retries: 100 start_period: 1s ``` 2. Run `docker-compose up -d` to start in the current directory 3. After startup, access `http://127.0.0.1:3000/docs` --- # Whisper turbo ## ONNX 1. Test audio duration: 14 minutes 05 seconds [mp3](https://anchor.fm/s/f7cac464/podcast/play/92312327/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2024-8-28%2F387196719-44100-2-f0f9be974d84c.mp3) | Backend | Model | Audio length | Inference time | Ratio | |--------------------------|:-----------|--------------------|:----------|-----------------------------------| | whisper (refers to the official native implementation) | large-v3 | 14:05 UTC | 349.49 seconds | 2.42 (approximately 1 second of inference for 2.42 seconds of audio) | | onnx + CUDA | turbo | Same as above | 141.05 seconds | 5.99 | | onnx + Tensorrt | turbo | Same as above | Testing... | | | whisper | turbo | Same as above | 89.90 seconds | 9.38 | | whisper_trt | tiny.en | Same as above | 26.38 | 32.03 | | whisper | tiny.en | Same as above | 70.12 | 12.05 | | FunASR Framework | SenseVoice | Same as above | 24.11 | 35.04 | | FunASR framework | SenseVoice | 20 minutes: Mandarin and Cantonese | 34.31s | 34.97 | 2. Test audio duration: 30 seconds | Backend | Model | Voice length | Inference time | |-----------------|:-----------|----------|:----------| | whisper | turbo | 30s | 3.51 seconds | | onnx + CUDA | turbo | 30s | 5.54 seconds | | onnx + Tensorrt | turbo | 30s | Testing in progress... | | whisper_trt | tiny.en | 30 seconds | 1.80 seconds | | whisper | tiny.en | 30 seconds | 3.00 seconds | | funasr | SenseVoice | 30s | 1.08 seconds | | | | | | --- # LLM As context length increases, token generation speed slightly decreases. ## llama.cpp loader | Model | CPU | GPU (cuda) | |:----------------------------------------|:--------------|:--------------------------| | Meta-Llama-3.1-70B-Instruct-Q5_K_S.gguf | 0.29 tokens/s | 3.04 tokens/s (Power: 60W) | | llama-2-7b-chat.Q4_0.gguf | | 28 tokens/s (Power: 43W) | | | | | ## Using the MLC Loader: MLC Loading is Faster than llama.cpp and Direct Python Transformers This is normal usage, with response speed degrading slightly as question length increases. | Model | CPU | GPU (cuda) | |:----------------------------------------|:----|:--------------------------| | mlc-ai/Qwen2.5-72B-Instruct-q4f16_1-MLC | | 3.3 tokens/s (Power: 56W) | | mlc-ai/Qwen2.5-32B-Instruct-q4f16_1-MLC | | 7.5 tokens/s (Power: 53W) | | mlc-ai/Qwen2.5-14B-Instruct-q4f16_1-MLC | | 16.3 tokens/s (Power: 55W) | | mlc-ai/Llama-2-13b-chat-hf-q4f16_1-MLC | | 16.4 tokens/s (Power: 58W) | The official benchmark [benchmark](https://www.jetson-ai-lab.com/benchmarks.html) limits the generated token length and prompt length, and both are first-time loads with no context. Here are the results obtained using the official test. | Model | CPU | GPU (cuda) | |:-----------------------------------------|:----|:------------------------------------| | mlc-ai/Qwen2.5-72B-Instruct-q4f16_1-MLC | | 3.49 tokens/s | | mlc-ai/Llama-3-70B-Instruct-q4f16_1-MLC | | 4.1 tokens/s (Power: 62W) | | mlc-ai/Qwen2.5-32B-Instruct-q4f16_1-MLC | | 8.2 tokens/s | | mlc-ai/Qwen2.5-14B-Instruct-q4f16_1-MLC | | 17.6 tokens/s | | mlc-ai/Llama-2-13b-chat-hf-q4f16_1-MLC | | 20.6 tokens/s (official: 25.1 tokens/s) | | mlc-ai/Llama-3.1-8B-Instruct-q4f16_1-MLC | | 32.3 tokens/s (Power: 55W) | | mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC | | 32.2 tokens/s (official: 40 tokens/s) | | mlc-ai/Qwen2.5-7B-Instruct-q4f32_1-MLC | | 25.73 tokens/s | # ASR Language Recognition Test audio: A 17-minute meeting recording (Chinese) | Model | Framework | Inference time | Ratio (processing length / s) | Effect | |:--------------|:--------|:---------|:-------------------|--------------------------------| | whisper turbo | whisper | 192.298s | 5.3125 | | | paraformer-zh | funasr | 81.551s | 12.59 | | | SenseVoice | funasr | 37.49 | 27.20 | | Test audio: A Chinese movie audio clip (22 minutes 32 seconds) | Model | Framework | Inference time | Ratio (processing length / s) | Effect | |:--------------|:--------|:---------|:-------------------|----------------------------------------------------------| | whisper turbo | whisper | 230s | 5.878 | Partial Chinese errors, but most dialog can be recognized | | whisper tiny | whisper | 252s | 5.36 | Most Chinese errors, poor quality | | paraformer-zh | funasr | 86.384 | 15.65 | The generated quality is quite good. | | SenseVoice | funasr | 44.83 | 30.15 | The generation speed is the fastest, but it feels like many details are missing—only the most prominent sounds are being recognized. | Test audio: A 14-minute English UP speech | Model | Framework | Inference time | Ratio (processing length / s) | Effect | |:----------------|:--------|:---------|:-------------------|--------------------------------------------| | whisper turbo | whisper | 162.48s | 5.16 | The difference in performance between turbo and tiny is not significant. | | whisper tiny | whisper | 127s | 6.614 | For English, tiny.en and tiny make no significant difference. | | whisper tiny.en | whisper | 123s | 6.82 | | | SenseVoice | funasr | 36.12s | 23.25 | | The difference between `whisper turbo` and `whisper tiny` lies in the `turbo` model's stronger learning capability (better generalization ability), enabling it to handle more unknown scenarios. In contrast, `tiny` performs well in English but fails in other languages. `whisper` is particularly suitable for English-language environments such as `up主说话` or movies, with excellent sentence segmentation, punctuation, and sentence structure. Both `paraformer-zh` and `SenseVoice` produce a single long sentence in output (this is likely due to FunASR-related post-processing that reorganizes and recombines words after recognition). However, this can lead to long audio segments where two separate sentences are incorrectly merged into one. ```sh ssh nvidia2 cd lnks jetson-containers run -v `pwd`:/workspace funasr_whisper_test:latest cd /workspace/funasr python3 funasr_paraformer-zh_test.py python3 funasr_sense_voice-test.py ``` --- # PyNINIs not available on Jetson; must be built manually --- # fish-speech - Extract voice characteristics from existing WAV audio files fish-speech requires Python 3.10. Only with JetPack 6.0 or later does Jetson PyTorch provide a Python 3.10 build; otherwise, you must build it yourself. Therefore, run this model on Ubuntu 22.40 with JetPack 6.0 and CUDA 12.x or later whenever possible. 1. Install libopenblas-dev ``` sudo apt-get -y update; sudo apt-get -y install libopenblas-dev; ``` 2. Download the PyTorch 2.3 wheel corresponding to JetPack 6 [link](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048/1414) Install torchaudio, torchvision, and torch directly. ``` conda create -n fish-speech python=3.10 gcc=12.1.0 pip3 install ./torch-2.3.0-cp310-cp310-linux_aarch64.whl ./torchaudio-2.3.0+952ea74-cp310-cp310-linux_aarch64.whl ./torchvision-0.18.0a0+6043bc2-cp310-cp310-linux_aarch64.whl ``` 3. Install according to the [tutorial](https://github.com/fishaudio/fish-speech/blob/main/docs/zh/index.md) for fish-speech ``` pip install -e . ``` Without acceleration, the speed during the semantic token generation phase is 5.5 tokens/s. 4. Compile the model in fish-speech for acceleration by using PyTorch's torch.compile, which compiles the model similarly to JIT acceleration. NVIDIA PyTorch provides a `Torch-TensorRt` torch compile backend, but it does not offer a precompiled aarch64 package, requiring manual compilation. The Jetson AGX Orin has a JetPack 5.x prebuilt container, but no JetPack 6 version is available. Using this container on JetPack 6 results in errors related to libc and other libraries. ``` root@tegra-ubuntu:/workspace# python3 -c 'import torch_tensorrt; print(torch_tensorrt.__version__)' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/__init__.py", line 39, in import tensorrt File "/usr/lib/python3.8/dist-packages/tensorrt/__init__.py", line 68, in from .tensorrt import * ImportError: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/__init__.py", line 82, in ctypes.CDLL(_find_lib(lib, LINUX_PATHS)) File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__ self._handle = _dlopen(self._name, mode) OSError: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/aarch64-linux-gnu/nvidia/libnvdla_compiler.so) ``` 5. Use the prebuilt container on an Ubuntu 20.0 and JetPack 5.x system. However, fish-speech uses Python 3.10 while the container packages only support Python 3.8, so you would still need to build a Python 3.8 version of torch_tensorrt yourself. Packages such as PyTorch also need to be built because no prebuilt Python 3.10 packages are available for Jetson AArch64. ``` sudo docker run -d --runtime nvidia -it --rm --network host --env HF_ENDPOINT=https://hf-mirror.com -v `pwd`:/workspace --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/nvidia/lnks/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock dustynv/torch_tensorrt:r35.3.1 ``` 6. Build the Torch TensorRT module in the Ubuntu 22.04 JetPack 6.0 Conda Python 3.10 environment - dpkg -l | grep tensorrt -i to check the current installed TensorRT version; do not switch to a different version - Based on the current versions of all system components, check out a supported commit using git - git fetch --depth=1 origin commit-xxx - git checkout commit-xxx - If errors occur, modify the BUILD or WORKSPACE file to adjust the CUDA version - Specify the LD_LIBRARY_PATH directory, otherwise the linker will fail, leading to build failure. (In Bazel, environment variables must be explicitly passed as arguments) - bazel build --action_env=LD_LIBRARY_PATH=/path/to/lib your_target ``` git clone --branch=main --depth=1 https://github.com/pytorch/TensorRT torch_tensorrt git fetch --depth=1 origin 4c3d026ea64bd646a840bf174ef9d1818f10c20f git checkout 4c3d026ea64bd646a840bf174ef9d1818f10c20f python3 setup.py bdist_wheel --use-cxx11-abi ``` 7. Rebuild the Triton package (it is necessary to specify the GCC version within the conda environment, otherwise a libstdc++ version mismatch issue will occur). Alternatively, you can temporarily change it using the following method: ``` git clone https://github.com/triton-lang/triton.git; cd triton; pip install ninja cmake wheel pybind11; # build-time dependencies pip install -e python --verbose mv /ssd/miniconda3/envs/fish-speech/lib/libstdc++.so.6 /ssd/miniconda3/envs/fish-speech/lib/libstdc++.so.6.old ln -s /usr/lib/aarch64-linux-gnu/libstdc++.so.6 /ssd/miniconda3/envs/fish-speech/lib/libstdc++.so.6 ``` If `triton.common` is not found, switch to the `2.3.x` branch. ``` git switch release/2.3.x ``` 8. Triton 3.0 requires Torch 2.5 to function, but the official release for Jetson AGX Orin currently only provides 2.4. With Torch 2.3, using Triton 3.0 results in a missing triton.common error. Torch 2.3 requires Triton 2.3.x, but the Triton 2.3.x branch does not support AArch64. 9. After successfully building Torch, the torchaudio library must be rebuilt. [link](https://pytorch.org/audio/stable/build.jetson.html) ``` USE_CUDA=1 python setup.py bdist_wheel ``` 10. After successfully building Torch, rebuild the torchvision library. ``` git clone https://github.com/pytorch/vision.git python setup.py bdist_wheel ``` 11. https://github.com/triton-lang/triton/issues/3505 --- # Jetson AGX Orin The key to the `LLM` model's inference speed lies in memory bandwidth. Assuming each token is queried once during inference. For models like NousResearch/Llama-2-7b-chat-hf, the official release is in fp16 (model size: 14G). When quantized to int8, the size reduces to 7G, achieving a speed of approximately 200/7 ~ 30 tokens/s. The Jetson AGX Orin features two PCIe 4.0 x8 channels. According to online data, the bandwidth of a PCIe 4.0 x8 channel is approximately 16 GB/s. --- You can use the Jetson officially packaged text-generation-webui to test LLMs. ```sh jetson-containers run dustynv/text-generation-webui:r36.2.0 ``` `r36.2.0` It is the latest provided version, but the `llama.cpp python` in it is outdated, and during loading of the `llama3.1` and `70b` models, an `Attributte: model` error occurs. Rebuild the `r36.2.0` AI Pod `llama.cpp python whl` ```sh pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple sed -i 's@//.*ports.ubuntu.com@//mirrors.ustc.edu.cn@g' /etc/apt/sources.list apt update && apt install vim pip install poetry cd /opt/llama-cpp-python make update.vendor make build.cuda ``` After construction, you can use the following command to build the whl package, making it easy to install in other locations. ``` CMAKE_ARGS="-DGGML_CUDA=on" python3 -m pip wheel --verbose -e . ``` --- You can use the Jetson officially packaged MLC to test the speed of MLC LLM. ```sh jetson-containers run dustynv/mlc:0.1.1-r36.3.0 ``` Or directly use the docker command to bind the corresponding location. ``` cd /ssd/lnks/jetson-containers docker run -d --runtime nvidia -it --rm --network host --shm-size=8g -v `pwd`:/workspace --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /ssd/lnks/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-3 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-6 --device /dev/i2c-7 --device /dev/i2c-8 dustynv/mlc:0.1.1-r36.3.0 cd /workspace/mymodels/mlc-ai mlc chat Qwen2.5-72B-Instruct-q4f16_1-MLC ``` # Configuration Issues When loading the `mlc Qwen2.5-XB` model downloaded from Hugging Face, errors such as `mean_gen_len` or `isArray` occur. These can be resolved by adding missing fields to the `mlc-chat-config.json` file. ``` "shift_fill_factor": 0.3, "mean_gen_len": 128, "max_gen_len": 512, ... "system_prefix_token_ids": [], ``` # Test Using the Official Benchmark Script ``` root@nvidia-desktop:/workspace/packages/llm/mlc# python3 benchmark.py --model /workspace/mymodels/mlc-ai/Qwen2.5-32B-Instruct-q4f16_1-MLC/ --max-new-tokens 128 --max-num-prompts 4 --prompt /data/prompts/completion_16.json --save /workspace/myresults/mlc.csv ``` # Comparison of mlc_llm versions in MLC and issue with Qwen2-1.5B: lm_head.weight missing https://github.com/mlc-ai/mlc-llm/issues/2458 ``` ln -s /usr/bin/python3 /usr/bin/python cd /opt/mlc-llm git pull origin main # create build directory mkdir -p build && cd build # generate build configuration python ../cmake/gen_cmake_config.py # build mlc_llm libraries cmake .. && cmake --build . --parallel $(nproc) && cd .. ``` It seems we still can't resolve this issue. --- # whisper ``` providers = [ ('TensorrtExecutionProvider', { 'device_id': 0, # Select GPU to execute 'trt_max_workspace_size': 2147483648, # Set GPU memory usage limit 'trt_fp16_enable': True, # Enable FP16 precision for faster inference 'trt_dla_enable': True, 'trt_int8_enable': True, }), ('CUDAExecutionProvider', { 'device_id': 0, 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_mem_limit': 2 * 1024 * 1024 * 1024, 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'do_copy_in_default_stream': True, }) ] ``` ```python import torch import torchaudio import numpy as np import onnxruntime as ort from scipy.io import wavfile from transformers import WhisperProcessor, WhisperTokenizer import os # 加载 ONNX 模型 sess_encoder = ort.InferenceSession("/ssd/lnks/onnx/turbo-encoder.onnx", providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider']) sess_decoder = ort.InferenceSession("/ssd/lnks/onnx/turbo-decoder.onnx", providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider']) os.environ['all_proxy'] = 'http://127.0.0.1:1080' processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3-turbo") tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-large-v3-turbo") audio_path = "/ssd/lnks/assets/speech.wav" # 替换为您的音频文件路径 sample_rate, waveform = wavfile.read(audio_path) waveform = waveform.astype(np.float32) / 32768.0 # 归一化到 [-1, 1] # 确保音频是单声道的 if len(waveform.shape) > 1: waveform = np.mean(waveform, axis=1) # 重采样到16kHz if sample_rate != 16000: waveform = torchaudio.functional.resample(torch.from_numpy(waveform), sample_rate, 16000).numpy() # 使用处理器准备输入 input_features = processor(waveform, sampling_rate=16000, return_tensors="np").input_features out_encoder = sess_encoder.run(None, {"mel": input_features}) tokens = list(tokenizer.sot_sequence_including_notimestamps) session = ort.InferenceSession("/ssd/lnks/onnx/model.onnx", ) # 运行推理 output = session.run(None, {"x": input_features})[0] # 解码输出 predicted_ids = np.argmax(output, axis=-1) transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0] # 使用示例 result = transcribe_audio(audio_path) ``` --- # qwen-agent 1. Load LLM model using ollama (all models are quantized to 4bit) 2. Test document: Archimedes' Retaliation.pdf 3. Set context length of qwen2.5 to 32k in ollama (qwen2.5 supports 128k context length) PDF pages: 124 PDF parsing time: 25.250 seconds ![qwen agent assistant](../../olddocs/qwen-agent-assistant.png) ![qwen agent assistant 3b](../../olddocs/qwen-agent-assistant2.png) ![qwen agent assistant 3b](../../olddocs/qwen-agent-assistant3.png) qwen-agent assistant: fast parsing qwen-agent docqa: higher parsing cost but higher accuracy, suitable for long-document question answering | Model | Solution | Parse the total number of tokens in the PDF | Inference time | Is the answer correct | |:-----------------|:---------------------|-------------------|:---------|--------------| | qwen2.5:0.5b 32k | qwen-agent assistant | 128829 (approximately 130,000) | 15s | No | | qwen2.5:0.5b 32k | qwen-agent docqa | 149482 (approximately 150,000) | 150s | No | | qwen2.5:1.5b 32k | qwen-agent assistant | 128829 | 20s | Yes | | qwen2.5:1.5b 32k | qwen-agent docqa | 149482 | 160s | No | | qwen2.5:3b 32k | qwen-agent assistant | 128829 | 21s | Yes | | qwen2.5:3b 32k | qwen-agent docqa | 149482 | 230s | Yes | | qwen2.5:14b 32k | qwen-agent assistant | 128829 | 50s | Yes | | qwen2.5:14b 32k | qwen-agent docqa | 149482 | 750s | Yes | ## Infer multi-language keywords from query information qwen2.5:0.5b: {"keywords_zh": ["伏尔泰", "vulgate"], "text": "伏尔泰\n在书中说了"} qwen2.5:1.5b: {"keywords_zh": ["伏尔泰书", "伏尔泰的书籍"], "keywords_en": ["Voltaire's book"], "text": "伏尔泰书中的内容是什么"} qwen2.5:3b: {"keywords_zh": ["伏尔泰", "书", "话", "内容"], "keywords_en": ["Voltaire", "book", "sentence", "content"], "text": "伏尔泰在书中说了什么话?"} qwen2.5:14b: {"keywords_zh": ["伏尔泰", "书", "说话", "内容"], "keywords_en": ["Voltaire", "book", "quote", "content"], "text": "伏尔泰在书中说了什么话"} When performing RAG, it's necessary to adjust the model's context length; otherwise, the results will be inaccurate. During testing, we found that qwen2.5:14b fails to answer correctly when using Ollama's default 2048-context setting, but answers accurately when using a 32k context. The larger the LLM, the more closely its description of keywords (or the refined results) aligns with the actual question, leading to more precise matching results. [Poor QA performance when using Ollama's local service](https://github.com/netease-youdao/QAnything/blob/qanything-v2/FAQ_zh.md#%E4%BD%BF%E7%94%A8ollama%E6%9C%AC%E5%9C%B0%E6%9C%8D%E5%8A%A1%E6%97%B6%E9%97%AE%E7%AD%94%E6%95%88%E6%9E%9C%E4%B8%8D%E4%BD%B3) --- # AI Pod Evaluation 1. Document Translation 1. Summary 2. Translate books 3. Translate subtitles 4. Phone call transcripts 2. Cameras 1. Semantic understanding 2. 3. Smart Speakers 1. Text-to-speech synthesis 2. Multi-modal model 4. Video Search 1. Context-aware semantic understanding 5. Cross-border Communication 1. Real-time translation 2. Voice synthesis (synthesize speech using custom voice from translated text) 6. Web-to-Podcasting 1. Content recognition 2. Text-to-speech 7. RAG Full-Text Search 1. Initially support similarity search 2. Build upon similarity search to enable document retrieval, summarization, and document comprehension - [x] 2024-09-18 11:35:35 Reviewed Jetson documentation; identified useful components - [] 2024-09-24 16:12:43 When using multiple models' engines simultaneously, is there power limitation leading to performance degradation? - [] fish-speech model --- # fish-speech | Model | Text | Output sound duration | Inference time | Ratio (how much audio can be generated per 1s of inference time) | |:------------|:---------------------|:-------------|-----------|--------------------------------------| | fish-speech | The Big Goose Story (2522 characters) | 9:10 | 37m8.903s | 0.24 | # f5-tts Memory requirement: 4GB (so a card with 8GB of memory can also run this model) The model code is under the MIT open-source license, but the model weights are under CC-BY-NC. Supports adding pauses using punctuation or whitespace—this feature is not available in fish-speech. You can specify an example audio file to clone voice characteristics. | model | Text | Output sound duration | Inference time | Ratio (how much audio can be generated per 1s of inference time) | |:-------|:---------------------|:-------------|----------|--------------------------------------| | E2-TTS | The White Goose Story (2522 characters) | 8:44 | 23:50 | 0.36 | | F5-TTS | The Big White Goose Story | 8 minutes 44 seconds | 17:3 | 0.51 | ``` conda activate f5-tts cd /ssd/lnks/tts/F5-TTS bash example/run2.sh ``` `E2-TTS` sounds more expressive and emotional compared to `F5-TTS`, while the voice quality of `F5-TTS` is roughly on par with that of `fish-speech`. # chattts # Test file: The Story of the Big Goose ```txt 一 从前，有一只胖胖的母鹅。她已经到了做妈妈的年龄，可是她一直没做成妈妈。因为，这只鹅从来没有下过一只蛋，没有孵出过一只鹅来。 可怜的胖鹅，她是一只孤单单的鹅。 她多么希望能孵出一只胖胖的小鹅，听小鹅脆生生地叫她一声妈妈。她多么希望街坊邻居也能尊敬地叫她一声鹅妈妈。 可这一切离开她是多么的遥远，简直不可能。每当夜深人静的时候，胖鹅常常会独自流下伤心的泪，孤独的泪。 一天，胖鹅告别了乡亲们独个儿出外旅行去了。 胖鹅在蜿蜒的小路上，步履蹒跚地走着。她在晶亮的小河上，划动着一双红红的脚掌，体态悠闲地游着。 她看看山，看看水，看看变幻莫测的白云，看看盛开在原野、田梗上的小花。美丽的风景慰藉了她孤独的心灵，胖鹅渐渐变得开朗和愉快起来。 一天傍晚的时候，胖鹅爬上了河岸，她在田埂上一摇一摆地走着。田埂边上，是一片已经收获过的瓜田。瓜田里除了爪藤外，已经看不到西瓜了。 胖鹅走着走着，忽然她看见在几张瓜叶前，有半个看来是被刺猬啃破的西瓜，就在这只被啃破的西瓜对面，居然有个十分完整的圆西瓜。胖鹅高兴极了。她想正好用这只西瓜来解旅途的饥渴。 胖鹅抱住这只西瓜，不知为什么，她有一种感觉，觉得这只西瓜不同于一般的西瓜，它像一只蛋，一只圆圆的、硬硬的有生命的蛋。 也许，胖鹅太想有一个蛋了，她说什么也摆脱不了这个奇怪的想法，她认定这只西瓜是一只蛋。 胖鹅小心翼翼地，把这只“西瓜蛋”搬到了不远处，那儿有看瓜人留下的破草棚。胖鹅在草棚里，她围着“西瓜蛋”转了几圈，她毫不怀疑自己的眼光，这一定是个“西瓜蛋”。胖鹅细心地伏在“西瓜蛋”上，她满怀着热情和信心，在这个破草棚里孵起蛋来了。 胖鹅的体温，渐渐传到了“西瓜蛋”里。 二 一天、两天、三天、五天。 西瓜并没有被胖鹅孵烂。 恰恰相反，“西瓜蛋”的壳硬得很，只是变得很暖和很暖和了。 八天、九天、十天，胖鹅似乎感觉到“西瓜蛋”有了动静，她已经能听到嚓嚓的声响了。 “天哪，我的宝宝快出壳了，我好像听到它在啄蛋壳了！”胖鹅喃喃自语地说着，“我要当妈妈了，我要让宝宝自己啄破蛋壳，当它的可爱的小脑袋从破蛋壳里伸出来时，我一定要亲亲它。” 又过了几天，只听到格拉一声响，“西瓜蛋”的壳破了一个洞——只见一只小秃脑袋，从这个“窗户”里伸了出来，他长着一只尖尖的嘴，他张开一对小小的眼睛，惊奇地注视着这个有趣的世界。 胖鹅激动地伸长脖子，去亲了亲这个小秃脑袋一下，小东西吓得把头缩进了蛋壳。 由于小东西一惊慌，“西瓜蛋”格拉格拉一阵响，整个儿破了。这只被胖鹅从“西瓜蛋”里孵出来的，不知道姓名的小怪物，降临到这个世界上了。 胖鹅当妈妈了，她终于有了自己的孩子。 三 鹅妈妈给自己的孩子取了个名字，叫“小尖嘴”。小尖嘴出壳不久，就会站立，就会走路，还会发出呱呱叽叽的声音。这真是个讨人喜欢的小宝宝。 鹅妈妈是这样的疼爱自己的孩子。 她给小尖嘴找东西吃，她帮小尖嘴梳理几根刚出来的羽毛，还给他讲好听的故事。 鹅妈妈带着她的孩子回到了家乡，街坊邻居不再叫她胖鹅，而改口叫她鹅妈妈了。你说，鹅妈妈心里有多甜蜜——她终于有了孩子，她是个十足的妈妈了。 可惜的是她的孩子，长得越来越不像她。 小尖嘴的腿越长越长，以至于他看自己妈妈的时候，必须低下头来。他的脖子也越长越长，小小的脑袋、尖尖的嘴儿，显得有几分滑稽。 街坊们议论鹅妈妈的儿子，说他不像鸡，不像鸭，也不像鹅。 每当听到这样的话，鹅妈妈就有点不高兴，她会大声地说：“可他像我的儿子。不，就是我的儿子，他多有男子汉的气概！” 鹅妈妈几次想教他的宝贝游泳，不过都没有成功，小家伙站在小河里，河水刚浸到他的肚皮，可是当他蜷起腿来，像他的鹅妈妈那样游泳，就几乎被水呛死。 这是鹅妈妈最伤心的——她的儿子没法学会游泳。 四 有一天，鹅妈妈闲来无聊，因为儿子出门玩去了，她买来一本刚出版的杂志读着。它刊登来自世界各地的新闻和趣事。其中有一篇旅行家爱尔莎太大的回忆文章，吸引了她。 有一次爱尔莎太太和丈夫画家雷诺先生出门旅行。他们在一片异国的瓜田里走着。爱尔莎太太文章里有这样一段： “走着，走着，我的肚子忽然剧痛起来，我当时怀着孕，我就在瓜田里下了我这辈子的第一个，也是唯一的一个蛋，那是一个圆滚滚的，非常惹人喜爱的大蛋。” “可是，我的丈夫雷诺，他看见瓜田里有个可爱的小西瓜，突然画兴大发，他要让我们的孩子知道，他是降生在瓜田里的，雷诺要在蛋上留下一张有意义的写生画。于是，他就挤出了颜料，把对面西瓜的色彩画在了我刚下的那只可爱的蛋上，我的蛋就变成了一只活灵活现的小西瓜了。” “就在这时，”爱尔莎太太伤心地说，“突然，从远处跑来了一个黑点儿，根据我们在草原上生活多年的经验，可以断定，那是一只狼。我们不得不暂时躲一躲，撒腿离开了瓜田。” “过了好久，当我们回来时，瓜田里只剩下半只被啃过的西瓜，我们被画成西瓜的蛋，无影无踪了。” 爱尔莎太太在文章最后说：“我抬头四望，田野上除了一个看瓜人留下的破草棚，什么也没有，我伤心透了——我的唯一的孩子，就这样失去了⋯⋯” 看到这里，鹅妈妈眼前一阵黑。这场面她太熟悉了，鹅妈妈想说：“爱尔莎太太，你当时为什么不到草棚里来看看呢？可怜的人。” 鹅妈妈再看看文章后面，那里印着爱尔莎太太的照片，原来她和雷诺先生是一对驼鸟夫妻，他们的家乡在澳大利亚。 瞧，她那高高的腿，她那长长的脖子和小脑袋，这不和小尖嘴一模一样吗？ “天哪。”鹅妈妈说，“我的小尖嘴是只小驼鸟。” 五 接下来的日子，是鹅妈妈一生最难忘的日子。她翻来覆去地想这件事。 鹅妈妈想把小尖嘴留在身边，她完全可以隐瞒这件事。是她从“西瓜蛋”里孵出了小尖嘴，是她喂养大了小尖嘴。也是小尖嘴给了她妈妈的称呼，赶走了她的孤单和寂寞。她怎么可以让小尖嘴离开自己，去到一个遥远而又陌生的地方，管一对陌生的夫妻，叫做爸爸妈妈呢⋯⋯ 可她再一想，小尖嘴毕竟是一只小驼鸟，他是个学不会游泳的小家伙，他的天地是在草原上，他会成为鸟类中的长跑冠军。 鹅妈妈想象着，小尖嘴张开翅膀，迈开长腿，像一匹骏马似地在草原上奔驰着，这多威风，多有男子汉气概⋯⋯ “唉，这才像个驼鸟。”鹅妈妈叹口气说，“要是老在我这儿，准会变得鹅不像鹅，驼鸟不像驼鸟的。” 鹅妈妈的心里乱极了。 六 电报毕竟还是发出去了。 “孤独”，这难忘的滋味，让鹅妈妈下定了决心。鹅妈妈想起，小尖嘴没来到她身边以前，她是多么的孤独，多么的寂寞啊。是小尖嘴给她带来了希望、愉快和热热闹闹的生活。 可是如今呢，鹅妈妈说：“我把‘孤独’送给了爱尔莎太太和她的丈夫了，这多么自私。” 是的，鹅妈妈早就从爱尔莎文章的字里行间，读到了孤独，读到了寂寞，读到了懊恨和思念⋯⋯反正，读到了一颗妈妈的破碎的心。 鹅妈妈在给爱尔莎太太的电报中写着：“我再也不会孤独了，因为我有过小尖嘴这个宝贝，他让我成了妈妈，他会永远记住我的。他在我心中，我也在他的心中，他应该成为我们共同的儿子。” 写到这里，鹅妈妈好像看见了爱尔莎太太和雷诺先生喜出望外的模样。 不知怎么的，鹅妈妈突然在电报的最后，想出了一句只有诗人才会想出来的话： “让孤独永远从我们中间离开吧！” 七 如今，当人们敲敲鹅妈妈的门，而一直没有动静的时候，你就会知道—— 鹅妈妈是在给大草原上的小尖嘴写信，而忘了周围的一切。 要么，就是她出远门，去探望那只当年她从“西瓜蛋”里孵出的，如今奔驰在澳大利亚大草原上的小尖嘴了⋯⋯ ``` --- Currently, most video search methods rely on frame extraction from videos. A simple approach is to directly feed the extracted images and user queries into a vision-language model (VLM). A slightly more advanced method includes not only the images but also incorporates timestamp information, allowing the model to return the corresponding video time points when queried. However, this approach cannot handle very long videos due to memory limitations. The context length becomes too large, causing memory overflow—this is precisely why most current vision models cannot process long videos. [Vlog: Video as a Long Document](https://github.com/showlab/VLog?tab=readme-ov-file) addresses this by first transcribing video segments into text: speech is transcribed into text via speech recognition, and image frames are converted into text using image description models. The entire long video is thus transformed into a long document. When a user queries, it operates similarly to RAG (Retrieval-Augmented Generation), retrieving relevant data from a database and summarizing it. Another approach is designed for long videos, especially surveillance videos. In such scenarios, most scenes show no movement of objects, resulting in very few key frames. This redundancy can be significantly reduced, greatly alleviating context overload. [LVNet](https://github.com/showlab/VLog?tab=readme-ov-file) is designed from the perspective of reducing information redundancy to solve context overflow. However, optimizations can be tailored to specific scenarios. For example: ## Scenario 1: Home monitoring — After work, asking the AI, "What happened at home today?" In the scenario of home monitoring, video length is unpredictable and typically long. If all video content were fed into vision-language models (e.g., Qwen2-VL, LLaVA), it would lead to memory overflow. For this use case, I believe the following approach can be adopted: 1. During real-time monitoring, use a lightweight, mature model (such as YOLO for object detection or a model designed for monitoring human activities) to detect human presence and actions (e.g., entering, sitting down, standing up), recording these events. 2. When monitoring activities, use audio recognition models to capture and analyze sounds at specific time points. 3. Store all detected events and data in a database. 4. When users query, retrieve relevant data from the database and summarize it—this approach mirrors the functionality of RAG (Retrieval-Augmented Generation). In this way, high computational power is not required for home monitoring. [ChatVideo](https://www.wangjunke.info/ChatVideo/) implements a similar approach. ![ChatVideo](../../olddocs/ChatVideo.png) ## Scenario 2: Video Search For video search—similar to image search—there's no need to feed all frames into a large language model for understanding. Instead, frames can be extracted, and each frame is encoded into a feature vector and stored in a database. Models like CLIP or other image encoding models can be used for this purpose. ## Scenario 3: Video Understanding Similar to Hongmeng XiaoYi, such as understanding the current screen (a single image) or a short video segment, these can be achieved using vision-enabled large models (e.g., Qwen2-VL). Internally, the model processes video frames and feeds the resulting images into the large model for interpretation. After understanding the content, the model can answer questions based on the image content. ## Demo: Multimodal-RAG-on-Jetson 1. Extract audio from video using ffmpeg 2. Slice the audio into segments and use `whisper turbo` to recognize the text. 3. Extract frames from the video, then use `ollama llava-llama3` to summarize the content of the images, and save the output text. 4. Then perform vectorization using `bge-m3`. 5. Finally, use `llama-index` together with `llava-llama3` to perform `RAG` search FFmpeg on Jetson needs to be compiled separately to utilize Jetson's hardware acceleration [jetson-ffmpeg](https://github.com/Keylost/jetson-ffmpeg) ## Qwen2-VL Model Testing A 12-second video, processed at 1 fps, results in 12 frames. These frames are fed into Qwen2-VL-7B-Instruct for analysis. A summary of the video content is then generated based on the processed frames. The model can only summarize or understand the content present in each frame of the video, without specifying what object or feature appeared at a particular second. | Model | frames per second | Encoding time | Inference time | |:---------------------|:-----|----------|:---------| | Qwen2-VL-7B-Instruct | 12 | 1.35s | 13s | | Qwen2-VL-2B-Instruct | 12 | 1.15s | 6.92s | # Qwen2-VL-7B-Instruct Test Output This video shows a busy urban street scene. In the scene, there are many vehicles on the road, including cars, motorcycles, and bicycles. On both sides of the street, there are trees and buildings, indicating a city environment. The video also features two shared bicycles, which are yellow and blue, parked along the roadside. # Qwen2-VL-2B-Instruct Test Output In the video, for a brief moment, two shared bicycles appear. --- # Ollama API The AI Pod comes pre-deployed with `Ollama`, allowing applications within the LCMD to directly use the exposed `Ollama API`. - `ollama` endpoint is at: - `ollama` compatible with `OpenAI API` endpoint is at: If you have the LCMD client installed locally, you can use the above endpoints to call the service from your local applications. For example, you can directly use curl to access the endpoint. ```sh curl https://ollama-ai.${微服名称}.heiyu.space/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "qwen3:4b", "messages": [{"role": "user", "content": "Hello!"}]}' ``` The AI Pod's `ollama` port is also exposed on the local network. Access `http://算力舱ip:11434` to use the `ollama` service. You can find the AI Pod's local network IP address in the device list [here](/en/os/introduction.html#ssh). ## Python example ```python import base64 import requests import json # 获取图片并进行 base64 编码 image_url = "https://i-blog.csdnimg.cn/direct/7184675ec6ad495e8b7469c17edd56d9.png" try: response = requests.get(image_url) response.raise_for_status() # 检查请求是否成功 image_data = response.content encoded_image = base64.b64encode(image_data).decode('utf-8') except requests.exceptions.RequestException as e: print(f"Error downloading image: {e}") exit(1) # 构建请求的 JSON 数据 request_data = { "model": "qwen3-vl:2b-instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "提取文字为 Markdown 格式，要求格式正确。" }, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{encoded_image}" } } ] } ] } ollama_api_url = "https://ollama-ai.${微服名称}.heiyu.space/v1/chat/completions" headers = { "Content-Type": "application/json" } try: response = requests.post(ollama_api_url, headers=headers, data=json.dumps(request_data)) response.raise_for_status() # 检查请求是否成功 # 打印响应 print(response.json()) except requests.exceptions.RequestException as e: print(f"Error sending request to Ollama API: {e}") ``` --- # Benchmark | Model Alias | Actual Model | Tokens/s | Notes | |:----------------|:--------------------|:---------|:------| | qwen3-coder:30b | - | 31.84 | - | | qwen3.5:9b | qwen3.5:9b-q4_K_M | 15 | Tool-calling accuracy has been improved compared to Qwen3. | | qwen3:35b-a3b (262k) | qwen3.5:35b-a3b-q4_K_M | 25 | Manually adjust the context length from 8K to 262K; tool calling capability is stable; VRAM usage remains consistently between 24.4G and 30G. | --- # How to Delete Models in Ollama ## Delete Model in Open WebUI ![open webui delete model](../../ollama/images/openwebui-delete-model.png) 1. Click the icon in the bottom-right corner and select "Admin Settings" from the dropdown menu. 2. In the "Admin Settings" section, click the "Settings" tab. 3. Go to the "Model Management" section. 4. Select the model you want to delete. 5. Click the "Delete Model" button. ## Delete Model in Webollama ![webollama delete model](../../ollama/images/webollama-delete-model.png) 1. Click the model list to view currently available models. 2. Select the model you want to delete and click the delete icon on the right side. ## Delete Model via Ollama Command Line 1. SSH into the AI Pod, for example: `ssh nvidia@192.168.1.130` 2. Execute: `docker exec -ti cloudlazycataipodwebollama-ollama-1 bash` 3. Execute: `ollama ls` to list all currently available models ```bash NAME ID SIZE MODIFIED qwen3-vl:8b 901cae732162 6.1 GB 37 minutes ago qwen3:30b-a3b-instruct-2507-q4_K_M 19e422b02313 18 GB 5 hours ago qwen3-vl:30b eda0be100877 19 GB 25 hours ago phi3:14b cf611a26b048 7.9 GB 3 weeks ago qwen3:4b-instruct-2507-q4_K_M 088c6bc07f1d 2.5 GB 5 weeks ago qwen3:4b e55aed6fe643 2.5 GB 2 months ago qwen3:14b 7d7da67570e2 9.3 GB 6 months ago qwen3:8b e4b5fd7f8af0 5.2 GB 6 months ago qwen3:1.7b 458ce03a2187 1.4 GB 6 months ago ``` 4. Execute the model you want to delete, for example: `ollama rm qwen3:1.7b` ![ollama cli delete model](../../ollama/images/ollama-cli-delete-model.png) --- # Adding a Model to Ollama Open `https://ollama.com/search` to check the models supported by `ollama`, and select a model you'd like to pull. For example, I want to pull the `deepseek-r1:70b` model. ## Pulling a Model via Open-WebUI - Install the Open WebUI application; [click to jump](https://appstore.lazycat.cloud/#/shop/detail/cloud.lazycat.aipod.openwebui) - Click the model name in the upper-left corner of `Open-WebUI`. This will open a dropdown menu allowing you to switch models. ![Open-WebUI model switch dropdown](../../ollama/images/Open-WebUI-switch-model-dropdown.png) - When the dropdown list does not contain the model you need, manually enter a `deepseek-r1:70b` in the search box. The interface will then prompt: "Pulling deepseek-r1:70b from Ollama.com" ![Open-WebUI pulling model from Ollama](../../ollama/images/Open-WebUI-pull-model-from-ollama.png) - Click "Pulling deepseek-r1:70b from Ollama.com", which will initiate the background model download. The interface will also display the progress. ![Open-WebUI model pull progress](../../ollama/images/Open-WebUI-pull-model-progress.png) ## Pulling a Model via Ollama Command Line - First, you need to log into the AI Pod system via SSH or directly through the display screen. - Execute `docker exec -ti cloudlazycataipodwebollama-ollama-1 bash` to enter the container `ollama` - Execute `ollama pull deepseek-r1:70b` accordingly ## Pulling models with webollama 1. Install the `webollama` application, [click to jump](https://appstore.lazycat.cloud/#/shop/detail/cloud.lazycat.aipod.webollama) 2. After opening, you will see the following interface. Follow the prompts to pull the model. ![webollama-pull-model.png](../../ollama/images/webollama-pull-model.png) When using `webollama` to pull the model, no progress is shown. Please wait patiently (it is recommended to use the above two methods to pull the model, as they allow you to monitor the progress). --- # Single Usage You can deploy services independently within an AI Pod using docker-compose. Please select the corresponding configuration based on the AI Pod model. ## AGX Orin ```yml services: ollama: image: registry.lazycat.cloud/catdogai/jetson-ollama:0.17.5 volumes: - /etc/timezone:/etc/timezone:ro - ./data:/root/.ollama environment: - LANGUAGE=en_US:en - LANG=en_US.UTF-8 - LC_ALL=en_US.UTF-8 - OLLAMA_DEBUG=1 - OLLAMA_ORIGINS=* - OLLAMA_HOST=0.0.0.0 - OLLAMA_FLASH_ATTENTION=1 - OLLAMA_NUM_PARALLEL=4 - OLLAMA_CONTEXT_LENGTH=8192 - OLLAMA_KV_CACHE_TYPE=q8_0 - OLLAMA_LLM_LIBRARY=cuda_jetpack6 - LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_jetpack6:/usr/local/cuda/compat:/usr/local/cuda/lib64 ports: - 11434:11434 mem_limit: 60G memswap_limit: 60G ``` ## Thor ```yml services: ollama: image: registry.lazycat.cloud/catdogai/jetson-ollama:0.17.5 volumes: - /etc/timezone:/etc/timezone:ro - ./data:/root/.ollama environment: - LANGUAGE=en_US:en - LANG=en_US.UTF-8 - LC_ALL=en_US.UTF-8 - OLLAMA_DEBUG=1 - OLLAMA_ORIGINS=* - OLLAMA_HOST=0.0.0.0 - OLLAMA_FLASH_ATTENTION=1 - OLLAMA_NUM_PARALLEL=4 - OLLAMA_KV_CACHE_TYPE=q8_0 - LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/cuda/lib64:/usr/local/cuda-13.0/lib64:/usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu ports: - 11434:11434 mem_limit: 60G memswap_limit: 60G ``` Save the configuration for the corresponding model to `docker-compose.yml`, then start it using `docker-compose up -d` in the same directory. After successful startup, access it via the `11434` port on the local network. The latest version of `ollama` automatically sets the context length based on available memory. For the AGX Orin AI Pod, with 64G of memory, the default context length may be large. For example, use `OLLAMA_CONTEXT_LENGTH=8192` to limit the context length to `8k`. ::: info Info If you cannot access `registry.lazycat.cloud` on your AI Pod, you can switch to `dockerhub` and access it via `docker pull catdogai/jetson-ollama:0.17.5`. Ollama frequently updates, and the documentation may not be updated in time. The latest Ollama version tags can be found on the `https://hub.docker.com/r/catdogai/jetson-ollama/tags` page. --- # Cleaning Data When the `lzc-ai-agent` service starts with LCMD, it automatically removes outdated Docker container images. After you uninstall the corresponding `AI应用` from LCMD, the AI Pod automatically cleans up its container images and model data. ## Clean Up Old AI Application Container Images The system intelligently scans the installation location of your services to identify and remove unused old versions of container images. ```sh /ssd/lzc-ai-agent/bin/client.arm64 -clean-lzc-images ``` ## Clean Up Container Abnormal States The following do not affect normal system operation ```sh docker ps -a | grep '[Dead|Exited]' | awk '{print $2}' | xargs -i docker rm {} ``` ## Clean Up Container Images ::: warning Warning This command does not affect the currently running system. Running the following command will clean up unused images. Docker will automatically skip any image that is being used by a container. However, note that locally built or pulled images not currently in use will also be deleted. ```sh docker images | awk '{print $2}' | xargs -i docker rmi {} ``` ## Docker Cleanup Clean Docker cache and unused data ```sh docker system prune ``` --- # Hard Drive The AI Pod comes with a solid-state drive in version `1T` by default. However, the AI Pod has two `pcie4` interfaces, allowing you to replace the hard drive according to your needs. ## Replacing the Hard Drive 1. Copy the data from the removed hard drive to the new one (the original drive contains some model data and the AI Pod client). 2. Replace the new hard drive. ::: warning Warning When replacing the hard drive, ensure that the mount path in `/etc/fstab` is correctly configured; otherwise, the system will not boot properly. The following is an example, where the solid-state drive is mounted to the `/ssd` directory. ``` UUID=942a4d75-e509-4fa2-8d3a-23cb6925b68c /ssd ext4 defaults 0 2 /ssd/home /home none bind 0 0 ``` ## Migrate the System to SSD In the current AI Pod system, the system is running on a 64G `emmc` storage, while `docker` and `home` are on `SSD` storage. You can follow the commands below to migrate the system to `SSD`. 1. /ssd Directory This directory is the mounted path for the current 1T SSD drive. You can see directories such as `docker`, `home`, and `lzc-ai-agent` here. 2. Copy the current system to the NVMe drive. Assume your NVMe drive is nvme0n1p1. ``` sudo mount /dev/nvme0n1p1 /mnt sudo rsync -axHAWX --numeric-ids --info=progress2 --exclude={"/dev/","/proc/","/sys/","/tmp/","/run/","/mnt/","/media/*","/lost+found", "/ssd/", "/home/"} / /mnt sudo mkdir -p /mnt/ssd # 迁移之前的 /ssd 到新的目录 sudo systemctl stop lzc-ai-agent sudo systemctl stop docker systemctl stop nvzramconfig systemctl disable nvzramconfig mkdir -p /mnt/ssd mv /mnt/docker /mnt/ssd/ mv /mnt/lzc-ai-agent /mnt/ssd/lzc-ai-agent ``` 3. Modify `/etc/fstab` Comment out the lines related to /ssd/, and the updated `/etc/fstab` will look similar to the following text: ``` # /etc/fstab: static file system information. # # These are the filesystems that are always mounted on boot, you can # override any of these by copying the appropriate line from this file into # /etc/fstab and tweaking it as you see fit. See fstab(5). # # /dev/root / ext4 defaults 0 1 UUID=A9EE-ABCF /boot/efi vfat defaults 0 1 ``` 4. Write `/etc/systemd/system/setssdroot.service` ``` [Unit] Description=Change rootfs to SSD in M.2 key M slot (nvme0n1p1) DefaultDependencies=no Conflicts=shutdown.target After=dev-nvme0n1p1.device Requires=dev-nvme0n1p1.device Before=shutdown.target ConditionPathExists=/dev/nvme0n1p1 ConditionPathExists=/etc/setssdroot.conf ConditionVirtualization=!container [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/setssdroot.sh [Install] WantedBy=default.target ``` 5. Write `/sbin/setssdroot.sh` ``` #!/bin/sh # Runs at startup, switches rootfs to the SSD on nvme0 (M.2 Key M slot) NVME_DRIVE="/dev/nvme0n1p1" CHROOT_PATH="/nvmeroot" INITBIN=/lib/systemd/systemd EXT4_OPT="-o defaults -o errors=remount-ro -o discard" modprobe ext4 mkdir -p ${CHROOT_PATH} mount -t ext4 ${EXT4_OPT} ${NVME_DRIVE} ${CHROOT_PATH} cd ${CHROOT_PATH} /bin/systemctl --no-block switch-root ${CHROOT_PATH} ``` 6. Execute command ``` sudo chmod 777 /sbin/setssdroot.sh systemctl daemon-reload sudo systemctl enable setssdroot.service sudo cp /etc/systemd/system/setssdroot.service /mnt/etc/systemd/system/setssdroot.service sudo cp /sbin/setssdroot.sh /mnt/sbin/setssdroot.sh sudo cp /etc/fstab /mnt/etc/fstab sudo touch /etc/setssdroot.conf ``` `/etc/setssdroot.conf` is a marker file; the system will use the rootfs on NVMe only if this file exists. Therefore, if you need to switch to the `emmc` system, you can: - Unplug the NVMe drive - Manually mount the eMMC drive, then delete the `/etc/setssdroot.conf` file and restart 7. Restart to take effect ### Configure Swapfile After migrating the system, restart the system. Running lsblk will show that the system is now running on NVMe. Configure the swapfile. ``` fallocate -l 32G /swapfile mkswap /swapfile ``` Create a `/etc/systemd/system/swapfile.swap` file with the following content: ``` [Swap] What=/swapfile [Install] WantedBy=swap.target ``` Execute the following command to start it. ``` systemctl daemon-reload systemctl status swapfile.swap systemctl enable swapfile.swap systemctl start swapfile.swap ``` ## Fix apt man database error ``` # 修复权限 sudo chown -R man:root /var/cache/man sudo chmod -R 755 /var/cache/man # 重建数据库 sudo mandb ``` --- # Driver # Wi-Fi Adapter AW-XB560NF ``` sudo modprobe cfg80211 sudo insmod 8852ce_20250707.ko mv rtl8852bu_config rtl8852bu_fw rtl8852cu_config rtl8852cu_fw /lib/firmware sudo rmmod rtk_btusb sudo insmod rtk_btusb.ko ``` --- # LC-X3 Flashing Guide (Jetson AGX Orin / JetPack 7.2.1) Use `catdogai/lzc-x3:jetpack-7.2.1` to flash an NVMe system on an **LC-X3 / Jetson AGX Orin with a T912 v1.2 carrier board**. The image includes the Jetson Linux R39.2.1 BSP, vendor adaptations, flashing dependencies, a custom T234 partition XML, and preinstalled `lzc-jetson-orin-flash` 0.2.6. Once the container starts, you can run the flashing command. JetPack 7.2.1 corresponds to Jetson Linux R39.2.1. For version details, see the [NVIDIA JetPack Downloads page](https://developer.nvidia.com/embedded/jetpack/downloads). For USB flashing parameters, see the [R39.2.1 Quick Start](https://docs.nvidia.com/jetson/archives/r39.2.1/DeveloperGuide/IN/QuickStart.html) and [Flashing Support](https://docs.nvidia.com/jetson/archives/r39.2.1/DeveloperGuide/SD/FlashingSupport.html). The target system is based on Ubuntu 24.04. Other carrier boards require a matching vendor BSP. ::: danger Data will be erased The full flashing command in this guide includes `--erase-all`, which erases the device's storage. Back up your system, models, containers, and application data first. For the first flash, install only the target NVMe drive and disconnect other data drives. Subsequent provisioning, reset, and storage expansion operations also write to disk; verify the drive names and their purpose before proceeding. ::: ## 1. Prepare for Flashing - A native x86_64 Linux host, preferably Ubuntu 22.04, with Docker installed and permission to run privileged containers. The image architecture is **linux/amd64**, while the device rootfs inside it is ARM64. - At least **80 GB** of free host disk space is recommended for the image, the container's writable layer, and the generated 32 GiB system image. Temporary files are also created during generation. - An LC-X3 with a target NVMe drive of at least **128 GB** installed and a stable power supply connected. - A USB Type-C cable that supports data transfer, connecting the host to the LC-X3's Recovery port. Connect it directly to the host if possible. - A monitor and keyboard, or a serial console, for the first boot and troubleshooting LVM initialization. The container relies on the host's USB, network, and kernel, so this procedure is intended for native Linux. Connect only one device for your first flash. The host's `lsblk` output lists host drives; it cannot identify the Jetson's internal NVMe drive while the device is in Recovery mode. ## 2. Enter Force Recovery Mode When the device is powered on, hold **RECOVERY**, briefly press and release **RESET**, then release RECOVERY. The buttons are inside the enclosure; remove the bottom cover if necessary. When the device is powered off, hold RECOVERY, press POWER, then release RECOVERY. If the device can still enter UEFI, you can also press **Esc** at the NVIDIA logo during startup and look for **Device Manager → NVIDIA Configuration → Boot Configuration → Boot Into Recovery**. Enable it, save, and reboot. Menus vary by firmware version; use USB detection to confirm that the device is in Recovery mode. Check on the **flashing host**: ```bash lsusb -d 0955: ``` Common AGX Orin Recovery USB IDs are `0955:7023` and, for some modules, `0955:7223`. The device name usually contains NVIDIA or APX. For the corresponding module numbers, see the [official Quick Start](https://docs.nvidia.com/jetson/archives/r39.2.1/DeveloperGuide/IN/QuickStart.html#to-determine-whether-the-developer-kit-is-in-force-recovery-mode). If no device is detected, check the cable, port, power supply, and button sequence first. **USB Force Recovery** and **LZC Orin APP recovery**, described later, are different environments. The latter is the recovery system on the SSD. ## 3. Start the Flashing Container After obtaining the supplied offline archive `lzc-x3-jetpack-7.2.1.tar.zst`, import it on the **flashing host** (requires `zstd`): ```bash set -o pipefail zstd -dc lzc-x3-jetpack-7.2.1.tar.zst | docker load ``` Once the image is published to a registry, you can also obtain it with `docker pull catdogai/lzc-x3:jetpack-7.2.1`. After importing or pulling the image, start the container: ```bash mkdir -p ~/lc-x3-flash/workspace cd ~/lc-x3-flash docker run -it --rm \ --privileged \ --network host \ -v /dev/bus/usb:/dev/bus/usb \ -v /dev:/dev \ -v "${PWD}/workspace:/workspace" \ catdogai/lzc-x3:jetpack-7.2.1 \ bash ``` If you do not have Docker permissions, prefix the Docker commands with `sudo`. `--privileged` and the device mounts let the tool access the Recovery USB device and its device nodes after re-enumeration. `--network host` is used for initrd flashing communication. The container starts in `/Linux_for_Tegra` by default. NVIDIA binaries, vendor adaptations, the default user, and the Orin deb package are already prepared. **There is no need to rerun** `apply_binaries.sh` or `twsetup/install.sh`, or install flashing dependencies online. Check **inside the container**: ```bash cd /Linux_for_Tegra lsusb -d 0955: cat rootfs/etc/nv_tegra_release ls -l ../flash_l4t_t234_nvme.xml ../lzc-jetson-orin-flash_arm64.deb chroot rootfs dpkg-query -W lzc-jetson-orin-flash ``` You should see R39.2, REVISION 2.1, and Orin package version 0.2.6. The original deb package is kept in `/usr/local/share/jetson-orin-flash/`; `/lzc-jetson-orin-flash_arm64.deb` is a fixed link to it. ::: tip Save logs `/workspace` is mapped to the host directory, while `/Linux_for_Tegra` resides in the container's writable layer. `--rm` deletes the container and any generated flashing artifacts when it exits. Save any logs or files you want to keep in `/workspace`. ::: ## 4. Flash the NVMe Drive Run the following command **inside the container, in `/Linux_for_Tegra`**: ```bash sudo ./tools/kernel_flash/l4t_initrd_flash.sh \ --external-device nvme0n1p1 \ -c ../flash_l4t_t234_nvme.xml \ --erase-all jetson-agx-orin-devkit external ``` - `nvme0n1p1` is the target partition **on the Jetson**, not a host drive. - `../flash_l4t_t234_nvme.xml` points to the custom LC-X3 XML at the image's root directory. It retains the R39.2.1 T234 boot partitions, fixes APP at **32 GiB**, and adds **DATA_PV (partition ID 16)** for subsequent LVM setup. - The image already sets `jetson-agx-orin-devkit.conf`'s `ROOTFSSIZE` to **32GiB**, so this command does not need an additional `-S` argument. If you use the original BSP yourself, you must also specify `-S 32GiB` and install the matching vendor adaptations and deb package. - NVIDIA's original XML is at `tools/kernel_flash/flash_l4t_t234_nvme.xml`. Its partitions serve different purposes; keep the custom XML path above if you intend to use provision afterward. The template retains the `EXT_NUM_SECTORS` placeholder. By default, the BSP generates partitions using a 57 GiB external disk geometry. During provisioning, DATA_PV is expanded to the actual end of the SSD. The minimum 128 GB requirement provides space for a block-level clone of APP. This layout uses a single, unencrypted rootfs. Keep the power supply and USB cable connected during flashing. Wait for the tool to report success and the device to reboot. To save terminal output as well, run **the same command** in this form: ```bash set -o pipefail sudo ./tools/kernel_flash/l4t_initrd_flash.sh \ --external-device nvme0n1p1 \ -c ../flash_l4t_t234_nvme.xml \ --erase-all jetson-agx-orin-devkit external \ 2>&1 | tee /workspace/lc-x3-flash.log ``` ## 5. First Boot and LVM Initialization Run the following commands **on the LC-X3 system**. The default username is `nvidia` and the password is `nvidia`; change the password after logging in. Log in using a monitor, or connect over SSH using the device's LAN address. If USB networking is ready, you can also try `ssh nvidia@192.168.55.1`. First, wait for the vendor's `twInit.service` to finish initialization. The device may automatically reboot once: ```bash systemctl status twInit.service --no-pager ls /boot/twbsp lsblk -o NAME,SIZE,FSTYPE,PARTLABEL,MOUNTPOINTS /dev/nvme0n1 ``` You should see an approximately 32 GiB APP partition (p1) and DATA_PV (p16). `/boot/twbsp` should contain the vendor DTB and `tegra234-p3737-0000-twboard-config.dtbo`. If APP occupies almost the entire drive or DATA_PV is missing, recheck the flashing XML. Do not turn the current root partition directly into an LVM PV. For the first provisioning operation, run from the writable APP system: ```bash sudo lzc-orin provision --dry-run sudo lzc-orin provision --yes ``` The first stage prepares the initramfs and boot entries, then reboots automatically. The second stage boots from read-only APP into a temporary provision target and automatically clones APP to `ssd_vg/ssd_lv`. On success, it reboots again directly into the writable LVM runtime. **The second stage has no SSH access; you do not need to log in again to start the clone manually.** You can monitor progress through a serial console. After completion, verify: ```bash sudo lzc-orin provision status findmnt / findmnt /factory findmnt /boot sudo lvs ``` `/` should be `/dev/mapper/ssd_vg-ssd_lv`. APP is mounted read-only at `/factory`, and its `/boot` is bind-mounted read-only into the running system. If the device still boots from eMMC, select the NVMe boot entry in UEFI. With a network connection, initialize JetPack, Docker, Compose, and the AI Pod services in the **LVM runtime**: ```bash sudo lzc-orin all ``` The flashing image contains the base system and initialization tools. This step installs JetPack runtime software and Docker into the LVM runtime. The default Docker data directory is `/ssd/docker`. After the user is added to the Docker group, log out and log in again. Packages that provide APP's kernel, DTB, and initrd are placed on hold; do not upgrade the read-only `/boot` separately from the runtime. ## 6. Recovery, Storage Expansion, and Troubleshooting If automatic cloning fails, the device will not reboot automatically. Check the errors through a serial console. If you need the regular recovery environment, reboot, manually select **LZC Orin APP recovery** in the L4TLauncher menu, and retry from read-only APP: ```bash sudo lzc-orin maintenance --yes ``` This command clones the LV again. The regular APP recovery environment does not run it automatically. For a factory reset, first enter **LZC Orin APP recovery** as well: ```bash sudo lzc-orin reset --dry-run sudo lzc-orin reset --yes ``` Reset deletes the LVM runtime data and preserves the APP recovery system. All LVM member drives must be connected. After completion, run `sudo lzc-orin all` again. When adding a second, empty NVMe drive, first verify its model, serial number, and device name in the **LVM runtime**, then run: ```bash lsblk -d -o NAME,SIZE,MODEL,SERIAL sudo lzc-orin expand-lvm --disk /dev/nvme1n1 --dry-run sudo lzc-orin expand-lvm --disk /dev/nvme1n1 ``` A nonempty drive requires the explicit flags `--force-wipe --yes`, which delete its contents. After expansion, the root filesystem depends on all member drives; you cannot simply disconnect any member drive. | Symptom | What to check | | --- | --- | | The host cannot detect the Recovery USB device | Check the data cable, Recovery port, power supply, and buttons; run `lsusb -d 0955:` on both the host and in the container | | The XML or flashing script is missing | Confirm that you are using `catdogai/lzc-x3:jetpack-7.2.1` and that the current directory is `/Linux_for_Tegra` | | Not enough space to generate the image | Check the disk containing the host's Docker data directory and the container's writable layer; reserve space for temporary images | | Initial setup fails | Check the `twInit.service` logs, `/boot/twbsp`, and the vendor FDT/OVERLAYS boot configuration | | SSH is unavailable after provision | The temporary provision target has no SSH; use a serial console to check progress and errors | | The device reboots into eMMC | Check the NVMe boot order in UEFI | | The LVM root cannot be found | Confirm that all PV members are connected, and check the initramfs and provision logs | Collect the following information on the device: ```bash cat /proc/cmdline findmnt / sudo lzc-orin provision status sudo journalctl -b -u twInit.service --no-pager sudo journalctl -b -u lzc-orin-provision.service --no-pager ``` --- # GPIO Key 1. Programmable key (function key next to the power button) ``` sudo su gpioget $(gpiofind PM.07) # 安装后会输出 0, 松开后会输出 1 ``` 2. LED Lamp ``` sudo su gpioset -m signal $(gpiofind PP.04)=1 # 点亮红色 gpioset -m signal $(gpiofind PP.04)=0 # 关闭红色 gpioset -m signal $(gpiofind PR.00)=1 # 点亮白色 gpioset -m signal $(gpiofind PR.00)=0 # 关闭白色 ``` --- # Software Installation ## lzc-earlyoom The AI Pod is built for architecture `jetson agx orin`, belonging to `iGPU`. The default version of `earlyoom` cannot detect `GPU` memory usage. Our modified `lzc-earlyoom` resolves this issue. Please refer to the documentation at the following link for installation: https://gitee.com/lazycatcloud/lzc-aipod-resources#lazycat-aipod-early-oom-killer ## lzc-thermal https://gitee.com/lazycatcloud/lzc-aipod-resources#lazycat-aipod-thermal-control-module ## Install Firefox ``` sudo apt install snap snap download snapd --revision=24724 sudo snap ack snapd_24724.assert sudo snap install snapd_24724.snap sudo sudo snap refresh --hold snapd sudo snap install firefox ``` Related questions available at https://elinux.org/Jetson/L4T/r36.4.x_patches#Web_Browser ## Install input method ``` sudo apt install fcitx5 fcitx5-rime fcitx5-pinyin ``` ## todesk ``` curl 'https://dl.todesk.com/linux/todesk-v4.7.2.0-arm64.deb' \ -H 'DNT: 1' \ -H 'Referer: https://www.todesk.com/' \ -H 'Upgrade-Insecure-Requests: 1' \ -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36' \ -H 'sec-ch-ua: "Not;A=Brand";v="99", "Google Chrome";v="139", "Chromium";v="139"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "Linux"' -O sudo dpkg -i todesk-v4.7.2.0-arm64.deb ``` --- # Introduction The AI Pod runs `Ubuntu 22.04.5 LTS`. You can connect a keyboard, mouse, and display and use it as an independent host. The `lzc-ai-agent` service integrates it with the LCMD platform and deploys and starts the related AI application services. Services on the AI Pod run in `docker`. ```/etc/os-release PRETTY_NAME="Ubuntu 22.04.5 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.5 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` ::: warning Warning The AI Pod's `Linux` system cannot be replaced with other distributions. This `Ubuntu` is officially adapted for `nvidia jetson`. If you change `Linux` to another distribution, you may encounter unknown issues. If you restore to the factory environment, you can use a system flash to restore the system. ::: ## SSH The default username and password for the AI Pod is `nvidia:nvidia`. 1. You can check your current IP address in the AI Pod's management list. ![ip address](../../os/images/aipod-ip-address.png) 2. You can connect to the AI Pod system via `ssh nvidia@ip`. 3. If you are not on the same local network as the AI Pod, you can also connect through the LCMD as a jump host, for example `ssh -J root@微服名称.heiyu.space nvidia@ip`. --- # Resource Usage ## View AI Pod Resource Usage in LCMD 1. Open the AIPOD application and click "Quick Jump" ![Launcher](../../os/images/aipod-launcher.png) 2. Click the settings button in the top-right corner. ![Settings](../../os/images/aipod-home.png) 3. Click "Details" in the AI Pod Management list. ![View Details](../../os/images/aipod-vram-detail.png) 4. A new page will open, displaying the memory usage of each service in your AI Pod. (Note: Only AI application services are shown here. Manually deployed services will not be displayed.) ![Memory List](../../os/images/aipod-vram-list.png) ## View AI Pod Resource Usage via jtop 1. Open jtop and connect ![jtop-ai](../../os/images/aipod-jtop.png) The bottom navigation bar can be clicked or accessed via `1, 2, 3, 4, 5, 6, 7` to switch between views. ![jtop-ai-mem](../../os/images/aipod-jtop-mem.png) 2. SSH into the AI Pod and execute the `jtop` command to view ![jtop-ai-cpu](../../os/images/aipod-jtop-cpu.png) ## Why is nvidia-smi not used in the AI Pod? > It is not possible to use nvidia-smi on a Jetson. This program/utility requires a PCI bus. Jetsons have an integrated GPU (iGPU) which is wired directly to the memory controller. Only a discrete GPU (dGPU) on a PCI slot can use that utility. > Cannot use nvidia-smi on Jetson. The tool requires a PCI bus. Jetson has an integrated GPU (iGPU) directly connected to the memory controller. Only a discrete GPU (dGPU) on a PCI slot can use this utility. > An interesting proposal for NVIDIA: Create a surrogate nvidia-smi program that appears identical to the original nvidia-smi when running GPU queries, but instead communicates with the iGPU. Probably not practical, but if it worked, it would greatly enhance the GPU's flexibility and interchangeability with other software. > EDIT (August, 2024): The above is still true, except that Orin supports L4T R36.x, and a partial surrogate nvidia-smi is available for this. This is a limited-functionality nvidia-smi that may suffice in some cases, but the iGPU on Orin with R36.x still has no PCI bus functionality. > Edit (August 2024): The above still holds true, except that the Orin can use L4T R36.x, and a partial substitute nvidia-smi is available for this. This is a limited-functionality nvidia-smi that may be sufficient in some cases, but the iGPU on Orin with R36.x still lacks PCI bus functionality. 1. https://forums.developer.nvidia.com/t/nvidia-smi-not-present-in-jetson-linux/239757/2 --- # Network Configuration The AI Pod supports both Wi-Fi and wired connections. Wired connections support `2.5G` and `10G` network interfaces. ## Configure Wi-Fi via `HDMI + 鼠标键盘` After connecting the display to the AI Pod, click the Wi-Fi icon in the top-right corner of the screen to connect. ## Configure Wi-Fi via LCMD `AI POD` ::: warning Warning AI Pods already in existence will not be detected. If you wish to re-add one, you must first remove it from the AI Pod's management list. ::: 1. Open the configuration interface of `AIPOD`, and click `添加算力舱`. 2. Ensure your phone has Bluetooth enabled and has granted `懒猫微服` client permission to access nearby Bluetooth devices. 3. Search for devices. In the resulting list, devices with an IP address were found via Ethernet cable; those without are found via Bluetooth. ![scan device list](../../os/images/scan-devices-list.jpg) 4. Select the device you wish to bind and click "Start Binding". 5. A Wi-Fi selection list will appear. You can choose your Wi-Fi network and enter the password to connect. ![join wifi](../../os/images/join-wifi.jpg) --- # AI Pod Port Forwarding In the AI Pod system, a default `lzc-ai-agent` `systemd` service is provided, which is responsible for pulling container images, managing container startup, and connecting with the LCMD (Microservice). In certain scenarios, you may need to directly access a service running inside the AI Pod via its IP address. However, since internal services do not expose ports by default, you need to set up port forwarding to expose the service's ports to a local network address within the AI Pod. 0. The AI Pod's IP address can be found in the device management list of `aipod`. ![copy-aipod-ip](../../os/images/copy-aipod-ip.jpg) 1. Open the URL `http://算力舱的ip:40793/ui`. If you're accessing this through a browser on the AI Pod, you can `http://127.0.0.1:40793/ui` open it and add the service you wish to forward, such as `bgem3vllm-ai`. ![client-port-forward.jpg](../../os/images/client-port-forward.jpg) 3. Test Open `http://ip:端口`, for example, ports `bgem3vllm-ai` and `5666`. ![client-port-forward-test.jpg](../../os/images/client-port-forward-test.jpg) Or directly access it within the AI Pod. ``` > curl http://127.0.0.1:5666/v1/models ``` ::: info Info Note: Services will not be automatically started at this time; under processing. --- # Using the LCMD Network to Connect to the Internet If network restrictions prevent the AI Pod from downloading container images, forward all AI Pod traffic to an LCMD device on the same local network and use it as the internet gateway. 1. Run on the LCMD device SSH into the LCMD device and run the following commands (replace 192.168.31.0 with the actual local network segment): ```sh iptables -t nat -I POSTROUTING --source 192.168.31.0/24 -j MASQUERADE iptables -P FORWARD ACCEPT ``` 2. Run on the AI Pod (replace 192.168.31.60 with the LCMD device's IP address) ```sh sudo ip route add default via 192.168.31.60 ``` 3. Test it out ```sh curl -v https://registry-static.lazycat.cloud ``` --- # AI App Skill `lzc-aipod-aiapp` is an Agent Skill for AI Pod AI App development. It organizes the AI App's resource directory, `aipod.yml`, `agxorin`/`thor` services, browser extensions, loading startup logic, dependency declarations, and old package migration rules into a reusable development context. Skill repository address: ```text https://github.com/lazycatcloud/lzc-aipod-aiapp-skill/tree/master/lzc-aipod-aiapp ``` ## Applicable Scenarios Recommend enabling this skill for the following tasks: - Creating a new AI Pod AI App. - Migrating the old `pkg/ai-pod-service/docker-compose.yml` to the resource export layout. - Adding `agxorin/docker-compose.yml` and `thor/docker-compose.yml` to the same AI App. - Adjusting shortcuts, dependencies, and localization configurations in `config/aipod.yml`. - Integrating `/startup/gate` or initiating loading. - Troubleshooting AI App service host, dependency resolution, browser extensions, and resource directory issues. ## Codex Installation The skill for Codex is installed by default at: ```text ~/.codex/skills//SKILL.md ``` The simplest way is to directly request installation of a skill from a GitHub repository within Codex: ```text 请安装这个 Codex skill: https://github.com/lazycatcloud/lzc-aipod-aiapp-skill/tree/master/lzc-aipod-aiapp ``` If manual installation is required, execute: ```bash tmpdir="$(mktemp -d)" git clone --depth 1 https://github.com/lazycatcloud/lzc-aipod-aiapp-skill.git "$tmpdir" mkdir -p ~/.codex/skills rm -rf ~/.codex/skills/lzc-aipod-aiapp cp -R "$tmpdir/lzc-aipod-aiapp" ~/.codex/skills/ rm -rf "$tmpdir" ``` After installation, a newly opened Codex chat will automatically detect this skill. When the query involves scenarios such as AI Pod AI App creation, migration, or debugging, Codex will automatically load it. It can also be explicitly specified in the prompt using `使用 lzc-aipod-aiapp skill`. ### Codex Example ```text 使用 lzc-aipod-aiapp skill，帮我把当前仓库里的旧 AI App 迁移到 resource export 布局。 要求: 1. 将 pkg/manifest.yml 里的 aipod 配置拆到 config/aipod.yml 2. 将 pkg/ai-pod-service/docker-compose.yml 拆成 agxorin/docker-compose.yml 3. 如果已有 thor 镜像，补充 thor/docker-compose.yml 4. 检查 Traefik Host、depends_on_hosts 和启动 loading 5. 最后运行仓库现有验证命令 ``` ## Claude Code Installation Claude Code's personal skill is installed by default to: ```text ~/.claude/skills//SKILL.md ``` Installation command: ```bash tmpdir="$(mktemp -d)" git clone --depth 1 https://github.com/lazycatcloud/lzc-aipod-aiapp-skill.git "$tmpdir" mkdir -p ~/.claude/skills rm -rf ~/.claude/skills/lzc-aipod-aiapp cp -R "$tmpdir/lzc-aipod-aiapp" ~/.claude/skills/ rm -rf "$tmpdir" ``` If you only want the current project to use this skill, you can place the directory inside the project: ```bash mkdir -p .claude/skills cp -R ~/.claude/skills/lzc-aipod-aiapp .claude/skills/ ``` Claude Code can directly input `/lzc-aipod-aiapp` to invoke it, or describe tasks related to the AI Pod AI App. Claude will automatically load the relevant skills based on the skill description. ```text /lzc-aipod-aiapp 请检查这个 AI App 的 resource 导出是否正确: 1. aipod-resources/config/aipod.yml 是否只包含 AI Pod 专属配置 2. agxorin 和 thor compose 是否暴露相同的 Host 3. lzc-build.yml 是否正确声明 resource_exports 4. lzc-manifest.yml 是否可以直接使用 application.upstreams 转到服务 UI ``` ## Update When updates are available in the skill repository, simply re-run the corresponding installation command. The installation command will first remove the local old directory and then copy the latest `lzc-aipod-aiapp` directory. After updating, it is recommended to start a new Codex or Claude Code session to ensure the tools reload the latest `SKILL.md` and references. ## References - [lzc-aipod-aiapp skill repository](https://github.com/lazycatcloud/lzc-aipod-aiapp-skill/tree/master/lzc-aipod-aiapp) - [Claude Code Skills Documentation](https://code.claude.com/docs/en/skills) --- # AI App Installation and Service Deployment Progress API This document describes manual AI App installation, device-side service deployment, streaming progress, and the API interface to query the latest deployment status using `serviceId`. For integration and testing workflows, refer to the deployment progress testing documentation in the project repository. ## 1. Basic Conventions - AI Pod external domain: `https://.` - `subdomain` is derived from the app's `lzc-manifest.yml`; currently defaults to `ai` - `box-domain` is prioritized by fetching from `lzc-cli box default`; if unavailable, defaults to `13gxg.heiyu.space` Platform-side API base path: - `https://./backend/aiapps` - `https://./backend/services` Device-side API base path: - `http://:/v1/service` ## 2. Manual AI App Installation ### 2.1 `POST /backend/aiapps/install/stream` Purpose: - Upload an AI App ZIP file - Complete registration - If the package includes deployable services, continue triggering device deployment. The new resource export service paths are `agxorin/docker-compose.yml`, `thor/docker-compose.yml`. Fallback to legacy `ai-pod-service/docker-compose.yml` if missing. - Return installation progress via SSE. Request: - Method: `POST` - Content-Type: `multipart/form-data` - Required fields: `file` Response: - Content-Type: `text/event-stream` - Event name: `progress` Event structure: ```json { "stage": "pulling", "status": "downloading", "progress": 37, "message": "registry.lazycat.cloud/demo/app:latest: Downloading", "pkgId": "demo-app", "serviceId": "demo-app", "currentImage": "registry.lazycat.cloud/demo/app:latest", "currentLayer": "sha256:abcd", "current": 5242880, "total": 10485760, "error": "" } ``` Note: - During the local package processing phase, status messages such as `upload_saved`, `unzipping`, `validating`, and `registering` are still sent. - The `progress` token in these phases is fixed to `0` and is not counted in the installation progress bar. - Actual progress in installation occurs during the service deployment phase: - `pulling`: `0-80` - `starting`: `85` - `waiting`: `95` - `completed`: `100` - `failed`: retain progress prior to failure - If the AI App has no deployable services, it directly returns: - `stage=completed` - `progress=100` - `message=installed without deployable service` Typical return: ```text event:progress data:{"stage":"registering","progress":0,"message":"registering ai app","pkgId":"demo-app"} event:progress data:{"stage":"pulling","status":"downloading","progress":31,"message":"registry.lazycat.cloud/demo/app:latest: Downloading","pkgId":"demo-app","serviceId":"demo-app","currentImage":"registry.lazycat.cloud/demo/app:latest","currentLayer":"sha256:abcd","current":5242880,"total":10485760} event:progress data:{"stage":"starting","status":"starting","progress":85,"message":"starting service","pkgId":"demo-app","serviceId":"demo-app"} event:progress data:{"stage":"waiting","status":"waiting","progress":95,"message":"waiting for service health","pkgId":"demo-app","serviceId":"demo-app"} event:progress data:{"stage":"completed","status":"complete","progress":100,"message":"service deployed","pkgId":"demo-app","serviceId":"demo-app"} ``` Failure example: ```text event:progress data:{"stage":"failed","status":"error","progress":42,"message":"pull images failed","pkgId":"demo-app","serviceId":"demo-app","error":"pull access denied"} ``` ### 2.2 `DELETE /backend/aiapps/apps/:pkgId` Purpose: - Delete a manually installed AI App - Remove the corresponding service directory, cache directory, and container image Return: ```json {"removed":true,"pkgId":"demo-app"} ``` ### 2.3 `GET /backend/aiapps/apps` Purpose: - List the current AI App entries ### 2.4 `GET /backend/aiapps/installed/:pkgId` Purpose: - Check if a specified `pkgId` has been registered to the AI App store Return: ```json {"installed":true} ``` ## 3. Platform-side Service Interfaces ### 3.1 `GET /backend/services/progress?serialNumber=&serviceId=` Purpose: - Query the most recent deployment progress of a specified service on a device - Suitable for automatic deployment scenarios, or for manual installation with interruption and subsequent recovery Return structure: ```json { "serviceId": "demo-app", "stage": "pulling", "status": "extracting", "progress": 56, "message": "registry.lazycat.cloud/demo/app:latest: Extracting", "currentImage": "registry.lazycat.cloud/demo/app:latest", "currentLayer": "sha256:abcd", "current": 7340032, "total": 10485760, "startedAt": 1742880000, "updatedAt": 1742880012, "finishedAt": 0, "error": "" } ``` Idle state return: ```json { "serviceId": "demo-app", "stage": "idle", "progress": 0, "message": "deployment not started" } ``` ### 3.2 Existing Platform-side Interfaces - `GET /backend/services/list?serialNumber=` - `GET /backend/services/start?serialNumber=&serviceId=` - `GET /backend/services/stop?serialNumber=&serviceId=` - `GET /backend/services/delete?serialNumber=&serviceId=` - `GET /backend/services/log?serialNumber=&serviceId=` Note: - `start` is a synchronous trigger and does not return streaming progress. - Automatic deployment and `start` both continuously update the latest deployment status of `serviceId`. - For real-time progress, prefer using the AI App installation stream or the device-side `/service/start/stream`. ## 4. Device-side Service Interfaces ### 4.1 `POST /v1/service/start` Purpose: - Triggers a one-time synchronous deployment. - The system internally updates the latest deployment status. Request Fields: - `file`: service zip - `serviceId` - `dirOverWrite`: `true|false` - `forceRecreate`: `true|false` - `extraEnv[KEY]=VALUE`: optional ### 4.2 `POST /v1/service/start/stream` Purpose: - Trigger a streaming deployment - Shares the same underlying deployment logic as `/v1/service/start` - Adds SSE-based progress feedback Response: - Content-Type: `text/event-stream` - Event name: `progress` - Data structure matches `backend.ServiceDeployStatus` ### 4.3 `GET /v1/service/progress?serviceId=` Purpose: - Query the latest deployment status of the service ### 4.4 `GET /v1/service/log?serviceId=` Purpose: - Continuously monitor service logs ### 4.5 `GET /v1/service/installed` Purpose: - View the currently installed services on the device along with `docker-compose.yml` summary information ## 5. Semantic Meaning of Deployment Progress Unified status fields: - `stage`: High-level stage - `status`: Fine-grained status - `progress`: 0–100 - `message`: Human-readable message Current stage definitions: | stage | progress | Meaning | | --- | --- | --- | | `idle` | `0` | No deployment records available | | `prepare` | `0` | Start deployment: prepare directory, unzip, write `.env` | | `pulling` | `0-80` | Pull the compose container image | | `starting` | `85` | Execute `docker-compose up -d --wait` | | `waiting` | `95` | Container has been started, waiting for health check or `--wait` to complete | | `completed` | `100` | Deployment successful | | `failed` | Keep current value | Deployment failed. Error details see `error`. | `pulling.status` Currently possible: - `prepare` - `preparing` - `waiting` - `downloading` - `download_complete` - `extracting` - `verifying` - `already_exists` - `complete` Explanation: - The percentage for `pulling` comes from the structured byte progress of Docker Engine `ImagePull`, no longer relying on `docker-compose pull` PTY text. - In multi-image scenarios, the overall `0-80` progress is calculated by averaging the weights of individual images. - `starting` and `waiting` are currently phase-based progress indicators and are not precise container-level percentages. - The system retains only the most recent deployment status for each `serviceId`. ## 6. Call Examples ### 6.1 Manually installing an AI App and monitoring streaming progress ```bash curl --no-buffer \ -F "file=@./demo-app.zip" \ "https://ai.13gxg.heiyu.space/backend/aiapps/install/stream" ``` ### 6.2 Query the recent deployment status of a specific service ```bash curl \ "https://ai.13gxg.heiyu.space/backend/services/progress?serialNumber=&serviceId=demo-app" ``` ### 6.3 Directly Call Device-Side Stream Deployment Interface ```bash curl --no-buffer \ -F "file=@./demo-app-service.zip" \ -F "serviceId=demo-app" \ -F "dirOverWrite=true" \ "http://:/v1/service/start/stream" ``` ### 6.4 Directly Call the Device-side Deployment Status Query Interface ```bash curl "http://:/v1/service/progress?serviceId=demo-app" ``` ## 7. ZIP Structure Requirements AI App ZIP supports both resource export structure and legacy manual installation structure. ### 7.1 Resource Export Structure Recommended structure: ```text demo-app/ └── resources/ └── aipod/ └── demo-app/ ├── package.yml ├── lzc-manifest.yml ├── icon.png ├── config/ │ └── aipod.yml ├── agxorin/ │ └── docker-compose.yml ├── thor/ │ └── docker-compose.yml └── extensions/ └── extension.zip ``` After platform installation, the AIApp resource is expanded to: ```text /lzcapp/run/resources/aipod// ``` Service selection rules: - Thor devices prioritize `thor/docker-compose.yml` - All other devices default to `agxorin/docker-compose.yml` - When model-specific compose is missing, fallback to legacy `ai-pod-service/docker-compose.yml` Plugin selection rules: - Prefer `extensions/*.zip|*.crx` - Fall back to legacy `browser-extensions/extension.zip` if missing ### 7.2 Legacy Manual Installation Structure Legacy manual installation ZIP remains compatible: ```text demo-app/ └── pkg/ ├── manifest.yml ├── icon.png ├── extension.zip └── ai-pod-service/ └── docker-compose.yml ``` Device-side service zip structure: ```text demo-app/ ├── docker-compose.yml └── ... ``` AI App Rules: - The ZIP file containing resource export must contain exactly one AIApp resource root directory and be parsable to extract only one such root. - Legacy manual installation ZIP files must contain exactly one `pkg/manifest.yml`. - The legacy manual installation service path is `pkg/ai-pod-service/docker-compose.yml`. - The new resource export service paths are `agxorin/docker-compose.yml`, `thor/docker-compose.yml`, or fallback to `ai-pod-service/docker-compose.yml`. - The compose file must include a resolvable Traefik `Host(...)` label for platform identification of AI host and startup loading status. ## 8. Compatibility Notes - Automatic deployment, manual installation with streaming deployment, and platform-side `start` interface all share the same underlying deployment logic and the same state repository. - During manual installation, the AI App SSE directly passes through the device-side deployment progress. - During automatic deployment, no SSE is returned; the caller should query the latest deployment status via `serviceId`. - The `pkg/ai-pod-service/docker-compose.yml` in legacy ZIP files is only used as a compatibility path for manual installation and is not the sole service format for the new resource export. --- # Release AI Applications 1. [AI Pod Categories](https://appstore.lazycat.cloud/#/shop/category/27) 2. [LCMD Application Release](https://developer.lazycat.cloud/publish-app.html) Releasing AI applications is similar to releasing LCMD applications—both can be done via `lzc-cli` or through the LCMD developer platform. When releasing an application, include `AI Pod`, `算力舱`, or relevant details to help the review team categorize the application under `算力舱`. --- # AI Application First, let's understand the following concepts: 1. `lzc-cli` is the scaffolding used to develop LazyCat LCMD applications. With `lzc-cli`, you can quickly package, install, debug, and release applications. 2. The package generated by `lzc-cli` uses the `lpk` format, the application format of the LCMD ecosystem. 3. The LazyCat AI Browser supports plugin installation for both `PC` and `Android` platforms. `AI 应用` refers to applications built on top of LCMD applications by importing an `AIApp resource`. These applications follow the same download and installation process as regular LCMD applications, with the following key differences: - **Supports AI service integration**: After installation, the app automatically deploys the matching device model's AI service onto the AI Pod. - **Supports AI Browser plugin integration**: After installation, the AI Browser automatically loads the corresponding plugin. - **Supports defining AI Browser shortcuts and dependencies**: Configuration for shortcuts, upstream services, and storage aliases is defined in the resource via `config/aipod.yml`. ## Resource Import Declaration LCMD applications import AIApp resources through `lzc-manifest.yml`, using its `import_resources` declaration: ```yml import_resources: - kind: aipod ``` After the platform installs an application, the AIApp resource is expanded to the following runtime directory: ```text /lzcapp/run/resources/aipod// ``` Among them, `` is the application package ID. The platform reads the AI Pod configuration, service compose file, browser plugin, and fallback metadata from this directory. ## Resource Directory Structure The recommended AIApp resource directory structure is as follows: ```text aipod/ ├── package.yml ├── lzc-manifest.yml ├── icon.png ├── config/ │ └── aipod.yml ├── agxorin/ │ └── docker-compose.yml ├── thor/ │ └── docker-compose.yml └── extensions/ ├── desktop.zip └── android.crx ``` Field Description: - `config/aipod.yml`: AI Pod configuration file, without outer `aipod:`. - `agxorin/docker-compose.yml`: Service definition for AGX Orin and default Jetson devices. - `thor/docker-compose.yml`: Service definition for Thor devices. - `extensions/*.zip|*.crx`: AI browser plugin file, supporting one or more. - `package.yml`, `lzc-manifest.yml`, `icon.png`: fallback sources for metadata and icons. When compatible with old packages, the platform will still recognize the following legacy paths: - `ai-pod-service/docker-compose.yml`: legacy service definition. Used only as fallback when the compose file is missing under the device model directory. - `browser-extensions/extension.zip`: legacy browser plugin. Used only as fallback when the plugin under `extensions/` is missing. ## `config/aipod.yml` `config/aipod.yml` contains only the AI Pod configuration itself, without an outer `aipod:`. For example: ```yml shortcut: disable: false title: ComfyUI url: /_lzc/aipod_backend/startup/gate/cloud.lazycat.aipod.comfyui?redirect=https%3A%2F%2Fcomfyui-ai favicon: /icon.png locales: en: shortcut: title: ComfyUI depends_on_others: - ollama-ai depends_on_hosts: - comfyui-ai storage_alias: models: /ssd/models ``` ### `shortcut` AI applications installed in the LCMD are displayed by default in the AI Browser shortcut. This can be disabled via `shortcut.disable`, or the shortcut's title, entry URL, and icon can be overridden: - `shortcut.disable`: Whether to hide the shortcut, default is `false`. - `shortcut.title`: Shortcut title. - `shortcut.url`: Shortcut entry URL. It is recommended to use the platform startup gate for long initialization services. - `shortcut.favicon`: Shortcut icon URL. ### `locales` `locales` is used to configure multi-language metadata. The platform selects appropriate titles, descriptions, or shortcut fields based on the browser's language setting, falling back to default configurations if missing. ### `depends_on_others` `depends_on_others` declares the other AI hosts that the current AI application depends on. The platform considers these dependencies during entry and startup checks. ### `depends_on_hosts` `depends_on_hosts` declares the AI hosts that the current AI application itself needs to check. Typically, this is filled with the `Host(...)` value exposed via Traefik in the compose file. `storage_alias` `storage_alias` Declare storage aliases used in the service to enable the platform to mount models, caches, or user data to stable paths. ## Add AI Services AI services are described via the `docker-compose.yml` in the resource. The platform selects the service directory based on device model: 1. Thor devices prefer `thor/docker-compose.yml`. 2. Other devices default to `agxorin/docker-compose.yml`. 3. If the model-specific directory is missing, fallback to the legacy `ai-pod-service/docker-compose.yml`. If `agxorin`, `thor`, and legacy services coexist, the AI host exposed to the AI browser must remain consistent across devices to prevent the same application from generating different access addresses on different devices. ### Service Access Rules Services within the AI Pod use `traefik` combined with `Host` forwarding rules. Compose files must include resolvable Traefik `Host(...)` labels and have host values ending with `-ai`. ```yml services: comfyui: image: registry.lazycat.cloud/demo/comfyui:latest labels: - "traefik.enable=true" - "traefik.http.routers.${LZC_SERVICE_ID}-comfyui.rule=Host(`comfyui-ai`)" networks: default: external: true name: traefik-shared-network ``` Rule Description: - The value in `Host(...)` is the AI host that the microservice and AI Browser use to access this service. - Only services joined in `traefik-shared-network` can be managed by Traefik. - The AI host must be exposed consistently across multiple device directories' compose files. - Long initialization services must configure `healthcheck`, using container health status to control the ready state. The platform's installation progress and startup loading will reuse this status. For more usage of `traefik`, see [Host Rule](https://doc.traefik.io/traefik/routing/routers/#host-and-hostregexp). ::: info Info The `docker` in the AI Pod defaults to `nvidia-runtime`. The container can directly use `gpus` without explicitly configuring `gpus`. For more details, [click here](https://forums.developer.nvidia.com/t/whats-difference-between-gpus-and-runtime-nvidia-for-the-docker-container/283468/4). ::: ### Service Environment Variables The following environment variables are provided by the AI Pod when the service runs: 1. Data Persistence Path - Definition: `/ssd/lzc-ai-agent/data/` - Environment Variable: `LZC_AGENT_DATA_DIR` - Example: `/ssd/lzc-ai-agent/data/cloud.lazycat.aipod.ai` ```yml services: ollama: volumes: - ${LZC_AGENT_DATA_DIR}/data:/root/.ollama ``` 2. Data Caching Path - Definition: `/ssd/lzc-ai-agent/cache/` - Environment Variable: `LZC_AGENT_CACHE_DIR` - Example: `/ssd/lzc-ai-agent/cache/cloud.lazycat.aipod.ai` ```yml services: ollama: volumes: - ${LZC_AGENT_CACHE_DIR}/cache:/root/.cache ``` 3. Service ID - Definition: The value of `` minus `.` - Environment variable: `LZC_SERVICE_ID` - Example: `cloudlazycataipodfishspeech` ```yml services: ollama: labels: - "traefik.http.routers.${LZC_SERVICE_ID}-ollama.rule=Host(`ollama-ai`)" ``` ## Add AI Browser Plugin The AI browser plugin is placed in the resource directory under `extensions/`, supporting `zip` and `crx`: ```text extensions/ ├── desktop.zip └── android.crx ``` `zip` For browser-compatible plugin formats. If the plugin is developed using [wxt](https://wxt.dev/), it can be directly packaged as a zip file; alternatively, the plugin directory can be manually zipped. When supporting legacy packages, the platform will still read `browser-extensions/extension.zip` as a fallback. ## Metadata and Icon Fallback The platform prioritizes reading `config/aipod.yml` from the resource. If missing metadata such as title, description, or icon is found, the following fallback order is applied: 1. `package.yml` 2. `lzc-manifest.yml` 3. `icon.png` If metadata is present in both the LCMD application manifest and the AIApp resource, the AI Browser displays the resource configuration. ## Startup Entry and Loading It is no longer recommended to include a built-in `caddy` within each AI application for health checks during long initialization of AI services. Instead, it is recommended to reuse the startup gate provided by the platform. ```text /_lzc/aipod_backend/startup/gate/:appid?redirect= ``` ### Quick access to directly enter gate If the application entry point is an AI host, you can directly access the gate via the shortcut: ```yml shortcut: title: ComfyUI url: /_lzc/aipod_backend/startup/gate/cloud.lazycat.aipod.comfyui?redirect=https%3A%2F%2Fcomfyui-ai ``` Rule Description: - `:appid` is the AI application package ID. - `redirect` is the target URL to redirect to after the service becomes ready, which must be URL encoded and is typically the AI host or microservice page address exposed by compose. - Static frontend can directly use the LCMD platform routing. - API or model service access routes through Traefik upstream. - Reuse platform capabilities for loading, health checks, and failure alerts; no need to implement these repeatedly for each application. ### Static Frontend Integration with Startup Gate If the AI application itself has an LCMD static frontend, follow the approach in `cloud.lazycat.aipod.imgtrans`: use `lzc-manifest.yml` and `application.injects` to intercept page requests. Users then enter the platform startup loading phase before any frontend page; once the service is ready, they return to the original page with deep links and query parameters preserved. ```yml application: subdomain: imgtrans injects: - id: aipod-startup-gate 'on': request auth_required: false when: - /* unless: - /api/* - /debug/backend/* - /assets/* - /favicon.ico - /icon.png - /robots.txt - /manifest.json - /_lzc/* do: - src: > const appid = "cloud.lazycat.aipod.imgtrans"; const cookieName = "aipod_startup_ready_" + appid.replace(/[^A-Za-z0-9]/g, "_"); const cookies = String(ctx.headers.get("Cookie") || ""); if (cookies.split(";").map((item) => item.trim()).some((item) => item === cookieName + "=1")) { return; } const boxDomain = String(ctx.request.host || "").replace(/^[^.]+\./, ""); const query = ctx.request.raw_query ? "?" + ctx.request.raw_query : ""; const redirect = "https://" + ctx.request.host + ctx.request.path + query; const location = "https://" + boxDomain + "/_lzc/aipod_backend/startup/gate/" + encodeURIComponent(appid) + "?redirect=" + encodeURIComponent(redirect); ctx.response.send(307, "", { location }); routes: - /=file:///lzcapp/pkg/content/dist upstreams: - location: /api/ backend: http://backend:3000 disable_trim_location: true public_path: - /debug/backend/ - /api/ ``` Key points of this example: - `when: /*` intercepts all page entry points, `unless` allows access to APIs, debugging interfaces, static resources, icons, and `/_lzc/*` internal platform paths, to prevent resource requests and the gate itself from being repeatedly redirected. - `appid` must be filled with the current AI application package ID; the startup gate will use it to query the installation and service ready status. - After the gate becomes ready, it writes a value to the `aipod_startup_ready_` cookie. The inject module first checks this cookie; if ready, it directly allows the request, avoiding the loading state on every page open. - `boxDomain` derives the host by removing the first subdomain segment from the current host, e.g., from `imgtrans.13gxg.heiyu.space` to obtain `13gxg.heiyu.space`. - `redirect` preserves the user's original access address and query parameters, and passes them to the gate via `encodeURIComponent`. After the service becomes ready, the platform redirects back to this original address. - Static frontend files continue to be exposed via `routes`, while application-specific APIs are forwarded via `upstreams`. The startup gate only handles ready/loading states and does not replace business routing. In the AIApp resource, dependencies can be declared, and the platform will include these dependencies in the ready check: ```yml shortcut: disable: false depends_on_others: - trans - ocr ``` ## Access a Service on a Specific AI Pod When there are multiple AI Pods, access can be achieved via `f-{算力舱的序列号}-{服务名称}-ai.{微服名称}.heiyu.space`. For example: ```text https://f-1420225016421-dozzle-ai.catdog.heiyu.space ``` --- # Quantum Computing `jetson agx orin` theoretically supports `CUDA-Q`, but `nvidia` currently promotes `dgx`. The probability of adaptation on `jetson` is estimated to be low. ## Test According to the example in [Quick Start](https://nvidia.github.io/cuda-quantum/latest/using/quick_start.html#quick-start), compilation succeeds, but an error occurs during runtime. https://forums.developer.nvidia.com/t/cuda-q-sample-code-initialization-error/340400 ## Links 1. [NVIDIA CUDA-Q](https://nvidia.github.io/cuda-quantum/latest/using/quick_start.html) 2. [quantum-computing](https://www.nvidia.com/en-us/glossary/quantum-computing/) 3. [Low-Cost, Low-Power, High-Capacity Quantum Simulation Device](https://www.ezchips.org/low-cost-low-power-high-capacity-quantum-simulation-device/) 4. [dependencies-and-compatibility](https://nvidia.github.io/cuda-quantum/latest/using/install/local_installation.html#dependencies-and-compatibility) 5. [NVIDIA CUDA Quantum](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/quantum/containers/cuda-quantum) --- # Jetson Thor Validation Report - Test date: `2026-03-17` - Thor reference doc: `/home/compiler/lzc-aipod-docs/thor/benchmark.md` - Controller: `192.168.1.104` - Machine inventory and credentials: `/home/compiler/lzc_thor/inventory/hosts.private.yaml` - pip cache endpoint: `http://192.168.1.104:5000/index/` - Docker proxy template: `http://192.168.1.128:1085` ## Test Objects `T4000` is `192.168.1.184`, reports as `NVIDIA Jetson Thor Developer Kit`, runs `Ubuntu 24.04.3 LTS`, `L4T R38.4.0`, kernel `6.8.12-tegra`, `12` CPU cores, and about `59.9 GiB` system memory. `T5000` is `192.168.1.213`, reports as `NVIDIA Jetson AGX Thor Developer Kit`, runs `Ubuntu 24.04.3 LTS`, `L4T R38.4.0`, kernel `6.8.12-tegra`, `14` CPU cores, and about `122.8 GiB` system memory. ## Environment and Setup The controller-side proxpi-compatible pip cache was deployed and used through `http://192.168.1.104:5000/index/`. Remote Docker daemon proxy configuration was aligned to the user-specified upstream proxy `http://192.168.1.128:1085`, with `*.lazycat.cloud` and `*.heiyu.space` excluded through `no_proxy`. Both Thor machines passed network, container, CUDA, and multimedia baseline checks. The host-side framework check still reports `torch=False` and `vllm=False`, which is expected for this run because the framework validation was performed inside containers rather than by installing those packages directly on the host. Containerized PyTorch validation passed on both machines, and `vLLM 0.16.0` import validation also passed on both machines. The main framework exception is `T5000` `vllm_container_serve`: the container import succeeded, but the HTTP endpoint on `127.0.0.1:18000` never became reachable during the validation window. This should be treated as a deployment issue rather than a CUDA or TensorRT issue. ## Model Inference Results The image and embedding workloads completed successfully on both machines. On `T4000`, the text embedding workload `clip_text_best_b8` sustained about `197.292 qps`, stayed below `61C`, and peaked at `33.15W`. The vision embedding workload `clip_vision_fp16_b8` sustained about `45.0585 qps`, reached `78C`, and peaked at `58.38W`. `dinov2_fp16_b8` was the slowest sustained image embedding run on `T4000` at `7.124 qps`, with `78C` and `58.36W` peak telemetry. `resnet50_best_b8` reached about `468.935 qps` and `yolov8s_best_b1` reached about `912.803 qps`, both with notably lower thermals than the embedding models. On `T5000`, isolated image and embedding workloads were faster but also hotter. `clip_text_best_b8` sustained about `253.238 qps`, peaked at `79C`, and hit `52.53W`. `clip_vision_fp16_b8` reached about `61.1175 qps`, but it also pushed the board to `105C` and `95.42W`, making it one of the hottest non-synthetic workloads in this test set. `dinov2_fp16_b8` sustained about `4.308 qps`, peaked at `80C`, and hit `62.51W`. `resnet50_best_b8` reached about `616.558 qps` and `yolov8s_best_b1` about `531.15 qps`. The aggregate pattern is straightforward: `T5000` offers higher headroom for large-model and vision workloads, but it also reaches thermal limits much faster than `T4000`. `T4000` is slower, yet materially easier to cool under sustained load. ## llama.cpp Results `llama.cpp` was built in Docker on `T5000` with image `thor-llamacpp:cuda130`. The smaller baseline model `Qwen2.5-1.5B-Instruct-Q4_K_M` completed the full `2,4,6,8,16,32` concurrent batched benchmark without tuning drama. Under the baseline setting of `1024` prompt tokens and `256` generated tokens, total throughput climbed to `3248.14 tok/s` at `32` parallel requests while staying relatively cool at `51C`, `35.82W`, and fan PWM `94`. The supplementary model requested by the user, `Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-GGUF`, was added and tested on `T5000`. Because both Thor machines only had `14G` to `17G` free before cleanup, the larger `Q4` quantizations were not safe to stage. The selected file was `Qwen3.5-27B.Q3_K_S.gguf`, which is about `11.23 GiB`. `T5000` needed image cleanup before the model could be copied safely. The first 27B attempt used `ctx=32768` and `batch=256`; it failed during `llama_decode()` with `failed to find a memory slot for batch of size 256`. The stable configuration was `2048` prompt tokens, `512` generated tokens, `ctx=131072`, `batch=128`, and `ubatch=64`. Under that configuration, the six requested concurrency levels all completed. Total throughput was `38.30 tok/s` at parallel `2`, `58.81 tok/s` at `4`, `56.30 tok/s` at `6`, `40.39 tok/s` at `8`, `99.62 tok/s` at `16`, and `125.75 tok/s` at `32`. The peak telemetry for this 27B run was `80C`, `62.19W`, CPU `69.94C`, and fan PWM `255`. The 27B result is important for planning. It proves that `T5000` can run a meaningful long-context, long-generation `27B` GGUF workload entirely through `llama.cpp`, but only after model quantization, root-disk headroom, and KV-cache sizing are handled deliberately. It is not a drop-in replacement for the small-model path. ## TTS Result The planned TTS target was `registry.lazycat.cloud/x/lzc-aipod-tts:763ebf7` using the `Kokoro` service path. This scenario did not pass on either machine. On `T5000`, the image pulled successfully, but the container failed during model warmup with `CUDA error: no kernel image is available for execution on the device`. This means the current TTS image is not built with kernels that can execute on the tested Thor GPU architecture. The failure is not a transient network issue. On `T4000`, the same image could not be unpacked because overlayfs ran out of space during layer extraction. Registry connectivity itself was normal, so the blocker there is disk headroom, not registry access. The practical conclusion is that the current TTS image cannot be marked Thor-ready. It needs a Thor-compatible rebuild, and `T4000` also needs either more free root space or a smaller image before the scenario can be rerun. ## Full Load Result The combined CPU plus GPU stress scenario was executed for `600` seconds on both machines. The workload used full-core Python floating-point loops on the CPU side and simultaneous `dinov2_fp16_b8` plus `clip_vision_fp16_b8` TensorRT runs on the GPU side. `T4000` completed the full-load run with average GPU utilization around `89.923%`. It peaked at `71C` GPU temperature, `61.406C` CPU temperature, `54.88W` GPU power, fan PWM `255`, `54.879W` `VDD_GPU`, and `11.105W` `VIN_SYS_5V0`. This is a heavy but still controlled thermal profile. `T5000` also completed the full-load run, but it reached a far more aggressive thermal envelope. Average GPU utilization was about `91.009%`, while peak GPU temperature reached `110C`, CPU temperature reached `97.406C`, and peak GPU power reached `95.33W`. Fan PWM hit the ceiling at `255`. This run is the clearest indication that `T5000` has the performance headroom for aggressive workloads, but it also needs much stricter cooling and thermal policy if similar mixed CPU plus GPU loads are expected in production. ## Overall Conclusion The validation confirms that both Thor machines are usable for containerized CUDA, TensorRT, PyTorch, image embedding, text embedding, and `llama.cpp` workloads. `T4000` is easier to keep within a reasonable thermal envelope and is a safer default for medium workloads. `T5000` is the right target for larger LLM inference, but it regularly drives the fan to maximum and can exceed `100C` under the heaviest vision or mixed-load scenarios. The main blockers that remain are operational rather than foundational. `vLLM` serving still needs a separate startup fix on `T5000`. The current `Kokoro` TTS image is not compatible with Thor GPUs, and the `T4000` root filesystem is too tight for large image extraction. For the supplementary 27B `llama.cpp` path, the validated combination is `Q3_K_S` quantization on `T5000` with explicit disk cleanup and enlarged KV-cache settings. --- # Jetson Thor Validation Report Test date: `2026-03-18`, controller: `192.168.1.104`. pip cache entry uses `http://192.168.1.104:5000/index/`, Docker proxy template: `http://192.168.1.128:1085`. This validation covers basic connectivity, Docker and pip proxying, CUDA and TensorRT, image and text embedding, llama.cpp concurrent stress testing, Kokoro TTS container, and CPU+GPU full-load thermal stability scenarios. Detailed raw data remains in CSV/JSON files. The following are only narrative conclusions. ## T4000 T4000 is located at `tegra-ubuntu-t4000.lan`, identified model is `NVIDIA Jetson Thor Developer Kit`, system version is `Ubuntu 24.04.3 LTS`, L4T/JetPack information is `# R38 (release), REVISION: 4.0, GCID: 43443517, BOARD: generic, EABI: aarch64, DATE: Wed Dec 31 00:15:19 UTC 2025`, kernel version `6.8.12-tegra`, CPU core count `12`, total memory `59.9GiB`. Basic capabilities include: AI Pod `PASS`, Docker pull `PASS`, CUDA `PASS`, PyTorch container `PASS`, TensorRT `PASS`, multimedia `PASS`, vLLM/LLM inference `PASS`. Model inference performance: - clip_text_best_b8 throughput 195.747 QPS, average latency 5.107 ms - clip_vision_fp16_b8 throughput 45.477 QPS, average latency 21.988 ms - dinov2_fp16_b8 throughput 7.033 QPS, average latency 142.176 ms - resnet50_best_b8 throughput 470.882 QPS, average latency 2.122 ms - yolov8s_best_b1 throughput 907.689 QPS, average latency 1.1 ms qwen25_1p5b_q4km covers parallelism of 2/4/6/8/16/32, with best total throughput 2560.92 tokens/s (parallel=32), peak temperature of 37°C, and peak power consumption of 22.04 W. qwen35_27b_claude46_q3ks covers parallelism of 2/4/6/8/16/32, with best total throughput 83.53 tokens/s (parallel=32), peak temperature of 45°C, and peak power consumption of 22.06 W. kokoro_tts completes _ requests, average latency _ s, peak temperature _ °C, peak power consumption _ W. A 600s CPU+GPU full-load test has been conducted. The GPU peak temperature is 75°C, the CPU peak temperature is 67.718 °C, tj peak temperature is 76.031 °C, GPU peak power consumption is 57.21 W, VIN_SYS_5V0 peak power is 12.014 W, and fan PWM peak value is _. The current aggregated system peak temperature is `75` °C, peak power consumption is `57.26` W, and fan PWM peak value is `124`. ## T5000 The T5000 is located at `tegra-ubuntu-t5000.lan`, with identified device model `NVIDIA Jetson AGX Thor Developer Kit`, system version `Ubuntu 24.04.3 LTS`, L4T/JetPack information `# R38 (release), REVISION: 4.0, GCID: 43443517, BOARD: generic, EABI: aarch64, DATE: Wed Dec 31 00:15:19 UTC 2025`, kernel `6.8.12-tegra`, CPU core count `14`, total memory `122.8GiB`. Basic capabilities include: pip proxy `PASS`, Docker pull `PASS`, CUDA `PASS`, PyTorch container `PASS`, TensorRT `PASS`, multimedia `PASS`, vLLM/LLM inference `PASS`. Note: Host Torch not installed; host vLLM not installed; vLLM verified by import; serve test pending or failed; TTS container test pending or failed. Inference results: clip_text_best_b8 throughput: 240.08 QPS, average latency: 4.164 ms; clip_vision_fp16_b8 throughput: 66.363 QPS, average latency: 15.067 ms; dinov2_fp16_b8 throughput: 9.651 QPS, average latency: 103.612 ms; resnet50_best_b8 throughput: 597.18 QPS, average latency: 1.673 ms; yolov8s_best_b1 throughput: 1279.56 QPS, average latency: 0.78 ms. qwen25_1p5b_q4km covers parallelism of 2/4/6/8/16/32, with best total throughput 3249.66 tokens/s (parallel=32), peak temperature 45C, peak power consumption 35.07W. qwen35_27b_claude46_q3ks covers parallelism of 2/4/6/8/16/32, with best total throughput 124.18 tokens/s (parallel=32), peak temperature 60C, peak power consumption 37.69W. kokoro_tts completed the request with an average latency of __LATENCY__, peak temperature of __TEMP__, and peak power consumption of __POWER__. Continuous CPU+GPU load test for 600s has been executed. GPU peak temperature: 112°C, CPU peak temperature: 98.843°C, tj peak: 111.906°C, GPU peak power consumption: 95.5W, VIN_SYS_5V0 peak power: 18.086W, fan PWM peak value: . The aggregated system peak temperature is `112`°C, peak power consumption is `95.59`W, and fan PWM peak value is `73`. Additional notes: host torch not installed; host vllm not installed; vLLM verified by import; serve test pending or failed; TTS container test pending or failed. ## lzc-pod-QlqOCy lzc-pod-QlqOCy is located at `lzc-pod-QlqOCy.lan`, identified device model is `NVIDIA Jetson AGX Thor Developer Kit`, system version is `Ubuntu 24.04.3 LTS`, L4T/JetPack information is `# R38 (release), REVISION: 2.2, GCID: 42205042, BOARD: generic, EABI: aarch64, DATE: Thu Sep 25 22:47:11 UTC 2025`, kernel version is `6.8.12-tegra`, CPU core count is `14`, total memory is `122.8GiB`. Basic capabilities: pip proxy `PASS`, Docker pull `PASS`, CUDA `PASS`, PyTorch container ``，TensorRT `PASS`，多媒体 `PASS`，vLLM/LLM 推理 ``. Model inference results: - clip_text_best_b8: throughput 257.857 QPS, average latency 3.877 ms - clip_vision_fp16_b8: throughput 65.38 QPS, average latency 15.294 ms - dinov2_fp16_b8: throughput 10.608 QPS, average latency 94.269 ms - resnet50_best_b8: throughput 636.926 QPS, average latency 1.569 ms - yolov8s_best_b1: throughput 1393.67 QPS, average latency 0.716 ms No Llama.cpp stress test performed. No TTS container test performed. No CPU+GPU full-load test performed. Current aggregated peak system temperature is ``C，峰值功耗为 `` W, fan PWM peak is `83`. Additional notes: No abnormal conditions reported. ## lzc-pod-juyIZt lzc-pod-juyIZt is located at `lzc-pod-juyIZt.lan`, identified as `NVIDIA Jetson AGX Orin Developer Kit`, with system version `Ubuntu 22.04.5 LTS`, L4T/JetPack information `# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan 8 01:49:37 UTC 2025`, kernel `5.15.148-tegra`, CPU core count `12`, total memory `61.4GiB`. Basic capabilities include pip proxy `FAIL`, Docker pull `FAIL`, CUDA `PASS`, PyTorch container `FAIL`, TensorRT `PASS`, multimedia `PASS`, vLLM/LLM inference `FAIL`. Inference performance metrics include: clip_text_best_b8 throughput: 71.087 QPS, average latency: 14.066 ms; clip_vision_fp16_b8 throughput: 19.281 QPS, average latency: 51.864 ms; dinov2_fp16_b8 throughput: 3.244 QPS, average latency: 308.234 ms; resnet50_best_b8 throughput: 501.002 QPS, average latency: 1.995 ms; yolov8s_best_b1 throughput: 566.905 QPS, average latency: 1.763 ms. LLaMA.cpp stress testing was not performed. TTS container testing was not performed. CPU+GPU full-load testing was not performed. Current aggregated peak system temperature is ``C，峰值功耗为 `` W, and maximum fan PWM value is `130`. Additional notes: Docker Hub image pull timeout; preloaded/synced images used; host torch not installed; host vLLM not installed; vLLM verified via import; serve test pending or failed. ## Raw Outputs - Comparison table: `/home/compiler/lzc_thor/results/summary/comparison.csv` - Workload table: `/home/compiler/lzc_thor/results/summary/workloads.csv` - Raw artifacts root: `/home/compiler/lzc_thor/results/raw` --- # ONNX View the inputs and outputs of ONNX # Validation - Automated validation report supplement see `2026-03-18-validation.md` - Control end working directory is `~/lzc_thor` - Prefer internal network path `https://dl.corp.linakesi.cn/...` for TensorRT model downloads; use `curl -k` if necessary - vLLM embedding results supplement see `../benchmark/qwen3-embedding-0.6b-vllm-thor.md` - vLLM Qwen3-VL embedding failure records see `../benchmark/qwen3-vl-embedding-2b-vllm-thor.md` ``` python -c " import onnx model = onnx.load('zh-clip-vit-roberta-large-patch14-vision-openvino.onnx') for input in model.graph.input: print(f'Input name: {input.name}') for dim in input.type.tensor_type.shape.dim: print(f' Dimension: {dim.dim_value}') " ``` # ollama 1. Image usage: ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04 2. ```sh docker run --gpus all --runtime nvidia --rm -ti -v /ssd/lzc-ai-agent/data/ollama/data:/data/models/ollama ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04 ``` | Models | Thor Tokens/s | Orin Tokens/s | |:----------|:---------------|:---------------| | qwen3:14b | 20.44 tokens/s | 13.36 tokens/s | | | | | | | | | # Resnet50 1. Data source: /usr/src/tensorrt/data/resnet50 2. Execution command: ```sh curl -L -O https://media.githubusercontent.com/media/onnx/models/refs/heads/main/validated/vision/classification/resnet/model/resnet50-v1-12.onnx /usr/src/tensorrt/bin/trtexec --onnx=resnet50-v1-12.onnx --shapes=data:4x3x224x224 --stronglyTyped --noDataTransfers --useCudaGraph --useSpinWait --duration=300 ``` | Equipment | Type | Batch | Throughput: images per second | Latency performance: per second per card | Temperature | Power supply | Input size | |------|--------|------|--------------------|------------------|------|-------|-------------| | Orin | Source type | 4 | 238.496 x 4 = 953 | 4.1875 ms | | 62.3w | 4×3×224×224 | | Thor | Source type | 4 | 588.473 x 4 = 2353 | 1.59375 ms | | - | 4×3×224×224 | | | | | | | | | | | Orin | Source type | 8 | 129.226 x 8 = 1033 | 7.73438 ms | 70 | 63.4w | 8×3×224×224 | | Thor | Source type | 8 | 317.383 x 8 = 2539 | 3.07031 ms | 78 | - | 8×3×224×224 | | | | | | | | | | | Orin | int4 | 8 | 129.157 x 8 = 1033 | 7.73438 ms | 72 | 63.4w | 8×3×224×224 | | Thor | int4 | 8 | 319.255 x 8 = 2554 | 3.07031 ms | 78 | 195w | 8×3×224×224 | | | | | | | | | | | Orin | int8 | 8 | 493.355 x 8 = 3946 | 2.02734 ms | 72 | 63.4w | 8×3×224×224 | | Thor | int8 | 8 | 486.597 x 8 = 3892 | 3.07031 ms | 78 | - | 8×3×224×224 | | | | | | | | | | | Orin | best | 8 | 129.797 x 8 = 1038 | 7.70264 ms | - | - | 8×3×224×224 | | Thor | best | 8 | 645.349 x 8 = 5162 | 1.51562 ms | - | - | 8×3×224×224 | # CLIP Vision 1. Data Source: curl -L -O http://dl.corp.linakesi.cn/lzc-image-search-model/onnx-export/vision/zh-clip-vit-roberta-large-patch14-vision-openvino.onnx 2. Execution Command ```sh /usr/src/tensorrt/bin/trtexec --onnx=zh-clip-vit-roberta-large-patch14-vision-openvino.onnx --shapes=pixel_values:8x3x224x224 --int8 --noDataTransfers --useCudaGraph --useSpinWait --duration=300 ``` | Equipment | Type | batch | Throughput: images per second | Latency performance: per second per card | Temperature | Power supply | Input size | |------|------|------|-------------------|------------------|------|-----------------------|-------------| | Orin | int8 | 8 | 8.72568 x 8 = 64 | 114.576 ms | - | - | 8×3×224×224 | | Thor | int8 | 8 | 54.6192 x 8 = 436 | 17.9922 ms | - | - | 8×3×224×224 | | | | | | | | | | | Orin | int4 | 8 | 7.25669 x 8 = 56 | 137.828 ms | 72 | 71W | 8×3×224×224 | | Thor | int4 | 8 | 16.6125 x 8 = 128 | 60.0859 ms | 82 | Average 112W (peak to 120W) | 8×3×224×224 | | | | | | | | | | | Orin | fp16 | 8 | 17.639 x 8 = 141 | 56.8906 ms | 75 | 74W | 8×3×224×224 | | Thor | fp16 | 8 | 60.4162 x 8 = 480 | 16.4219 ms | 100 | Average 150W (peak up to 180W) | 8×3×224×224 | | | | | | | | | | | Orin | best | 8 | 18.39 x 8 = 147 | 54.1875 ms | - | - | 8×3×224×224 | | Thor | best | 8 | 87.641 x 8 = 701 | 11.0312 ms | - | - | 8×3×224×224 | The Thor's temperature drops quickly, falling from 80°C to 60°C in just a few seconds. However, Thor's power consumption can occasionally reach above 180 W. # clip text 1. Data source: curl -L -O https://dl.corp.linakesi.cn/lzc-image-search-model/onnx-export/text/zh-clip-vit-roberta-large-patch14-text-openvino.onnx 2. Execution command ```sh /usr/src/tensorrt/bin/trtexec \ --onnx=zh-clip-vit-roberta-large-patch14-text-openvino.onnx \ --minShapes=input_ids:1x77,attention_mask:1x77 \ --optShapes=input_ids:8x77,attention_mask:8x77 \ --maxShapes=input_ids:32x77,attention_mask:32x77 \ --shapes=input_ids:8x77,attention_mask:8x77 \ --best \ --noDataTransfers \ --useCudaGraph \ --useSpinWait \ --duration=300 ``` | Equipment | Type | batch | Throughput: images per second | Latency performance: per second per card | Temperature | Power supply | Input size | |------|------|------|--------------------|------------------|------|------|----------| | Orin | best | 8 | 70.645 x 8 = 565 | 14.1562 ms | - | - | 8×77 | | Thor | best | 8 | 277.171 x 8 = 2217 | 3.57812 ms | - | - | 8x77 | | | | | | | | | | | Orin | fp16 | 8 | 70.6059 x 8 = 565 | 14.1562 ms | - | - | 8x77 | | Thor | fp16 | 8 | 177.429 x 8 = 1419 | 5.625 ms | - | - | 8×77 | | | | | | | | | | | Orin | int8 | 8 | 38.5414 x 8 = 208 | 25.9375 ms | - | - | 8x77 | | Thor | int8 | 8 | 246.038 x 8 = 1968 | 4.06298 ms | - | - | 8×77 | # yolov8s 1. Data source: curl -L -O http://dl.corp.linakesi.cn/lzc-ai/onnx/yolo/yolov8s.onnx (static ONNX model, does not support input shape specification; default input shape is 1x3x640x640) 2. Execution command ```sh /usr/src/tensorrt/bin/trtexec \ --onnx=yolov8s.onnx \ --best \ --noDataTransfers \ --useCudaGraph \ --useSpinWait \ --duration=300 ``` | Equipment | Type | batch | Throughput: images per second | Latency performance: per second per card | Temperature | Power supply | Input size | |------|------|------|--------------------|------------------|------|------|-------------| | Orin | best | 1 | 559.474 x 1 = 559 | 1.78125 ms | - | - | 1x3x640x640 | | Thor | best | 1 | 1320.01 x 1 = 1320 | 0.720703 ms | - | - | 1x3x640x640 | | | | | | | | | | | Orin | int4 | 1 | 188.694 x 1 = 188 | 5.29688 ms | - | - | 1x3x640x640 | | Thor | int4 | 1 | 565.56 x 1 = 565 | 1.73438 ms | - | - | 1x3x640x640 | | | | | | | | | | | Orin | int8 | 1 | 542.293 x 1 = 542 | 1.84375 ms | - | - | 1x3x640x640 | | Thor | int8 | 1 | 1203.38 x 1 = 1203 | 0.820312 ms | - | - | 1x3x640x640 | | | | | | | | | | | Orin | fp16 | 1 | 372.599 x 1 = 372 | 2.68652 ms | - | - | 1x3x640x640 | | Thor | fp16 | 1 | 1116.32 x 1 = 1116 | 0.890625 ms | - | - | 1x3x640x640 | # dino-v2 1. Data Source: curl -L -O http://dl.corp.linakesi.cn/lzc-ai/onnx/dinov2.onnx 2. Execution Command ```sh /usr/src/tensorrt/bin/trtexec \ --onnx=dinov2.onnx \ --minShapes=pixel_values:1x3x518x518 \ --optShapes=pixel_values:8x3x518x518 \ --maxShapes=pixel_values:32x3x518x518 \ --shapes=pixel_values:8x3x518x518 \ --best \ --noDataTransfers \ --useCudaGraph \ --useSpinWait \ --duration=300 ``` | Equipment | Type | batch | Throughput: images per second | Latency performance: per second per card | Temperature | Power supply | Input size | |------|------|------|------------------|------------------|------|------|-------------| | Orin | best | 8 | 3.01077 x 8 = 24 | 332.203 ms | - | - | 8x3x518x518 | | Thor | best | 8 | 3.48498 x 8 = 24 | 283.906 ms | - | - | 8x3x518x518 | | | | | | | | | | | Orin | int4 | 8 | 0.793952 x 8 = 6 | 1260.28 ms | - | - | 8x3x518x518 | | Thor | int4 | 8 | 1.60084 x 8 = 12 | 622.144 ms | - | - | 8x3x518x518 | | | | | | | | | | | Orin | int8 | 8 | 0.974057 x 8 = 8 | - ms | - | - | 8x3x518x518 | | Thor | int8 | 8 | 2.41794 x 8 = 16 | 397.922 ms | - | - | 8x3x518x518 | | | | | | | | | | | Orin | fp16 | 8 | 3.10437 x 8 = 18 | 322.125 ms | - | - | 8x3x518x518 | | Thor | fp16 | 8 | 9.98557 x 8 = 80 | 98.1719 ms | - | - | 8x3x518x518 | | | | | | | | | | | Orin | bf16 | 8 | 2.99196 x 8 = 24 | 334.617 ms | - | - | 8x3x518x518 | | Thor | bf16 | 8 | 10.9799 x 8 = 88 | 90.6172 ms | - | - | 8x3x518x518 | --- # Quick Translation (Neural Network Translation) Firefox Translations provides neural network translation models organized by language pair. The `mtrans` service in the AI Pod provides text translation, batch translation, language detection, and compatible endpoints for Immersive Translate, Kiss Translator, Selection Translate, and DeepLX. This document covers the `firefox-translations-93-pairs` model resource and the service verified at `http://192.168.1.224:18126` on 2026-10-12. Its API behavior matches [LinguaSpark Server](https://github.com/LinguaSpark/server). Mozilla's [training and inference documentation](https://mozilla.github.io/translations/docs/) describes the models and training process; the server defines the HTTP request format. ## Service Addresses and Models - AI Pod API address: - Health check: - LAN service address used for this verification: `http://192.168.1.224:18126`. For other devices, replace it with the actual IP address and mapped port. - Model resource ID: `firefox-translations-93-pairs`. - Full model repository ID: `manateelazycat/firefox-translations-93-pairs`. - POST requests use `Content-Type: application/json`. Requests do not need a `model` parameter. The service selects a loaded language-pair model based on `from` and `to`; the model resource ID identifies the model collection and is not a request parameter. The current service returns 404 for `/v1/models` and `/v1/chat/completions`. Use the translation endpoints below when integrating it, without adding `/v1` to the base address. ## Endpoint List All 8 endpoints below were tested on the service above and returned 200. | Method | Path | Purpose | Main Request Fields or Response Content | | --- | --- | --- | --- | | POST | `/translate` | Single or batch translation | `text`, `from`, `to`; returns translated text and language codes | | POST | `/detect` | Language detection | `text`; returns `language` | | POST | `/imme` | Immersive Translate compatible endpoint | `text_list`, `source_lang`, `target_lang`; returns `translations` | | POST | `/kiss` | Kiss Translator compatible endpoint | `text`, `from`, `to`; returns translated text and language codes | | POST | `/hcfy` | Selection Translate compatible endpoint | `text`, `source`, `destination`; translated text is in the `result` array | | POST | `/deeplx` | DeepLX compatible endpoint | `text`, `source_lang`, `target_lang`; translated text is in `data` | | GET | `/health` | Health check | Returns service status | | GET | `/metrics` | AI Pod usage statistics | Returns counters in Prometheus text format | The upstream [route definitions](https://github.com/LinguaSpark/server/blob/main/src/main.rs) and [endpoint field definitions](https://github.com/LinguaSpark/server/blob/main/src/endpoint.rs) can be used to check request formats. `/metrics` is an additional endpoint provided by the current AI Pod deployment. ## Single and Batch Translation ### Single Translation Request `POST /translate`: ```json { "text": "你好，世界。", "from": "zh", "to": "en" } ``` Actual response: ```json { "text": "Hello, world.", "from": "zh", "to": "en", "result": "Hello, world." } ``` | Field | Required | Description | | --- | --- | --- | | `text` | Yes | A string, or an array of strings for batch translation | | `from` | No | Source language code; detected automatically when omitted or set to `auto` | | `to` | Yes | Target language code, such as `en`, `zh`, `ja` | The response's `text` already contains the translation. The current deployment also returns `result` with the same content as `text`; the native upstream response only defines `text`, `from`, and `to`. ### Batch Translation Use `POST /translate` again, with `text` set to an array: ```json { "text": ["Hello, world!", "Good morning!"], "from": "en", "to": "zh" } ``` Actual response: ```json { "text": ["你好,世界!", "早上好!"], "from": "en", "to": "zh", "result": ["你好,世界!", "早上好!"] } ``` A batch uses the same source and target languages for all texts, and the returned array follows the input order. Automatic detection uses the first text to determine the source language, so do not mix source languages in one batch. The current service has no separate `/translate/batch` route. [Batch language handling](https://github.com/LinguaSpark/server/blob/main/src/translation.rs) ## Language Detection Request `POST /detect`: ```json { "text": "Hello, world!" } ``` Actual response: ```json { "language": "en" } ``` This endpoint returns a language code without translating. If the source language is known, you can specify it directly in the translation request. ## Compatible Endpoints for Browser Translation Extensions These endpoints use the same translation models, with request and response structures adapted to different clients. ### Immersive Translate Request `POST /imme`: ```json { "source_lang": "en", "target_lang": "zh", "text_list": ["Hello, world!", "Good morning!"] } ``` Actual response: ```json { "translations": [ {"detected_source_lang": "en", "text": "你好,世界!"}, {"detected_source_lang": "en", "text": "早上好!"} ] } ``` `source_lang` can be omitted or set to `auto`. Texts in `text_list` should use the same source language. ### Kiss Translator Request `POST /kiss`: ```json { "text": "Hello, world!", "from": "en", "to": "zh" } ``` Actual response: ```json { "text": "你好,世界!", "from": "en", "to": "zh" } ``` This endpoint takes a single string for `text`. ### Selection Translate Request `POST /hcfy`: ```json { "text": "Hello, world!", "source": "英语", "destination": ["中文(简体)"] } ``` Actual response: ```json { "text": "Hello, world!", "from": "英语", "to": "中文(简体)", "result": ["你好,世界!"] } ``` Here, `text` retains the original text, while the translation is in `result`. Upstream maps Chinese language names to the codes for Chinese, English, and Japanese; other languages can use their codes directly. The `destination` array selects one target language and does not imply that the response contains translations into multiple target languages. ### DeepLX Request `POST /deeplx`: ```json { "text": "Hello, world!", "source_lang": "EN", "target_lang": "ZH" } ``` The verified response structure is shown below; `id` varies with each request: ```json { "code": 200, "id": 1791789009008, "data": "你好,世界!", "alternatives": [], "source_lang": "EN", "target_lang": "ZH", "method": "Free" } ``` ## Health Check and Usage Statistics Request `GET /health`; the actual response is: ```json { "status": "ok" } ``` This response does not include the loading status of individual language pairs. To confirm that a translation direction is available, send a translation request for that direction. Request `GET /metrics` to return the following counters; their values change as requests are made: ```text # TYPE aipod:prompt_tokens_total counter aipod:prompt_tokens_total 48 # TYPE aipod:generation_tokens_total counter aipod:generation_tokens_total 46 # TYPE aipod:requests_total counter aipod:requests_total 10 ``` These are cumulative counters for the current deployment. They do not establish that tokens are counted in the same way as for OpenAI models. ## Model Language Coverage The [model repository](https://huggingface.co/manateelazycat/firefox-translations-93-pairs/tree/main) contains 93 directional language pairs covering 51 languages: 43 languages support translation both to and from English, for 86 directions in total, plus 7 one-way models translating into English. The number of model resource directories does not mean that all directions have been verified on the current service. All direct models have English as either the source or target language. Translation between other languages can use English as an intermediate language, provided that both models are loaded. For example, Chinese to French uses `zh → en → fr`; this direction was tested successfully. [Pivot inference implementation](https://github.com/LinguaSpark/server/blob/main/src/inference.rs) ### 43 Languages with Translation Both to and from English | Language Code | Language Name | | --- | --- | | ar | Arabic | | az | Azerbaijani | | bg | Bulgarian | | bn | Bengali | | ca | Catalan | | cs | Czech | | da | Danish | | de | German | | el | Greek | | es | Spanish | | et | Estonian | | fa | Persian | | fi | Finnish | | fr | French | | gu | Gujarati | | he | Hebrew | | hi | Hindi | | hr | Croatian | | hu | Hungarian | | id | Indonesian | | is | Icelandic | | it | Italian | | ja | Japanese | | kn | Kannada | | ko | Korean | | lt | Lithuanian | | lv | Latvian | | ml | Malayalam | | ms | Malay | | nl | Dutch | | pl | Polish | | pt | Portuguese | | ro | Romanian | | ru | Russian | | sk | Slovak | | sl | Slovenian | | sq | Albanian | | sv | Swedish | | ta | Tamil | | te | Telugu | | tr | Turkish | | uk | Ukrainian | | zh | Chinese | The code for English is `en`. Icelandic includes both English to Icelandic and Icelandic to English directions. ### 7 Languages with Models Only for Translation into English | Language Code | Language Name | | --- | --- | | be | Belarusian | | bs | Bosnian | | mt | Maltese | | nb | Norwegian (Bokmål) | | nn | Norwegian (Nynorsk) | | sr | Serbian | | vi | Vietnamese | These 7 languages can be used as source languages and translated through English into other languages that have an English-to-target model. The current model collection does not support them as target languages for translation from other languages. ### Simplified and Traditional Chinese Tags Language codes use ISO 639-1. The current service normalizes codes with region or script suffixes to their primary language code; for example, both `zh-Hans` and `zh-Hant` are treated as `zh`. Tests with `zh-Hant` as the target language still returned Simplified Chinese. This parameter cannot be used to request Traditional Chinese output. ## Authentication and Errors The service used for this verification accepted requests without authentication information. Upstream supports configuring authentication through `API_KEY`. When enabled, use the `Authorization: Bearer YOUR_API_KEY` request header or the `?token=YOUR_API_KEY` query parameter; `/health` does not require authentication. Unsupported translation directions return 400. The verified response for English to Belarusian was: ```json { "error": "Translation from 'en' to 'be' is not supported" } ``` The following paths return 404 on the current deployment: `/`, `/docs`, `/docs/index.html`, `/openapi.json`, `/version`, `/languages`, `/models`, `/v1/models`, `/translate/batch`, `/v1/chat/completions`. A 404 on the home page does not mean that the service is unavailable; use `/health` and translation requests to verify it. ## Standalone Usage Use the API directly through an HTTP client on a computer that can access the service. The examples below use the LAN address from this verification; to use the AI Pod domain, simply replace the base address. ### Command Line ```bash curl --fail-with-body --connect-timeout 5 --max-time 60 \ 'http://192.168.1.224:18126/translate' \ -H 'Content-Type: application/json' \ --data '{"text":"Hello, world!","from":"en","to":"zh"}' ``` ### Python The following example sends a request using the Python standard library, with no additional dependencies to install: ```python import json from urllib.request import Request, urlopen base_url = "http://192.168.1.224:18126" payload = { "text": ["Hello, world!", "Good morning!"], "from": "en", "to": "zh", } request = Request( f"{base_url}/translate", data=json.dumps(payload, ensure_ascii=False).encode("utf-8"), headers={"Content-Type": "application/json"}, method="POST", ) with urlopen(request, timeout=60) as response: result = json.load(response) print(result["text"]) ``` --- # Precise Translation (Large Language Model) An optimized high-performance large language model is built into the AI Pod, specifically designed for high-quality translation tasks, delivering accurate, fluent, and context-appropriate translations—ideal for scenarios with stringent translation quality requirements. Applications within the LCMD can directly invoke this service via the `trans` API exposed by the AI Pod. ## Model Features - **High-Quality Translation**: Powered by a large language model, delivering accurate and context-aware translations - **Multi-language Support**: Supports mutual translation between major languages - **Flexible Configuration**: Allows customization of translation requirements, such as language pairs and translation styles - **Strong Compatibility**: Fully compatible with OpenAI API format, easing integration - **High-Performance Deployment**: Optimized for deployment using the vLLM framework, with fast response times - **Excellent Foundation Model**: Based on the Qwen3 model, delivering superior performance in translation tasks ## Service Information - **API Endpoint**: - **OpenAI-Compatible Endpoint**: - **Default model name**: `aipod-trans` (current version points to Qwen3 model; it is recommended to use alias `aipod-trans` for easier API calls and seamless future upgrades) - **Response mode**: Supports both streaming and non-streaming responses - **Custom configuration**: Supports specifying translation language, style, and other requirements via system prompt ## Independent deployment usage 1. Save the following content to a `docker-compose.yml` file ```yml services: trans: image: registry.lazycat.cloud/x/lzc-aipod-trans:3480c9d ports: - 3000:3000 environment: - LANGUAGE=en_US:en - LANG=en_US.UTF-8 - LC_ALL=en_US.UTF-8 - VLLM_CONFIGURE_LOGGING=0 - VLLM_DO_NOT_TRACK=1 - VLLM_NO_USAGE_STATS=1 - MAX_MODEL_LEN=2048 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 10s timeout: 30s retries: 100 start_period: 1s ``` 2. Run `docker-compose up -d` to start in the current directory 3. After startup, the `http://127.0.0.1:3000/v1/chat/completions` can be invoked --- # Qwen3TTS (OpenAI-Compatible TTS) The `Qwen3TTS` is an OpenAI-compatible text-to-speech service packaged based on `faster-qwen3-tts`. The recommended service name within the AI Pod is `qwen3tts-ai`, suitable for integration with OpenWebUI, OpenAI-compatible SDKs, or any custom HTTP client. Compared to `Kokoro tts-ai`, this service features: - Compatibility with `POST /v1/audio/speech` - Support for `wav` / `pcm` streaming audio output - Warm-up of the model during startup, then marking ready externally - Support for multi-voice query interface `/v1/audio/voices` - Support for request-level `language`, `instruct`, and `seed` - In `clone` mode, supports generation and reuse of `speaker.pt` - Additional provision of `POST /v1/audio/voice-clone/pt` ## Service Endpoints - Recommended service domain: `qwen3tts-ai` - If using the current example compose file, `qwen3-tts-ai` can also be used - Health check: `/health` - Voice list: `/v1/audio/voices` - TTS endpoint: `/v1/audio/speech` - PT generation endpoint:`/v1/audio/voice-clone/pt` Example access forms after deployment to AI Pod: - Service address: `https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}` - Health check: `https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/health` - Voice list: `https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/voices` - TTS interface: `https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech` - PT generation interface: `https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/voice-clone/pt` ## Startup and Launch Behavior `qwen3tts-ai` differs from traditional "process startup equals ready" services in that it performs a startup warmup round before officially accepting requests. The startup process is as follows: 1. Load model and voice configuration 2. Initialize CUDA graphs 3. Execute one warmup request 4. `/health`'s `ready` changes to `true` 5. Only then should upstream AI applications or scripts initiate formal TTS requests Key considerations during execution: - The container process being up does not mean the service is ready - Applications should poll `/health` and consider `ready=true` as the readiness indicator - The Docker `healthcheck` in the documentation is primarily for detecting HTTP service liveness; if strict readiness is required, additional checks should be performed on the `/health` response body's `ready` field ## Health Check ### `GET /health` This interface is used to determine whether the service has completed model loading and startup warmup. Request: ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/health ``` Response example: ```json { "status": "ok", "ready": true, "model_loaded": true, "mode": "custom", "voices": ["aiden", "dylan", "vivian"], "startup_warmup_enabled": true, "startup_warmup_completed": true, "startup_warmup_seconds": 15.11 } ``` - Initiate formal TTS requests only when `ready=true` is active - The `mode` field can be used to determine the current service operating mode: `custom` indicates CustomVoice fixed voice mode, `clone` indicates Base voice fingerprint cloning mode ## Voice List Interface ### `GET /v1/audio/voices` Request: ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/voices ``` Response example: ```json { "voices": [ "aiden", "dylan", "eric", "ono_anna", "ryan", "serena", "sohee", "uncle_fu", "vivian" ], "default_voice": "vivian", "mode": "custom" } ``` - `voice` Request parameters must be obtained from here - If an invalid `voice` is provided, the service will fallback to `default_voice` ## TTS Interface ### `POST /v1/audio/speech` This interface is compatible with common OpenAI TTS API call patterns. Typical JSON request headers: ```http Content-Type: application/json Accept: audio/wav ``` JSON request body example: ```json { "model": "tts-1", "input": "欢迎使用 Qwen3TTS 服务。", "voice": "vivian", "response_format": "wav", "language": "Chinese", "instruct": "请用温和、清晰、偏慢一点的语气朗读。", "seed": 1234 } ``` Parameter Description: | Parameters | Type | Required | Description | |---|---|---|---| | `model` | string | no | Compatible field, suggest passing `tts-1` fixedly | | `input` | string | Yes | Pending synthesis text, must not be empty | | `voice` | string | no | Voice name; suggest first obtain via `/v1/audio/voices` | | `response_format` | string | no | Support `wav` / `pcm` / `mp3` | | `language` | string | no | Language at request level, overrides the default language configuration for this voice upon input | | `instruct` | string | no | Custom instructions for controlling tone, style, rhythm, or accent bias | | `seed` | integer | no | Per-request random seed. Same text, voice, language, instruction, and model version, fixing the same `seed` can reproduce the same audio output. | `instruct` Execution rules: - Available when the service is running in `custom` or `clone` mode - If `instruct` is passed in the request, it takes precedence over the default `instruct` configured in the server's static voice settings - If `instruct` is not provided in the request, the current service's default behavior is retained - If you're only concerned with OpenAI-compatible invocation, this can be understood as an optional extension field Request-level extension parameters: - Exposed and recommended fields: `language`, `instruct`, `seed` - `language` is used to override the default voice language - `instruct` is used to control tone, style, rhythm, or accent preference - `seed` enables reproducibility of the same request within the same service version - `voice_clone_pt` is only available in `clone` mode and requires file upload via `multipart/form-data` Usage recommendations for `seed`: - Recommend passing a non-negative integer, e.g., `1234` - To reproduce results, `input`, `voice`, `language`, `instruct` should all be fixed simultaneously,`response_format` - Reproduce the conclusion under the same container image version, same model weights, and same voice configuration. - When `seed` is not provided, the request follows the default random path and does not guarantee consistent results across multiple requests. `language` execution rules: - When `language` is provided in the request, it takes precedence over the default language set in the server's static voice configuration. - If `language` is not provided in the request, the current service default behavior is adopted; for Orin custom images, the default is generally `Auto`. - Common valid values can be used based on model capabilities, such as: `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`. `voice` execution rules: - The `voice` request parameter is case-insensitive. It is recommended to first obtain valid values via `/v1/audio/voices`. - If an invalid `voice` is provided, the service will fall back to `default_voice`. Currently unsupported request-level parameters: - `speaker` - `instructions` - `temperature` `do_sample` - `top_k` - `top_p` - `repetition_penalty` Notes: - In the OpenAI interface, use `voice` instead of `speaker` - In the OpenAI interface, use `instruct` instead of `instructions` - Internal sampling-related parameters are currently not exposed as OpenAI request fields; they should not be passed externally in the documentation - `voice_clone_pt` is not a JSON field; if `pt` file needs to be passed, use `multipart/form-data` instead Response Notes: | `response_format` | Content-Type | Is it streaming? | Description | |---|---|---|---| | `wav` | `audio/wav` | Yes | First return the WAV header, then continuously return PCM audio frames | | `pcm` | `audio/pcm` | Yes | Return 16-bit little-endian PCM audio stream | | `mp3` | `audio/mpeg` | no | First generate the complete audio, then encode it as an mp3. | ## PT Voice Cloning Interface The following capabilities are available only when the service runs in `clone` mode. Typical container image: - `registry.lazycat.cloud/x/faster-qwen3-tts:0.6b-base-clone-openai-orin-v2` In `clone` mode, it is recommended to fix `voice` as `dynamic`, and determine the speaker via request-level `voice_clone_pt` or static `voices.json`. ### `POST /v1/audio/voice-clone/pt` This interface extracts a speaker embedding from a reference audio segment and directly returns the corresponding `pt` file to the caller. Request methods: - `Content-Type: multipart/form-data` - File field: `ref_audio` - Optional form field: `filename` Request example: ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/voice-clone/pt \ -F "ref_audio=@ref_audio.wav;type=audio/wav" \ -F "filename=speaker.pt" \ --output speaker.pt ``` - The server does not persist the `pt` file - The calling party is responsible for saving the `pt` file, for example by storing it in object storage, NAS, database, or business-side file directories - For subsequent requests from the same speaker, the same `pt` can be reused ### Use `pt` directly in the request Once you have obtained `speaker.pt`, you can upload it directly in `POST /v1/audio/speech`. In this case, the request must use `multipart/form-data`, as both text and file fields need to be transmitted. Request example: ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Accept: audio/wav" \ -F "model=tts-1" \ -F "input=今天的风从江面慢慢吹过来，像有人把一句话轻轻放在耳边。" \ -F "voice=dynamic" \ -F "response_format=wav" \ -F "language=Chinese" \ -F "seed=1234" \ -F "voice_clone_pt=@speaker.pt;type=application/octet-stream" \ --output speech.wav ``` Available file field names: - `voice_clone_pt` - `speaker_pt` - `pt` Execution rules: - Available only in `clone` mode - When uploading `voice_clone_pt` via request, it takes precedence over the static `voices.json` in the server's `speaker_pt` - For the same `pt` paired with different text inputs, the same speaker characteristics should be maintained; to improve reproducibility, it is recommended to also fix `seed` ### Mount `pt` statically into the service If you don't want to re-upload `speaker.pt` on every request, you can mount the `pt` file along with `voices.json` into the container. `voices.json` example: ```json { "alice": { "speaker_pt": "/data/voices/alice.pt", "language": "Chinese" }, "narrator": { "speaker_pt": "/data/voices/narrator.pt", "language": "Chinese", "instruct": "请用平稳、自然的旁白语气朗读。" } } ``` After startup: - `GET /v1/audio/voices` returns `alice` and `narrator` - When calling `POST /v1/audio/speech`, pass `voice=alice` or `voice=narrator` directly - The request body can continue using standard JSON; there's no need to re-upload `pt` each time - `speaker_pt` and `instruct` can be used simultaneously; if `voices.json` is configured with a default `instruct`, passing `instruct` in the request will override the static configuration - However, in the `speaker_pt` scenario, `instruct` is still considered experimental and its stability is not guaranteed; if you strongly depend on instruction following, prefer using the ICL mode of `ref_audio + ref_text` ## Example of non-streaming call ### Generate WAV ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"欢迎使用 Qwen3TTS 服务。","voice":"vivian","response_format":"wav"}' \ --output speech.wav ``` ### Generate WAV and pass to instruct ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"欢迎使用 Qwen3TTS 服务。","voice":"vivian","response_format":"wav","instruct":"请用温和、清晰、偏慢一点的语气朗读。"}' \ --output speech.wav ``` ### Generate WAV and fix seed ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"欢迎使用 Qwen3TTS 服务。","voice":"vivian","response_format":"wav","seed":1234}' \ --output speech.wav ``` Explanation: - When the same request body is called repeatedly, using the same `seed` consistently yields the same result - If you replace the text, voice, language, instruction, model version, or container image version, the output will also change. ### Generate WAV with explicit language + instruct + seed ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"其实我真的有发现，我是一个特别善于观察别人情绪的人。","voice":"vivian","response_format":"wav","language":"Chinese","instruct":"用特别愤怒的语气说","seed":1234}' \ --output speech.wav ``` ### Generate MP3 ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"欢迎使用 Qwen3TTS 服务。","voice":"vivian","response_format":"mp3"}' \ --output speech.mp3 ``` - `mp3` encodes audio after full audio generation, making it unsuitable for low-latency first-packet scenarios - For edge reception and playback, prefer `wav` or `pcm` ## Streaming Call Specification The "streaming" here refers to direct return of audio byte streams, not SSE text event streaming: - `wav` / `pcm` return audio in chunks via HTTP chunked transfer - `mp3` does not support real-time generation and return ### curl streaming save ```bash curl -N https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"今天我们测试流式音频返回。","voice":"vivian","response_format":"wav"}' \ --output stream.wav ``` ### Python Stream Reading ```python import requests url = f"https://qwen3tts-ai.{LAZYCAT_BOX_DOMAIN}/v1/audio/speech" payload = { "model": "tts-1", "input": "今天我们测试流式音频返回。", "voice": "vivian", "response_format": "wav", "language": "Chinese", "instruct": "请用自然停顿、比较平稳的语气朗读。", } with requests.post(url, json=payload, stream=True, timeout=600) as resp: resp.raise_for_status() with open("stream.wav", "wb") as f: for chunk in resp.iter_content(chunk_size=4096): if chunk: f.write(chunk) ``` ### JavaScript Streamed Reading ```javascript const resp = await fetch(`https://qwen3tts-ai.${boxDomain}/v1/audio/speech`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ model: "tts-1", input: "今天我们测试流式音频返回。", voice: "vivian", response_format: "wav", language: "Chinese", instruct: "请用自然、平稳的旁白语气朗读。" }) }); if (!resp.ok) { throw new Error(`HTTP ${resp.status}`); } const reader = resp.body.getReader(); const chunks = []; while (true) { const { value, done } = await reader.read(); if (done) break; chunks.push(value); } const blob = new Blob(chunks, { type: "audio/wav" }); ``` ## Common Errors ### 400 Parameter Error ```json { "detail": "'input' text is empty" } ``` Or: ```json { "detail": "response_format 'aac' not supported. Use: wav, pcm, mp3" } ``` ### 400 Voice Tone Not Found ```json { "detail": "Voice 'unknown' is not configured. Available voices: ['aiden', 'vivian']" } ``` ### 503 Service Not Ready ```json { "detail": "Model not loaded" } ``` ## Independent Deployment ### Solution One: CustomVoice with Fixed Voice 1. Save the following content into a `docker-compose.yml` file. ```yml services: faster-qwen3-tts: image: registry.lazycat.cloud/x/faster-qwen3-tts:0.6b-custom-openai-orin-v11 environment: HF_HUB_OFFLINE: "1" TRANSFORMERS_OFFLINE: "1" HF_HOME: /root/.cache/huggingface XDG_CACHE_HOME: /root/.cache QWEN_TTS_MODEL: /opt/models/Qwen3-TTS-12Hz-0.6B-CustomVoice QWEN_TTS_MODE: custom QWEN_TTS_DEFAULT_VOICE: vivian QWEN_TTS_LANGUAGE: Auto QWEN_TTS_CHUNK_SIZE: "8" QWEN_TTS_WARMUP_MAX_NEW_TOKENS: "32" volumes: - /etc/timezone:/etc/timezone:ro - ${LZC_AGENT_CACHE_DIR}/cache:/root/.cache - ${LZC_AGENT_DATA_DIR}/data:/data labels: - "traefik.enable=true" - "traefik.http.routers.${LZC_SERVICE_ID}-qwen3tts.rule=Host(`qwen3tts-ai`)" expose: - 8000 healthcheck: test: [ "CMD", "curl", "-f", "http://localhost:8000/health" ] interval: 30s timeout: 5s retries: 100 start_period: 1s networks: default: external: true name: traefik-shared-network ``` 2. Run `docker-compose up -d` to start in the current directory 3. First use `http://127.0.0.1:8000/health` to check if the service is already `ready=true` 4. Then use `http://127.0.0.1:8000/v1/audio/voices` to get available voices 5. Finally invoke `http://127.0.0.1:8000/v1/audio/speech` Recommended post-start verification sequence: ```bash curl http://127.0.0.1:8000/health curl http://127.0.0.1:8000/v1/audio/voices curl http://127.0.0.1:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model":"tts-1","input":"启动校验。","voice":"vivian","response_format":"wav","instruct":"请用简短、清晰的方式朗读。"}' \ --output verify.wav ``` If you're debugging locally on the device instead of connecting to the AI Pod gateway, change the above `expose` to: ```yml ports: - "8000:8000" ``` Current recommended deployment guidelines: - Recommended to use `0.6b-custom-openai-orin-v11` - The recommended request fields for OpenAI interface currently are: `voice`, `language`, `instruct`, `seed` - If you need to reproduce the same audio, explicitly pass a fixed `seed` in the request - `speed` is currently accepted and compatible, but not yet activated; thus, it is not listed as a recommended parameter in the documentation ### Option Two: Base + Clone + PT If you want the caller to manage `speaker.pt` independently, or if you wish to bind the same `pt` into multiple named voice profiles, we recommend using the `clone` container image. `docker-compose.yml` example: ```yml services: faster-qwen3-tts: image: registry.lazycat.cloud/x/faster-qwen3-tts:0.6b-base-clone-openai-orin-v2 environment: HF_HUB_OFFLINE: "1" TRANSFORMERS_OFFLINE: "1" HF_HOME: /root/.cache/huggingface XDG_CACHE_HOME: /root/.cache QWEN_TTS_MODEL: /opt/models/Qwen3-TTS-12Hz-0.6B-Base QWEN_TTS_MODE: clone QWEN_TTS_DEFAULT_VOICE: dynamic QWEN_TTS_VOICES: /data/voices.json QWEN_TTS_LANGUAGE: Auto QWEN_TTS_CHUNK_SIZE: "8" QWEN_TTS_WARMUP_MAX_NEW_TOKENS: "32" volumes: - /etc/timezone:/etc/timezone:ro - ${LZC_AGENT_CACHE_DIR}/cache:/root/.cache - ${LZC_AGENT_DATA_DIR}/data:/data labels: - "traefik.enable=true" - "traefik.http.routers.${LZC_SERVICE_ID}-qwen3tts.rule=Host(`qwen3tts-ai`)" expose: - 8000 healthcheck: test: [ "CMD", "curl", "-f", "http://localhost:8000/health" ] interval: 30s timeout: 5s retries: 100 start_period: 1s networks: default: external: true name: traefik-shared-network ``` Recommended directory structure: ```text /data ├── voices.json └── voices ├── alice.pt └── narrator.pt ``` Recommended call sequence: 1. First invoke `POST /v1/audio/voice-clone/pt` to generate `speaker.pt` 2. Save `speaker.pt` to `/data/voices/*.pt` 3. Update `/data/voices.json` 4. Restart the service or redeploy to load the new `voices.json` 5. Then directly call `voice=<配置名>` to invoke `POST /v1/audio/speech` Alternatively, if you do not wish to restart the service, you may proceed with request-level upload: ```bash curl https://qwen3tts-ai.${LAZYCAT_BOX_DOMAIN}/v1/audio/speech \ -H "Accept: audio/wav" \ -F "model=tts-1" \ -F "input=这是一次请求级 pt 调用。" \ -F "voice=dynamic" \ -F "response_format=wav" \ -F "voice_clone_pt=@speaker.pt;type=application/octet-stream" \ --output speech.wav ``` ## Differences from the current `tts-ai` service - `tts-ai` This document corresponds to the `Kokoro` service - `qwen3tts-ai` is an OpenAI-compatible Qwen3TTS service - `qwen3tts-ai` is suitable for multi-voice, OpenAI-compatible clients, and streaming audio response scenarios --- # TTS (Text-to-Speech) An AI Pod has been deployed with the `Kokoro` `tts` service, which offers fast performance. If you need an OpenAI-compatible multi-voice TTS service, please refer to [Qwen3TTS API](/en/tts/qwen3tts_api.md). The entry point for the `qwen3tts-ai` service can be accessed via . . ## Online Demo You can access the online demo at . . The API of `tts` service can be accessed via . The API interface of `Qwen3TTS` is documented in [Qwen3TTS API](/en/tts/qwen3tts_api.md). If you need OpenAI-style `POST /v1/audio/speech`, or require querying `/v1/audio/voices`, using extended parameters `instruct` / `language`, or performing startup validation based on `/health`'s `ready=true`, you should use `qwen3tts-ai` instead of the `tts-ai` described in this page. `qwen3tts-ai` This document explicitly states: - OpenAI-style interface supports request-level `language` - OpenAI-style interface supports request-level `instruct` - OpenAI interface now supports request-level `seed` - Under the same container image version, model, and request parameters, fixed `seed` ensures reproducible audio output - This document only includes request fields actually recommended for public use; parameters like `speed` that are compatible but not effective are omitted from formal parameter descriptions ## Standalone Deployment 1. Save the following content into a `docker-compose.yml` file ```yml services: tts: image: registry.lazycat.cloud/x/lzc-aipod-tts:763ebf7 ports: - 8880:8880 environment: - PYTHONPATH=/app:/app/api - USE_GPU=true - PYTHONUNBUFFERED=1 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8880/health"] interval: 30s timeout: 5s retries: 100 start_period: 1s ``` 2. Run `docker-compose up -d` to start the AI Pod in the current directory 3. After startup, access `http://127.0.0.1:8880/docs` ::: details Example ```vue ``` ::: --- # vLLM Distributed Inference on Jetson Orin AGX [Official Documentation](https://github.com/dusty-nv/jetson-containers/blob/master/docs/distributed-inference.md) ## Test Container Images 1. mitakad/vllm:0.11.2-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated 2. mitakad/vllm:0.12.0-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated 3. mitakad/vllm:0.13.0-r36.4.tegra-aarch64-cp312-cu129-24.04-pre-release ## Steps Note: The model must be downloaded separately and placed in the current directory. Execute the command on the master node. ``` pip install modelscope ~/.local/bin/modelscope download --model openai-mirror/gpt-oss-120b --local_dir ./openai-mirror/gpt-oss-120b ``` Connect the 10G network port on the master node to the 10G port of the AI Pod on another node using a network cable. 1. Master Node ``` sudo ip addr add 192.168.38.1/24 dev eno1 VLLM_HOST_IP=192.168.38.1 && \ VLLM_IFNAME=eno1 && \ docker run -d --rm --runtime nvidia -it \ --network host \ --shm-size=8g \ --privileged \ --ipc=host \ -v /dev/shm:/dev/shm \ -e NVIDIA_DRIVER_CAPABILITIES=all \ -e VLLM_HOST_IP=${VLLM_HOST_IP} \ -e UCX_NET_DEVICES=${VLLM_IFNAME} \ -e NCCL_SOCKET_IFNAME=${VLLM_IFNAME} \ -e GLOO_SOCKET_IFNAME=${VLLM_IFNAME} \ -e TP_SOCKET_IFNAME=${VLLM_IFNAME} \ -v `pwd`/:/models \ -v `pwd`/cache:/root/.cache/huggingface \ --name vllm_xray \ mitakad/vllm:0.11.2-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated docker exec -ti vllm_xray bash ray start -v --head --port 6379 --node-ip-address 192.168.38.1 --num-gpus 1 sysctl vm.drop_caches=3 ``` 2. Other Nodes ``` sudo ip addr add 192.168.38.2/24 dev eno1 VLLM_HOST_IP=192.168.38.2 && \ VLLM_IFNAME=eno1 && \ docker run -d --rm --runtime nvidia -it \ --network host \ --shm-size=8g \ --privileged \ --ipc=host \ -v /dev/shm:/dev/shm \ -e NVIDIA_DRIVER_CAPABILITIES=all \ -e VLLM_HOST_IP=${VLLM_HOST_IP} \ -e UCX_NET_DEVICES=${VLLM_IFNAME} \ -e NCCL_SOCKET_IFNAME=${VLLM_IFNAME} \ -e GLOO_SOCKET_IFNAME=${VLLM_IFNAME} \ -e TP_SOCKET_IFNAME=${VLLM_IFNAME} \ -v `pwd`/:/models \ -v `pwd`/cache:/root/.cache/huggingface \ --name vllm_xray \ mitakad/vllm:0.11.2-r36.4.tegra-aarch64-cp310-cu126-22.04-truncated docker exec -ti vllm_xray bash ray start -v --address 192.168.38.1:6379 --node-ip-address 192.168.38.2 --num-gpus 1 sysctl vm.drop_caches=3 ``` 3. Copy the model to another worker node in the same directory. The directory I placed is `~`. ``` rsync -aP ~/openai-mirror nvidia@192.168.38.2:~ ``` 4. Execute the following command on the primary node machine ``` docker exec -ti vllm_xray bash # http://192.168.1.128:1085 是我本地的代理，您可以去掉或者更改成您的 curl -x http://192.168.1.128:1085 -L https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken -o fb374d419588a4632f3f557e76b4b70aebbca790 # 测试是否正常 TIKTOKEN_RS_CACHE_DIR=$(pwd) python -c 'from openai_harmony import load_harmony_encoding; load_harmony_encoding("HarmonyGptOss")' # 启动模型 sysctl vm.drop_caches=3 TIKTOKEN_RS_CACHE_DIR=$(pwd) TRANSFORMERS_OFFLINE=1 vllm serve /models/openai-mirror/gpt-oss-120b --port 8000 \ --served-model-name gpt-oss-120b \ --max-model-len 2048 \ --max-num-batched-tokens 2048 \ --kv-cache-memory-bytes 3000M \ --max-num-seqs 10 \ --tensor-parallel-size 2 \ --enable-expert-parallel \ --gpu-memory-utilization 0.8 \ --distributed-executor-backend ray ``` 5. Access Access via curl on the terminal. `192.168.1.175` is the IP address in my local network; you need to replace it with your machine's IP address. ``` curl 'http://192.168.1.175:8000/v1/chat/completions' \ --header 'Content-Type: application/json' \ --data '{ "model": "gpt-oss-120b", "messages": [ { "role": "user", "content": "who are you?" } ], "stream": false }' ``` ## Test Data | model | Tokens/s | |:---------------------------|:---------| | Qwen3-4B-Instruct-2507-FP8 | 10.5 | | openai/gpt-oss-120b | 10.1 | ## Question 1. If you encounter errors like "repo_id is incorrect" when loading a local model with vLLM, the simplest solution is to manually modify the verification function in the HuggingFace Hub validation script. --- # BGE-M3 Embedded Model `BGE-M3` is the vector embedding service built into the AI Pod, suitable for semantic retrieval, RAG recall, and text similarity tasks. ## Service Information - Service address: - Embeddings address: - Default model name: `bge-m3` - Container image: `registry.lazycat.cloud/x/lzc-aipod-bge-m3:a7d1e2b` ## Standalone Deployment 1. Save the following content into a `docker-compose.yml` file. ```yml services: bgem3vllm: image: registry.lazycat.cloud/x/lzc-aipod-bge-m3:a7d1e2b ports: - 8000:8000 environment: - LANGUAGE=en_US:en - LANG=en_US.UTF-8 - LC_ALL=en_US.UTF-8 - VLLM_CONFIGURE_LOGGING=0 - VLLM_DO_NOT_TRACK=1 - VLLM_NO_USAGE_STATS=1 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 10s timeout: 30s retries: 100 start_period: 360s ``` 2. Run `docker-compose up -d` in the current directory to start 3. After startup, you can call `http://127.0.0.1:8000/v1/embeddings` to invoke ## Invocation Examples ```bash curl http://127.0.0.1:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{"model":"bge-m3","input":"算力舱文档检索测试"}' ``` --- # BGE-Reranker-V2-M3 Reranking Model `BGE-Reranker-V2-M3` is an AI Pod-based reranking service built into the system, typically used to perform secondary sorting of candidate results after vector retrieval, thereby improving retrieval accuracy. ## Service Information - Service address: - Health check address: - Container image: `registry.lazycat.cloud/x/lzc-aipod-bge-rerank:a7d1e2b` ## Independent Deployment 1. Save the following content into a `docker-compose.yml` file. ```yml services: bgererankerv2m3vllm: image: registry.lazycat.cloud/x/lzc-aipod-bge-rerank:a7d1e2b ports: - 8000:8000 environment: - LANGUAGE=en_US:en - LANG=en_US.UTF-8 - LC_ALL=en_US.UTF-8 - VLLM_CONFIGURE_LOGGING=0 - VLLM_DO_NOT_TRACK=1 - VLLM_NO_USAGE_STATS=1 healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 10s timeout: 30s retries: 100 start_period: 360s ``` 2. Run `docker-compose up -d` to start the service in the current directory 3. After startup, access `http://127.0.0.1:8000/health` to verify service health --- # VLLM By default, the AI Pod uses `ollama` to provide large model services, as it allows easy switching among various models. If your requirements are fixed and you do not frequently switch or release models, you can deploy your service using `vllm`. We recommend selecting the appropriate vLLM container image based on your hardware. For Orin, you can use the vLLM container image packaged for the AI Pod (`vllm 0.16.0 + cu126`): bash `docker run --rm -it registry.lazycat.cloud/x/lzc-aipod-vllm:d59c2ca bash` For Jetson Thor, prioritize testing the official NVIDIA vLLM container image: bash `docker run --rm --runtime=nvidia --ipc=host --network=host nvcr.io/nvidia/vllm:26.04-py3 python3 -c "import vllm, torch; print(vllm.__version__, torch.__version__, torch.cuda.get_device_name(), torch.cuda.get_device_capability())"` The container image has been verified on Thor: it recognizes `NVIDIA Thor` and supports the OpenAI-compatible `/v1/models` and `/v1/chat/completions` endpoints. If the model configuration includes multimodal fields but only text inference is required, add `--language-model-only` during startup. Thor can also use the Jetson AI-IOT container image: `docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor` This container image has successfully run `Qwen3.5-27B-Text-NVFP4-MTP`. However, `Qwen3.5/3.6 27B FP8` is currently not recommended as the default configuration for the GDN linear-attention path on Thor. If you wish to use the official Jetson container image, refer to [dustynv/vllm](https://hub.docker.com/r/dustynv/vllm/tags), for example: `docker pull dustynv/vllm:r36.4-cu129-24.04`. ## Additional Resources 1. https://hub.docker.com/r/mitakad/vllm 2. [Resource List](../aipod/resources.md) 3. https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html 4. https://docs.nvidia.com/deeplearning/frameworks/vllm-release-notes/ ## Example The following example uses the translation model pre-installed in the AI Pod (Qwen/Qwen3-4B-Instruct-2507 + vLLM): 1. Run the following command directly in the AI Pod: `docker run --rm -p 9999:3000 -ti registry.lazycat.cloud/x/lzc-aipod-trans:3480c9d` 2. Access it via the local network. ``` curl http://192.168.1.167:9999/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "aipod-trans", "messages": [{"role": "user", "content": "hello!"}]}' ``` Note: Remember to change the IP address (`192.168.1.167`) to your local device's IP address. --- # vllm-xray 1. Primary Node ``` sudo ip addr add 192.168.38.2/24 dev eno1 docker run --gpus all --network=host --shm-size 10.24g --name vllm-xray -d -ti -v `pwd`:/models -v `pwd`/cache:/root/.cache/huggingface -e VLLM_HOST_IP=192.168.38.2 dustynv/vllm:r36.4-cu129-24.04 bash docker exec -ti vllm-xray bash ray start --block --head --port=6379 --node-ip-address=192.168.38.2 ``` 2. Other Nodes ``` sudo ip addr add 192.168.38.1/24 dev eno1 docker run --gpus all --network=host --shm-size 10.24g --name vllm-xray -d -ti -v `pwd`:/models -v `pwd`/cache:/root/.cache/huggingface -e VLLM_HOST_IP=192.168.38.1 dustynv/vllm:r36.4-cu129-24.04 bash docker exec -ti vllm-xray bash ray start --block --address=192.168.38.2:6379 ``` 3. Main node starts the model ``` pip install pandas -i https://mirrors.ustc.edu.cn/pypi/simple # 下载最新版本的 cupy 才支持 aarch64 all_proxy=http://192.168.1.128:1085 curl -L https://github.com/cupy/cupy/releases/download/v13.5.0/cupy_cuda12x-13.5.0-cp312-cp312-manylinux2014_aarch64.whl -O VLLM_HOST_IP=192.168.38.2 vllm serve /models/models/Qwen/Qwen3-30B-A3B-GPTQ-Int4 --enable-auto-tool-choice --tool-call-parser hermes --reasoning_parser deepseek_r1 --tensor-parallel-size 1 --pipeline-parallel-size 2 ``` 4. Testing ``` time curl http://192.168.38.2:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "/models/models/Qwen/Qwen3-30B-A3B-GPTQ-Int4", "stream": true, "messages": [{"role": "user", "content": "你觉得中国和美国的差异性在什么地方?"}]}' ```