DeepSeek R1

Ollama deepseek-r1:70b

The deepseek-r1:70b model image in Ollama is 43GB, and it occupies approximately 46GB of memory after loading into RAM.

deepseek-r1:70b    d37b54d01a76    46 GB    100% GPU     8192       4 minutes from now

total duration:       7m10.425334424s
load duration:        56.463007ms
prompt eval count:    25 token(s)
prompt eval duration: 44.639105ms
prompt eval rate:     560.05 tokens/s
eval count:           1480 token(s)
eval duration:        7m10.322503313s
eval rate:            3.44 tokens/s

llama.cpp Benchmark

The following data comes from ~/lzc-aipod-benchmark/deepseek-r1-distill-70b/cumulative-report.md, corresponding to the model DeepSeek-R1-Distill-Llama-70B-Q4_K_M.gguf, with runtime llama.cpp. This section is the Distill 70B GGUF benchmark, not the above ollama deepseek-r1:70b pull log.

Device	Runtime	Weight	Single Stream	Peak Aggregate	Power	GPU Temp
Jetson AGX Orin	llama.cpp	GGUF Q4_K_M	3.346 tok/s (g64 / c1)	33.851 tok/s (g64 / c16)	50.32~68.787 W	89.5 C
Thor T4000	llama.cpp	GGUF Q4_K_M	2.909~2.912 tok/s (g64~g1024 / c1)	46.165 tok/s (g64 / c48)	22.066~22.295 W for c1; 37.798 W at peak	52~53 C for c1; 48.562 C at peak
Thor	llama.cpp	GGUF Q4_K_M	4.392~4.415 tok/s (g64~g1024 / c1)	-	37.443~39.254 W	68~75 C
Thor T5000	llama.cpp	GGUF Q4_K_M	4.458~4.552 tok/s (g64~g1024 / c1)	-	37.699~38.937 W	61~63 C

Notes

• Jetson AGX Orin corresponds to the test machine lzc-pod-juyIZt in the benchmark repository.
• THOR and T5000 have completed the single-stream steady-state retest of g64,128,256,512,1024 / c1 / 300s in this round.
• High-concurrency peak performance results are currently only available for T4000 and Jetson AGX Orin; therefore, peak aggregate throughput for THOR and T5000 is pending.
• Data source: /home/catdog/lzc-aipod-benchmark/deepseek-r1-distill-70b/cumulative-report.md