TensorRT Edge-LLM ran Qwen3.6-27B on a single Jetson AGX Thor Developer Kit.
The system achieved 52.33 tokens per second in the MLPerf Inference v6.1 benchmark. It completed all 1,007 turns of the workload in 24 minutes and 36 seconds. This is 6.4 times faster than the llama.cpp reference run which took over two hours.
The submission used NVFP4 quantization for weights and activations. It also used FP8 for the key-value cache and tree-based multi-token prediction. These techniques reduced repeated computation across agent turns.
KV cache reuse served about 96% of prompt tokens from hot memory. This avoided re-computing history for every new tool call in the conversation. Tree-based multi-token prediction added roughly 40% decoding performance gain.
Developers can download the calibrated Qwen3.6-27B NVFP4 checkpoint. The MLCommons Edge Agentic example provides dataset and client configuration details. Explore the TensorRT Edge-LLM release branch for engine build commands.
