Overview

WEDA LLM Cognitive RAG via vLLM on NVIDIA® Jetson Thor™

Short summary: A highly optimized serving and verification environment tailored for edge intelligence deployment of the WEDA-LLM-Cognitive-RAG stack on NVIDIA Jetson AGX Thor, combining vLLM model serving, Open WebUI, and built-in diagnostics for stability and performance.

About Advantech Container Catalog (ACC)

Advantech Container Catalog is a comprehensive collection of ready-to-use, containerized software packages designed to accelerate the development and deployment of Edge AI applications. By offering pre-integrated solutions optimized for embedded hardware, it simplifies challenges related to software and hardware compatibility, especially in GPU/NPU-accelerated environments.

Feature / Benefit	Description
Accelerated Edge AI Development	Ready-to-use containerized solutions for faster prototyping and deployment
Hardware Compatible	Reduces hardware and package incompatibility issues
GPU/NPU Access Ready	Supports passthrough for efficient hardware acceleration
Model Conversion & Optimization	Built-in model conversion and quantization recommendations
Optimized for CV & LLM Applications	Optimized stacks for vision and language workloads

Container Overview

This environment provides a high-performance serving and verification stack for deploying WEDA-LLM-Cognitive-RAG on NVIDIA Jetson AGX Thor (sm_110a). It integrates vLLM for inference serving, Open WebUI for browser-based interaction, and a diagnostic/benchmark suite to verify CUDA availability, dependency integrity, and service health on Jetson’s unified memory (UMA) architecture.

Demo

Use Case

Primary use cases:
- Local deployment of cognitive RAG and LLM workloads on NVIDIA Jetson edge devices
- Rapid evaluation and benchmarking of different model profiles and quantization settings
- Edge-side verification of package compatibility, serving readiness, and endpoint health

Key Features

Persistent vLLM compilation caching to avoid repeated JIT graph compilation on every container start
Auto-patched JIT compiler integration for sm_110a-aware CUDA compatibility
Profile-based model swapping via config/llm_config.yaml without manual container restarts
UMA memory management and cache flushing to reduce OOM issues during model loading
Built-in verification suite (wise-bench) that checks CUDA libraries, Python dependencies, and service endpoints

Host Device Prerequisites

Item	Specification
Compatible Hardware	NVIDIA Jetson AGX Thor (`sm_110a`)
Platform Version	NVIDIA JetPack SDK 7.1 or newer (L4T with CUDA toolkit 13.x)
Host OS	Ubuntu-based JetPack environment
Required Packages	Docker, NVIDIA Container Toolkit, Python 3.12+, Hugging Face access token
Software Installation Guide	https://developer.nvidia.com/embedded/jetpack

Required Software Packages on Host Device

Component	Version	Description
CUDA® Toolkit	13.x (JetPack 7.1 / L4T)	GPU runtime and compiler support
cuDNN	JetPack-provided	Deep neural network library
TensorRT™	JetPack-provided	Inference optimizer and runtime
VPI/Vulkan	JetPack / OS runtime	GPU acceleration and graphics support

Container Environment Overview

Software Components in the Image

Component	Version	Description
CUDA®	JetPack / container runtime supplied	GPU computing platform
cuDNN	JetPack / container runtime supplied	Deep neural network library
TensorRT™	JetPack / container runtime supplied	Inference optimizer
Python	3.12+	Host-side verification and orchestration
vLLM	Container-managed	High-throughput LLM serving engine
Open WebUI	Container-managed	Browser-based chat interface
PyTorch	Base image dependent	ML framework support
ONNX Runtime	Base image dependent	Cross-platform inference engine

Container Quick Start Guide

For container quick start, including build scripts, orchestration files, serving scripts, and related resources, please refer to the Advantech WEDA Container Repository: Advantech WEDA Container Repository

Supported AI Capabilities

Vision Models

Model Family	Versions	Notes
Vision Models	N/A	Primary focus is language-model serving and RAG workflows rather than computer vision inference

Language Models Recommendation

Model Family	Suggested Sizes	Memory Req.
User-defined profiles	7B to 70B+ parameters	Varies by quantization and runtime profile

Optimization tips: Prefer quantized profiles for smaller footprints, pre-cache model checkpoints on the host, and use the built-in profile switching flow for rapid experimentation.

Supported AI Model Formats

Format	Support Level	Notes
ONNX	Partial	Useful when model artifacts are exported or adapted for edge deployment
TensorRT™	Partial	Recommended for optimized inference paths when available
PyTorch (JIT)	Full	Supports local experimentation and profile-driven serving
TensorFlow SavedModel	Partial	Depends on availability in the selected deployment stack

Hardware Acceleration Support

Accelerator	Support Level	Compatible Libraries	Notes
CUDA®	Full	PyTorch, TensorFlow, ONNX Runtime	Native GPU acceleration on Jetson Thor
TensorRT™	Full	ONNX, TensorFlow, PyTorch	Optimized inference path when supported
NVENC/NVDEC	Partial	GStreamer, FFmpeg	Depends on runtime and media workload requirements