Overview

🚀 LLM Ollama + OpenClaw on NVIDIA Jetson™

Overview

Stop just chatting with AI—start deploying agents that work for you. The LLM Ollama + OpenClaw container provides a streamlined, hardware-accelerated environment for developing and deploying autonomous AI applications on NVIDIA Jetson™. By integrating OpenClaw, this stack transforms standard LLMs into proactive assistants capable of executing tasks, managing files, and browsing the web across 20+ messaging channels (WhatsApp, Telegram, Slack, etc.).

Optimized for the edge, this container ensures high performance, low latency, and 100% private AI interaction, making it the ideal "Mission Control" for industrial, enterprise, and personal automation.

💻 Host System Requirements

To ensure optimal performance and hardware acceleration, your host system must meet the following specifications:

Component	Version / Requirement
JetPack	6.2 (Production Release)
CUDA	12.6.68
cuDNN	9.3.0.75
TensorRT	10.3.0.30
OpenCV	4.8.0 (with CUDA support)

[!NOTE] Software versions (CUDA, TensorRT, etc.) are tied to JetPack 6.x. Please refer to the NVIDIA JetPack Documentation for compatibility details.

🌟 Key Features

Feature	Description
Ollama Backend	Run large language models (LLMs) locally with simple setup and high-speed inference.
OpenClaw Agent	Beyond chat: A proactive assistant that executes commands and manages files via messaging apps.
20+ Channels	Connect your AI to WhatsApp, Telegram, Discord, Slack, and more out-of-the-box.
ClawHub Ecosystem	Access 700+ community-built skills (Gmail, GitHub, Calendar, IoT control).
Control UI	Browser-based dashboard (Port `18789`) for managing channels, models, and sessions.
Hardware Accelerated	Fully optimized for NVIDIA Ampere architecture (Jetson Orin series).
Offline-First	Entirely self-hosted. Your data stays on-device with no internet required after setup.
Developer Ready	Includes a REST API, Modelfile customization, and Docker-friendly architecture.

🏗️ Container Description

Quick Information

The build.sh script initializes the following environment:

Container Name	Description
LLM-Ollama-OpenClaw-Jetson	A unified, hardware-accelerated environment containing the complete AI stack.

Container Highlights

Ollama Engine: Serves as the "Local Brain," managing model quantization (.gguf) and streaming output.
OpenClaw Gateway: Runs as a single Node.js process to route messages, maintain persistent memory across sessions, and invoke tools.
Skill Integration: Allows the agent to use "eyes and hands" to interact with the local filesystem, terminal, and web browsers.

🚀 Use Cases: From Chatting to Doing

Use Case	Transformation: The Agentic Advantage
Proactive AI Assistant	Execute tasks, not just prompts. Send a message via Telegram to run scripts, summarize local logs, or move files.
Industrial Edge AI	Use Ollama as the reasoning core to parse commands and make decisions based on local sensor data.
Private RAG Systems	Query sensitive internal manuals or technical docs offline using a vector database integration.
Omnichannel Automation	Use ClawHub skills to bridge your edge device with external tools like GitHub, Jira, or Google Calendar.
Multilingual Support	Deploy localized assistants that translate and summarize in real-time without cloud latency or costs.

📦 Model Information

The container is configured to use Ollama with OpenClaw for reasoning. The default model and behavior can be adjusted via the .env file.

Item	Description
Default Model	`MODEL_NAME` in `.env` (Default: `kimi-k2.5:cloud`)
Model Source	Ollama Model Library
Pull Command	`ollama pull <model-name>`
Agent Link	`openclaw models set ollama/<model-name>`

🛠️ Software Components

Component	Version	Description
Ollama	0.5.7	High-performance LLM inference engine.
OpenClaw	Latest	The agentic gateway and multi-channel assistant.
PyTorch	2.0.0+	Deep learning framework for local development.
ONNX Runtime	1.16.3	Cross-platform inference engine support.

🏁 Quick Start

For a detailed walkthrough, including the docker-compose.yml file and advanced networking setup, please visit the:

Advantech Container GitHub Repository

⚡ Installation

Ensure Docker, Docker Compose, and NVIDIA Container Toolkit are installed.
Clone the repository and run the build script:

git clone <repository-url>
cd LLM-Ollama-OpenClaw-on-NVIDIA-Jetson
chmod +x build.sh && sudo ./build.sh

💡 Best Practices & Recommendations

Model Selection: For Jetson Orin Nano/NX (8GB), we recommend models ≤ 3B parameters with Q4/Q8 quantization.

GPU Offloading: Ensure models are 100% loaded into GPU memory (VRAM) for zero-latency interactions.

Jetson Clocks: Run sudo jetson_clocks to lock frequencies for maximum inference stability.

Context Window: OpenClaw performs best with a context window of 64k tokens or higher for complex tool-calling.

⚠️ Known Limitations

First-Run Setup: Internet access is required during the initial build to install OpenClaw via npm.

RAM Usage: The container occupies ~5GB RAM on Orin NX 8GB. For smaller devices, increasing the NVMe swap size is highly recommended.

Loading Delay: Initial model inference may take ~10 seconds as the model weights are loaded into the GPU.