# 🤖 Setting Up Hermes AI Agent on a VM: Complete Self-Hosted Guide
**Date:** July 16, 2026
**Tags:** AI, Hermes, Self-Hosted, Docker, GPU, Systemd, Nginx, GPU Acceleration, AI Agents
---
## 🎯 Overview
[Hermes](https://github.com/hermes-ai/hermes) is a powerful, self-hosted AI agent framework that runs locally on your infrastructure. Unlike cloud-based AI services, Hermes gives you complete control over your data, models, and privacy while providing a rich agent ecosystem with tools for coding, browsing, file operations, and more.
### Why Hermes?
| Feature | Benefit |
|---------|---------|
| **🔒 Privacy First** | All data stays on your infrastructure |
| **🤖 Multi-Agent** | Run multiple specialized agents simultaneously |
| **🔌 Extensible** | Plugin architecture for custom tools |
| **🖥️ GPU Support** | CUDA/ROCm acceleration for local LLMs |
| **🔌 MCP Compatible** | Works with Model Context Protocol servers |
| **🌐 Web UI** | Beautiful dashboard for agent management |
| **📡 API First** | REST + WebSocket APIs for integration |
---
## 🏗️ Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Your VM (Linux) │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Nginx (Port 80/443) │ │
│ │ • SSL Termination • Rate Limiting • Auth Proxy │ │
│ └─────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Hermes Gateway (Port 8000) │ │
│ │ • REST API • WebSocket • Auth • Rate Limiting │ │
│ └─────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌───────────────────────┼───────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Agent 1 │ │ Agent 2 │ │ Agent N │ │
│ │(Coding) │ │ (Research) │ │ (Custom) │ │
│ └────┬────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ └───────────────────────┼────────────────┘ │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────┐ │
│ │ Model Router (9Router) │ │
│ │ Groq • Cerebras • Together • Fireworks • DeepInfra │ │
│ └─────────────────────────────┬──────────────────────────────┘ │
│ │ │
└────────────────────────────────┼─────────────────────────────────┘
▼
┌─────────────────────────────────────────────────────────────────┐
│ GPU Acceleration (Optional) │
│ • NVIDIA: CUDA 12.x + cuDNN • AMD: ROCm 6.x │
│ • Local Models: Llama-3.1-70B, Qwen2.5-72B, Mixtral-8x22B │
└─────────────────────────────────────────────────────────────────┘
```
---
## 📋 Prerequisites
### Hardware Requirements
| Component | Minimum | Recommended | For Local LLMs |
|-----------|---------|-------------|----------------|
| **CPU** | 4 cores | 8+ cores | 16+ cores |
| **RAM** | 16 GB | 32 GB | 64+ GB |
| **GPU** | Optional | RTX 3080+ | RTX 4090 / A100 / H100 |
| **VRAM** | N/A | 12 GB+ | 24-80 GB |
| **Storage** | 50 GB | 200 GB NVMe | 500 GB+ NVMe |
| **OS** | Ubuntu 22.04+ | Ubuntu 24.04 LTS | Ubuntu 24.04 LTS |
### Software Prerequisites
```bash
# Update system
sudo apt update && sudo apt upgrade -y
# Install Docker & Docker Compose
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker $USER
newgrp docker
# Install NVIDIA Container Toolkit (for GPU support)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Verify GPU access
docker run --rm --gpus all nvidia/cuda:12.4-base nvidia-smi
```
---
## 🔐 Environment Configuration
### 1. Create Project Structure
```bash
# Create project directories
sudo mkdir -p /opt/hermes/{config,data,logs,models,cache,ssl}
sudo chown -R $USER:$USER /opt/hermes
cd /opt/hermes
```
### 2. Environment Variables
```bash
cat > .env << 'EOF'
# ===========================================
# Hermes Core Configuration
# ===========================================
HERMES_VERSION=latest
HERMES_HOST=0.0.0.0
HERMES_PORT=8000
HERMES_WORKERS=4
LOG_LEVEL=info
LOG_FORMAT=json
# ===========================================
# Database (PostgreSQL recommended for production)
# ===========================================
DATABASE_URL=postgresql://hermes:<DB_PASSWORD>@postgres:5432/hermes
# For SQLite (dev only):
# DATABASE_URL=sqlite:///data/hermes.db
# Redis for caching & sessions
REDIS_URL=redis://redis:6379/0
# ===========================================
# Authentication & Security
# ===========================================
SECRET_KEY=<GENERATE_WITH: openssl rand -hex 32>
JWT_ALGORITHM=HS256
JWT_EXPIRATION_HOURS=24
REFRESH_TOKEN_EXPIRATION_DAYS=7
# Optional: OAuth providers
# GOOGLE_CLIENT_ID=<GOOGLE_CLIENT_ID>
# GOOGLE_CLIENT_SECRET=<GOOGLE_CLIENT_SECRET>
# GITHUB_CLIENT_ID=<GITHUB_CLIENT_ID>
# GITHUB_CLIENT_SECRET=<GITHUB_CLIENT_SECRET>
# ===========================================
# Model Provider API Keys (PLACEHOLDERS)
# ===========================================
# Get keys from respective provider dashboards
GROQ_API_KEY=<GROQ_API_KEY>
CEREBRAS_API_KEY=<CEREBRAS_API_KEY>
TOGETHER_API_KEY=<TOGETHER_API_KEY>
FIREWORKS_API_KEY=<FIREWORKS_API_KEY>
DEEPINFRA_API_KEY=<DEEPINFRA_API_KEY>
NOVITA_API_KEY=<NOVITA_API_KEY>
OPENROUTER_API_KEY=<OPENROUTER_API_KEY>
# Local model inference (optional)
OLLAMA_BASE_URL=http://ollama:11434
VLLM_BASE_URL=http://vllm:8000
# ===========================================
# 9Router Integration (Free Model Router)
# ===========================================
NINEROUTER_URL=http://ninerouter:4000/v1
NINEROUTER_API_KEY=<NINEROUTER_API_KEY>
# ===========================================
# MCP (Model Context Protocol) Servers
# ===========================================
MCP_SERVERS_CONFIG=/app/config/mcp_servers.yaml
# ===========================================
# Monitoring & Observability
# ===========================================
PROMETHEUS_ENABLED=true
METRICS_PORT=9090
JAEGER_ENDPOINT=http://jaeger:14268/api/traces
LOG_FORMAT=json
LOG_LEVEL=info
# ===========================================
# GPU / Hardware Acceleration
# ===========================================
# NVIDIA
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
CUDA_VISIBLE_DEVICES=0
# AMD ROCm (if using AMD GPU)
# ROCR_VISIBLE_DEVICES=0
# HSA_OVERRIDE_GFX_VERSION=11.0.0
# ===========================================
# File Storage
# ===========================================
UPLOAD_DIR=/app/data/uploads
MODELS_DIR=/app/models
CACHE_DIR=/app/cache
MAX_UPLOAD_SIZE=100MB
EOF
```
### 3. Generate Secure Secrets
```bash
# Generate secure secrets
openssl rand -hex 32 # SECRET_KEY
openssl rand -hex 32 # DB_PASSWORD
openssl rand -hex 32 # JWT_SECRET
```
---
## 🐳 Docker Compose Deployment
### Complete docker-compose.yml
```bash
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
# ===========================================
# PostgreSQL Database
# ===========================================
postgres:
image: pgvector/pgvector:pg16
container_name: hermes-postgres
restart: unless-stopped
environment:
POSTGRES_DB: hermes
POSTGRES_USER: hermes
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_INITDB_ARGS: "--auth-host=scram-sha-256"
volumes:
- ./data/postgres:/var/lib/postgresql/data
- ./config/init-db.sql:/docker-entrypoint-initdb.d/init.sql:ro
healthcheck:
test: ["CMD-SHELL", "pg_isready -U hermes -d hermes"]
interval: 10s
timeout: 5s
retries: 5
networks:
- hermes-network
deploy:
resources:
limits:
memory: 2G
reservations:
memory: 512M
# ===========================================
# Redis Cache
# ===========================================
redis:
image: redis:7-alpine
container_name: hermes-redis
restart: unless-stopped
command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
volumes:
- ./data/redis:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 3s
retries: 5
networks:
- hermes-network
deploy:
resources:
limits:
memory: 1G
reservations:
memory: 256M
# ===========================================
# 9Router - Free Model Router
# ===========================================
ninerouter:
image: ghcr.io/9router/9router:latest
container_name: hermes-ninerouter
restart: unless-stopped
ports:
- "4000:4000"
env_file: .env
environment:
- NINEROUTER_PORT=4000
- NINEROUTER_HOST=0.0.0.0
volumes:
- ./config/ninerouter:/app/config:ro
- ./logs/ninerouter:/app/logs
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:4000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 10s
networks:
- hermes-network
deploy:
resources:
limits:
memory: 2G
reservations:
memory: 512M
# ===========================================
# Ollama (Local Model Inference)
# ===========================================
ollama:
image: ollama/ollama:latest
container_name: hermes-ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ./models/ollama:/root/.ollama
- ./cache/ollama:/root/.cache/ollama
environment:
- OLLAMA_HOST=0.0.0.0
- OLLAMA_ORIGINS=*
- OLLAMA_NUM_PARALLEL=2
- OLLAMA_MAX_LOADED_MODELS=2
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
networks:
- hermes-network
healthcheck:
test: ["CMD", "ollama", "list"]
interval: 30s
timeout: 10s
retries: 3
# ===========================================
# vLLM (High-throughput LLM Inference)
# ===========================================
vllm:
image: vllm/vllm-openai:latest
container_name: hermes-vllm
restart: unless-stopped
ports:
- "8001:8000"
command: >
--model meta-llama/Meta-Llama-3.1-70B-Instruct
--tensor-parallel-size 1
--gpu-memory-utilization 0.9
--max-model-len 8192
--dtype auto
--trust-remote-code
--host 0.0.0.0
--port 8000
volumes:
- ./models/vllm:/root/.cache/huggingface
- ./cache/vllm:/root/.cache/vllm
environment:
- HF_TOKEN=<HUGGINGFACE_TOKEN>
- CUDA_VISIBLE_DEVICES=0
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
networks:
- hermes-network
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
# ===========================================
# Hermes Gateway
# ===========================================
hermes-gateway:
image: ghcr.io/hermes-ai/hermes-gateway:latest
container_name: hermes-gateway
restart: unless-stopped
ports:
- "8000:8000"
env_file: .env
environment:
- HERMES_HOST=0.0.0.0
- HERMES_PORT=8000
volumes:
- ./config/gateway:/app/config:ro
- ./data:/app/data
- ./logs/gateway:/app/logs
- ./models:/app/models:ro
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
ninerouter:
condition: service_healthy
ollama:
condition: service_healthy
networks:
- hermes-network
deploy:
resources:
limits:
memory: 4G
reservations:
memory: 1G
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
# ===========================================
# Hermes Web UI
# ===========================================
hermes-web:
image: ghcr.io/hermes-ai/hermes-web:latest
container_name: hermes-web
restart: unless-stopped
ports:
- "3000:3000"
environment:
- NEXT_PUBLIC_API_URL=http://localhost:8000
- NEXT_PUBLIC_WS_URL=ws://localhost:8000/ws
- NEXT_PUBLIC_APP_NAME=Hermes AI
depends_on:
- hermes-gateway
networks:
- hermes-network
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000"]
interval: 30s
timeout: 10s
retries: 3
# ===========================================
# Nginx Reverse Proxy
# ===========================================
nginx:
image: nginx:alpine
container_name: hermes-nginx
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./config/nginx/conf.d:/etc/nginx/conf.d:ro
- ./ssl:/etc/nginx/ssl:ro
- ./logs/nginx:/var/log/nginx
depends_on:
- hermes-gateway
- hermes-web
networks:
- hermes-network
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost/health"]
interval: 30s
timeout: 10s
retries: 3
# ===========================================
# Prometheus Metrics
# ===========================================
prometheus:
image: prom/prometheus:latest
container_name: hermes-prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./data/prometheus:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
- '--web.enable-lifecycle'
networks:
- hermes-network
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:9090/-/healthy"]
interval: 30s
timeout: 10s
retries: 3
# ===========================================
# Grafana Dashboards
# ===========================================
grafana:
image: grafana/grafana:latest
container_name: hermes-grafana
restart: unless-stopped
ports:
- "3001:3000"
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=<GRAFANA_PASSWORD>
- GF_INSTALL_PLUGINS=grafana-piechart-panel
volumes:
- ./data/grafana:/var/lib/grafana
- ./config/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
- ./config/grafana/datasources:/etc/grafana/provisioning/datasources:ro
depends_on:
- prometheus
networks:
- hermes-network
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000/api/health"]
interval: 30s
timeout: 10s
retries: 3
# ===========================================
# Jaeger Distributed Tracing
# ===========================================
jaeger:
image: jaegertracing/all-in-one:latest
container_name: hermes-jaeger
restart: unless-stopped
ports:
- "16686:16686"
- "6831:6831/udp"
- "14268:14268"
environment:
- COLLECTOR_OTLP_ENABLED=true
- SPAN_STORAGE_TYPE=badger
volumes:
- ./data/jaeger:/var/lib/jaeger
networks:
- hermes-network
networks:
hermes-network:
driver: bridge
name: hermes-network
volumes:
postgres-data:
redis-data:
ollama-models:
vllm-cache:
prometheus-data:
grafana-data:
jaeger-data:
EOF
```
---
## ⚙️ Configuration Files
### 1. Nginx Configuration
```bash
mkdir -p /opt/hermes/config/nginx/conf.d
```
```bash
cat > /opt/hermes/config/nginx/nginx.conf << 'EOF'
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 2048;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Logging
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;
# Performance
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
keepalive_requests 1000;
types_hash_max_size 2048;
client_max_body_size 100M;
# Compression
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_proxied any;
gzip_comp_level 6;
gzip_types
text/plain text/css text/xml text/javascript
application/json application/javascript application/xml
application/rss+xml application/atom+xml;
# Rate Limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=60r/s;
limit_req_zone $binary_remote_addr zone=auth:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=ws:10m rate=100r/s;
# Upstream Servers
upstream hermes_gateway {
server hermes-gateway:8000;
keepalive 64;
}
upstream hermes_web {
server hermes-web:3000;
keepalive 32;
}
upstream ninerouter {
server ninerouter:4000;
keepalive 16;
}
# SSL Configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Security Headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "strict-origin-when-cross-origin";
add_header Permissions-Policy "geolocation=(), microphone=()";
# HTTP -> HTTPS Redirect
server {
listen 80;
server_name _;
return 301 https://$host$request_uri;
}
# Main HTTPS Server
server {
listen 443 ssl http2;
server_name _;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
# Health Check (No Rate Limit)
location /health {
access_log off;
proxy_pass http://hermes_gateway/health;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
# API Routes (Rate Limited)
location /api/ {
limit_req zone=api burst=20 nodelay;
proxy_pass http://hermes_gateway;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_buffering off;
}
# WebSocket Support
location /ws {
limit_req zone=ws burst=50 nodelay;
proxy_pass http://hermes_gateway;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
}
# Authentication Endpoints (Stricter Rate Limit)
location ~ ^/api/(auth|login|register|token) {
limit_req zone=auth burst=5 nodelay;
proxy_pass http://hermes_gateway;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Web UI
location / {
proxy_pass http://hermes_web;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
}
# 9Router Direct Access (Optional)
location /ninerouter/ {
limit_req zone=api burst=30 nodelay;
proxy_pass http://ninerouter/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Metrics (Internal Only)
location /metrics {
allow 127.0.0.1;
allow 10.0.0.0/8;
allow 172.16.0.0/12;
allow 192.168.0.0/16;
deny all;
proxy_pass http://hermes_gateway/metrics;
}
# Prometheus
location /prometheus/ {
allow 127.0.0.1;
allow 10.0.0.0/8;
allow 172.16.0.0/12;
allow 192.168.0.0/16;
deny all;
proxy_pass http://prometheus:9090/;
proxy_set_header Host $host;
}
# Grafana
location /grafana/ {
proxy_pass http://grafana:3000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Jaeger
location /jaeger/ {
proxy_pass http://jaeger:16686/;
proxy_set_header Host $host;
}
}
}
EOF
```
### 2. Gateway Configuration
```bash
mkdir -p /opt/hermes/config/gateway
```
```bash
cat > /opt/hermes/config/gateway/config.yaml << 'EOF'
# Hermes Gateway Configuration
server:
host: "0.0.0.0"
port: 8000
workers: 4
timeout: 300
max_request_size: 104857600 # 100MB
# Database
database:
url: "${DATABASE_URL}"
pool_size: 20
max_overflow: 10
pool_timeout: 30
pool_recycle: 3600
# Redis
redis:
url: "${REDIS_URL}"
max_connections: 50
socket_timeout: 5
socket_connect_timeout: 5
# Authentication
auth:
secret_key: "${SECRET_KEY}"
algorithm: "HS256"
access_token_expire_minutes: 1440 # 24 hours
refresh_token_expire_days: 7
password_min_length: 12
bcrypt_rounds: 12
# Rate Limiting
rate_limits:
default:
requests_per_minute: 60
requests_per_hour: 1000
auth:
requests_per_minute: 10
requests_per_hour: 50
api:
requests_per_minute: 60
requests_per_hour: 1000
websocket:
connections_per_ip: 10
messages_per_minute: 100
# Model Router (9Router)
model_router:
base_url: "${NINEROUTER_URL}"
api_key: "${NINEROUTER_API_KEY}"
timeout: 300
max_retries: 3
retry_delay: 1.0
fallback_providers: ["groq", "cerebras", "together", "fireworks"]
# Model Providers
providers:
groq:
api_key: "${GROQ_API_KEY}"
base_url: "https://api.groq.com/openai/v1"
models:
- "llama-3.1-70b-versatile"
- "llama-3.1-8b-instant"
- "mixtral-8x7b-32768"
- "gemma2-9b-it"
rate_limit: 30
priority: 1
cerebras:
api_key: "${CEREBRAS_API_KEY}"
base_url: "https://api.cerebras.ai/v1"
models:
- "llama-3.1-70b"
- "llama-3.1-8b"
rate_limit: 60
priority: 2
together:
api_key: "${TOGETHER_API_KEY}"
base_url: "https://api.together.xyz/v1"
models:
- "meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo"
- "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo"
- "mistralai/Mixtral-8x7B-Instruct-v0.1"
rate_limit: 20
priority: 3
fireworks:
api_key: "${FIREWORKS_API_KEY}"
base_url: "https://api.fireworks.ai/inference/v1"
models:
- "accounts/fireworks/models/llama-v3p1-70b-instruct"
- "accounts/fireworks/models/llama-v3p1-8b-instruct"
rate_limit: 30
priority: 4
deepinfra:
api_key: "${DEEPINFRA_API_KEY}"
base_url: "https://api.deepinfra.com/v1/openai"
models:
- "meta-llama/Meta-Llama-3.1-405B-Instruct"
- "meta-llama/Meta-Llama-3.1-70B-Instruct"
rate_limit: 100
priority: 5
novita:
api_key: "${NOVITA_API_KEY}"
base_url: "https://api.novita.ai/v3/openai"
models:
- "meta-llama/llama-3.1-405b-instruct"
- "meta-llama/llama-3.1-70b-instruct"
rate_limit: 50
priority: 6
openrouter:
api_key: "${OPENROUTER_API_KEY}"
base_url: "https://openrouter.ai/api/v1"
models:
- "meta-llama/llama-3.1-405b-instruct"
- "meta-llama/llama-3.1-70b-instruct"
rate_limit: 100
priority: 7
ollama:
base_url: "http://ollama:11434/v1"
models:
- "llama3.1:70b"
- "llama3.1:8b"
- "qwen2.5:72b"
- "mixtral:8x22b"
rate_limit: 100
priority: 8
vllm:
base_url: "http://vllm:8000/v1"
models:
- "meta-llama/Meta-Llama-3.1-70B-Instruct"
rate_limit: 10
priority: 10
# Agent Configuration
agents:
default_model: "free"
max_concurrent_agents: 10
default_temperature: 0.7
max_tokens: 8192
default_system_prompt: |
You are a helpful AI assistant running on a self-hosted Hermes instance.
You have access to various tools and can use them to help the user.
# Tools
tools:
enabled:
- "file_read"
- "file_write"
- "file_list"
- "bash"
- "python"
- "web_search"
- "web_fetch"
- "git"
- "docker"
- "database"
file_operations:
allowed_paths:
- "/app/data"
- "/app/models"
- "/tmp"
max_file_size: 104857600 # 100MB
bash:
allowed_commands: ["ls", "cat", "grep", "find", "head", "tail", "wc", "awk", "sed", "jq", "python3", "pip", "npm", "cargo", "go", "rustc"]
timeout: 300
working_dir: "/app/data"
# WebSocket
websocket:
ping_interval: 30
ping_timeout: 10
max_message_size: 10485760 # 10MB
# Logging
logging:
level: "info"
format: "json"
output: "stdout"
file_path: "/app/logs/gateway.log"
max_file_size: "100MB"
max_files: 10
# Metrics
metrics:
enabled: true
port: 9090
path: "/metrics"
# Tracing
tracing:
enabled: true
jaeger_endpoint: "http://jaeger:14268/api/traces"
service_name: "hermes-gateway"
sample_rate: 0.1
EOF
```
---
## 🔐 SSL Certificates
```bash
# Generate self-signed cert for development (use Let's Encrypt for production)
mkdir -p /opt/hermes/ssl
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /opt/hermes/ssl/privkey.pem \
-out /opt/hermes/ssl/fullchain.pem \
-subj "/C=US/ST=State/L=City/O=Hermes/OU=AI/CN=localhost"
# For production with Let's Encrypt:
# sudo certbot certonly --standalone -d ai.yourdomain.com
# cp /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem /opt/hermes/ssl/
# cp /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem /opt/hermes/ssl/
```
---
## 🚀 Deployment
### 1. Start All Services
```bash
cd /opt/hermes
# Pull all images
docker compose pull
# Start services
docker compose up -d
# View logs
docker compose logs -f --tail 100
```
### 2. Verify Deployment
```bash
# Check all services
docker compose ps
# Check health endpoints
curl -s https://localhost/health | jq .
curl -s https://localhost/api/health | jq .
curl -s http://localhost:4000/health | jq .
curl -s http://localhost:11434/api/tags | jq .
curl -s http://localhost:8001/health | jq .
# Check logs
docker compose logs hermes-gateway --tail 50
docker compose logs ninerouter --tail 20
docker compose logs ollama --tail 20
```
---
## 🤖 Creating Your First Agent
### Via Web UI
1. Open https://your-domain.com (or http://localhost:3000)
2. Register an account
3. Navigate to **Agents** → **Create Agent**
4. Configure your agent:
```json
{
"name": "Code Assistant",
"description": "Expert coding assistant with file operations",
"model": "free",
"temperature": 0.3,
"max_tokens": 8192,
"system_prompt": "You are an expert software engineer. You have access to file operations, bash, and web search. Write clean, tested, well-documented code.",
"tools": ["file_read", "file_write", "file_list", "bash", "web_search", "web_fetch", "git"],
"temperature": 0.3,
"max_tokens": 8192
}
```
### Via API
```bash
# Create agent via API
curl -X POST https://your-domain.com/api/v1/agents \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR_JWT_TOKEN>" \
-d '{
"name": "Research Agent",
"description": "Deep research agent with web browsing",
"model": "free",
"system_prompt": "You are a thorough researcher. Use web search and fetch to gather comprehensive information. Cite sources.",
"tools": ["web_search", "web_fetch", "file_read", "file_write"],
"temperature": 0.5,
"max_tokens": 8192
}'
```
---
## 🔌 Integration Examples
### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
base_url="https://your-domain.com/v1",
api_key="not-needed" # Hermes doesn't require auth for local
)
# Chat completion
response = client.chat.completions.create(
model="free", # Uses 9Router's free model selection
messages=[
{"role": "system", "content": "You are a security expert."},
{"role": "user", "content": "Explain CVE-2024-3094 in 3 sentences."}
],
max_tokens=200,
temperature=0.3
)
print(response.choices[0].message.content)
# Streaming
stream = client.chat.completions.create(
model="free",
messages=[{"role": "user", "content": "Write a Python port scanner"}],
stream=True,
max_tokens=500
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
```
### JavaScript/TypeScript
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://your-domain.com/v1",
apiKey: "not-needed",
});
// With specific model
const response = await client.chat.completions.create({
model: "groq/llama-3.1-70b-versatile",
messages: [
{ role: "system", content: "You are a Linux expert." },
{ role: "user", content: "How to debug a segfault in C?" }
],
max_tokens: 300,
temperature: 0.2
});
console.log(response.choices[0].message.content);
```
### cURL
```bash
# Chat completion
curl -X POST https://your-domain.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "free",
"messages": [
{"role": "system", "content": "You are a Linux expert."},
{"role": "user", "content": "How to debug a segfault in C?"}
],
"max_tokens": 300,
"temperature": 0.2
}'
# Streaming
curl -X POST https://your-domain.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "free",
"messages": [{"role": "user", "content": "Write a haiku about Docker"}],
"stream": true,
"max_tokens": 100
}' --no-buffer
```
---
## 📊 Monitoring & Maintenance
### Health Checks
```bash
#!/bin/bash
# /opt/hermes/healthcheck.sh
check_service() {
local name=$1
local url=$2
if curl -sf "$url" > /dev/null; then
echo "✅ $name: OK"
else
echo "❌ $name: FAILED"
return 1
fi
}
check_service "Hermes Gateway" "https://localhost/health"
check_service "9Router" "http://localhost:4000/health"
check_service "Ollama" "http://localhost:11434/api/tags"
check_service "vLLM" "http://localhost:8001/health"
check_service "PostgreSQL" "pg_isready -h localhost -U hermes"
check_service "Redis" "redis-cli ping"
check_service "Nginx" "https://localhost/health"
```
### Backup Script
```bash
#!/bin/bash
# /opt/hermes/backup.sh
BACKUP_DIR="/opt/backups/hermes"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p "$BACKUP_DIR"
# Backup database
docker compose exec -T postgres pg_dump -U hermes hermes | gzip > "$BACKUP_DIR/hermes_db_$DATE.sql.gz"
# Backup configs
tar -czf "$BACKUP_DIR/config_$DATE.tar.gz" /opt/hermes/config /opt/hermes/.env
# Backup models (optional, large)
# tar -czf "$BACKUP_DIR/models_$DATE.tar.gz" /opt/hermes/models
# Cleanup old backups (keep 7 days)
find "$BACKUP_DIR" -type f -mtime +7 -delete
echo "Backup completed: $BACKUP_DIR"
```
### Update Script
```bash
#!/bin/bash
# /opt/hermes/update.sh
cd /opt/hermes
# Backup first
./backup.sh
# Pull latest images
docker compose pull
# Restart with new images
docker compose up -d
# Verify
sleep 10
./healthcheck.sh
# Clean up old images
docker image prune -f
```
---
## 🔧 Troubleshooting
### Common Issues
| Issue | Cause | Solution |
|-------|-------|----------|
| `GPU not detected` | NVIDIA toolkit not installed | Install nvidia-container-toolkit, restart docker |
| `CUDA out of memory` | Model too large for VRAM | Use smaller model, enable `--gpu-memory-utilization 0.8` |
| `502 Bad Gateway` | Gateway not ready | Check `docker compose logs hermes-gateway` |
| `SSL certificate error` | Self-signed cert | Use Let's Encrypt or add cert to trust store |
| `Rate limited` | Provider limits hit | Add more providers, check rate limits |
| `Model not found` | Wrong model name | Check `/v1/models` for exact names |
| `WebSocket disconnect` | Nginx timeout | Increase `proxy_read_timeout` |
### Debug Commands
```bash
# View all logs
docker compose logs -f --tail 100
# Specific service logs
docker compose logs hermes-gateway -f --tail 50
docker compose logs ninerouter -f --tail 20
docker compose logs ollama -f --tail 20
# Check resource usage
docker stats --no-stream
# Check GPU usage
watch -n 1 nvidia-smi
# Enter container for debugging
docker compose exec hermes-gateway bash
docker compose exec ninerouter sh
# Test model directly
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"free","messages":[{"role":"user","content":"Hi"}],"max_tokens":10}'
# Check Ollama models
docker compose exec ollama ollama list
# Pull new model
docker compose exec ollama ollama pull llama3.1:70b
```
---
## 🎯 Next Steps
1. **Add Your API Keys** - Edit `.env` with real API keys
2. **Configure Domain** - Point DNS to your VM, update Nginx `server_name`
3. **Enable SSL** - Run `certbot --nginx -d your-domain.com`
4. **Add More Models** - Pull models in Ollama: `ollama pull qwen2.5:72b`
5. **Set Up Monitoring** - Import Grafana dashboards, configure alerts
5. **Configure Backups** - Set up daily cron job for backups
6. **Add Custom Agents** - Create specialized agents for your workflows
7. **Enable MCP** - Configure MCP servers for extended capabilities
---
## 📚 Resources
- **Hermes Documentation:** https://github.com/hermes-ai/hermes
- **9Router GitHub:** https://github.com/9router/9router
- **Ollama Models:** https://ollama.com/library
- **vLLM Documentation:** https://docs.vllm.ai
- **Nginx Config:** https://nginx.org/en/docs/
- **Docker Compose:** https://docs.docker.com/compose/
---
## 🎉 Summary
You now have a fully functional, production-ready Hermes AI agent platform running on your VM with:
- ✅ **Hermes Gateway** - REST + WebSocket API on port 8000
- ✅ **Web UI** - Beautiful dashboard on port 3000
- ✅ **9Router** - Free model router with 7+ providers on port 4000
- ✅ **Ollama** - Local model inference on port 11434
- ✅ **vLLM** - High-throughput inference on port 8001
- ✅ **Nginx** - Reverse proxy with SSL, rate limiting, auth
- ✅ **PostgreSQL + Redis** - Persistent storage & caching
- ✅ **Monitoring** - Prometheus + Grafana + Jaeger
- ✅ **GPU Support** - NVIDIA CUDA / AMD ROCm ready
- ✅ **Production Ready** - systemd, SSL, backups, monitoring
Your Hermes instance is ready at **https://your-domain.com** 🚀
---
*Last updated: July 16, 2026 | Hermes version: latest | Guide version: 1.0*