π οΈ
AI / AI Agents β Ops Runbook
Troubleshooting The Crew
Verbose runbook for restarting and recovering the full AI agent crew β Hermes, OpenClaw, Pi, OpenCode, Codex, Claude Code, 9Router, and every Telegram gateway. Exact systemd commands, health checks, and per-session CLI recovery.
# π οΈ Troubleshooting The Crew: Rebooting Every AI Agent & Telegram Gateway
**Date:** July 16, 2026
**Tags:** Troubleshooting, AI Agents, systemd, Telegram, Self-Hosted, Ops
---
## π― Overview
This is the **operator runbook** for bringing the agent crew back online. It answers one question fast: **what command reboots each agent, and what do I do if a Telegram gateway is down?**
The crew runs on a single Linux VM (user `ryan`, user-level systemd). Some agents are **long-lived systemd services** (restart with one command). Others are **per-session CLI agents** (Pi, OpenCode, Codex, Claude Code) that only live while a terminal/session is open β for those, \u201creboot\u201d means restarting their **Telegram bot service** (if one exists) or re-launching the CLI.
> β οΈ **Golden rule:** Never restart `hermes-gateway.service` *from inside* the Hermes gateway process (e.g. while chatting with Hermes). It kills its own session and the restart never completes. Always run restart commands from an **external shell** (a separate terminal, SSH, or another agent like this one).
---
## πΊοΈ The Crew at a Glance
| Agent | Form | Restart unit / method |
|-------|------|----------------------|
| π€ **Hermes** | systemd (gateway + dashboard) | `hermes-gateway.service`, `hermes-dashboard.service` |
| π¦
**OpenClaw** | systemd (gateway + HTTPS proxy) | `openclaw-gateway.service`, `openclaw-https-proxy.service` |
| π **9Router** | systemd | `9router.service` |
| π **Codex** | CLI + Telegram bot service | `codex-telegram-bot.service` |
| π **Claude Code** | CLI + Telegram bot service | `claude-code-telegram.service` |
| π₯§ **Pi** | per-session CLI | re-launch CLI / restart Telegram bot if present |
| π§ **OpenCode** | per-session CLI | re-launch CLI / restart Telegram bot if present |
| π§ **GBrain** | Bun CLI + HTTP MCP (`:3131`) | restart MCP server process |
---
## π€ Hermes
### Restart both (gateway + dashboard)
```bash
# From an EXTERNAL shell (not inside Hermes chat):
~/.hermes/scripts/restart-hermes-services.sh
```
### Restart only the Gateway
```bash
systemctl --user restart hermes-gateway.service
```
### Restart only the Dashboard
```bash
systemctl --user restart hermes-dashboard.service
```
### Health check
```bash
# Gateway is a web service β confirm it answers:
curl -s --max-time 5 http://127.0.0.1:/health || echo "HERMES GATEWAY DOWN"
# Or check unit state:
systemctl --user is-active hermes-gateway.service
```
---
## π¦
OpenClaw
### Restart the Gateway
```bash
systemctl --user restart openclaw-gateway.service
```
### Restart the HTTPS reverse proxy (nginx)
```bash
systemctl --user restart openclaw-https-proxy.service
```
### Health check
```bash
curl -s --max-time 5 http://127.0.0.1:18790/health || echo "OPENCLAW GATEWAY DOWN"
systemctl --user is-active openclaw-gateway.service
```
> π‘ If the gateway won\u2019t start due to a **SQLite plugin-index conflict**, run `openclaw doctor --fix` then restart. (See the [OpenClaw recovery post](/blog/openclaw-gateway-sqlite-plugin-conflict/).)
---
## π 9Router (model router)
9Router is the free model proxy every agent routes through. If models start 401-ing or timing out, bounce it.
```bash
systemctl --user restart 9router.service
# verify it answers with a model list:
curl -s --max-time 5 http://192.168.51.115:20128/v1/models | head
```
> π The 9Router API key lives in `/home/ryan/.hermes/scripts/9router.txt` and is injected via `~/.hermes/ninerouter.env` (`EnvironmentFile=`) into the gateway β it is **not** hardcoded.
---
## π Codex (CLI + Telegram bot)
Codex itself is launched per-session (`codex app-server`, `codex exec`, etc.). The always-on piece is its **Telegram bot**.
### Restart the Telegram bot
```bash
systemctl --user restart codex-telegram-bot.service
# working dir: /home/ryan/telegram-codex (venv python bot.py)
systemctl --user is-active codex-telegram-bot.service
```
### Re-launch a Codex CLI session
```bash
# from a fresh terminal:
codex exec ""
# or interactive:
codex
```
---
## π Claude Code (CLI + Telegram bot)
Claude Code is a per-session CLI. Its Telegram channel runs as a service.
### Restart the Telegram channel
```bash
systemctl --user restart claude-code-telegram.service
# also bounce its companion proxy/fallback if needed:
systemctl --user restart claude-nim-proxy.service
systemctl --user restart claude-telegram-fallback.service
```
### Re-launch Claude Code CLI (9Router-wired)
```bash
# from a fresh terminal (reads ANTHROPIC_BASE_URL from settings):
claude
# or one-shot:
claude -p "" --model Free
```
> π‘ Claude Code points at 9Router via `ANTHROPIC_BASE_URL=http://192.168.51.115:20128/v1` in `~/.claude/settings.json` (env block). (See the [Claude Code + 9Router guide](/blog/claude-code-9router-setup/).)
---
## π₯§ Pi & π§ OpenCode (per-session CLIs)
These are **not** systemd services β they only run while a session is alive. There is no `pi.service` or `opencode.service`.
### Pi
```bash
# re-launch from a fresh terminal:
pi
# or with 9Router explicitly:
pi --provider 9router --model Free
```
If a Pi **Telegram bot** service exists on your setup:
```bash
systemctl --user restart pi-telegram-bot.service # only if present
```
### OpenCode
```bash
# interactive CLI:
opencode
# headless server mode:
opencode serve --port --hostname 127.0.0.1
```
To recover from the known **SQLite session-corruption** (`NOT NULL constraint failed: session_message.seq`), the permanent fix is a DB trigger + a 5-minute cron watchdog (see the [OpenCode setup guide](/blog/opencode-agent-setup/)). Restarting the CLI alone does not fix the root cause.
---
## π§ GBrain (shared memory / MCP)
GBrain is a Bun CLI plus an **HTTP MCP server** on port `3131`. If agents report memory/tools missing:
```bash
# Health check:
/home/ryan/.bun/bin/gbrain get_health
# If unhealthy, restart the MCP server (process started via start-gbrain-mcp.sh):
# kill the running gbrain serve process, then:
nohup /home/ryan/.bun/bin/gbrain serve --http --port 3131 --bind 0.0.0.0 \
> /home/ryan/gbrain-mcp.log 2>&1 &
```
---
## π The One-Command Cheat Sheet
```bash
# Hermes (both):
~/.hermes/scripts/restart-hermes-services.sh
# Hermes Gateway only:
systemctl --user restart hermes-gateway.service
# Hermes Dashboard only:
systemctl --user restart hermes-dashboard.service
# OpenClaw Gateway:
systemctl --user restart openclaw-gateway.service
# OpenClaw HTTPS proxy:
systemctl --user restart openclaw-https-proxy.service
# 9Router:
systemctl --user restart 9router.service
# Codex Telegram bot:
systemctl --user restart codex-telegram-bot.service
# Claude Code Telegram:
systemctl --user restart claude-code-telegram.service
# Claude companions:
systemctl --user restart claude-nim-proxy.service
# Pi / OpenCode / Claude Code CLIs: re-launch in a fresh terminal
# GBrain MCP: restart gbrain serve --http --port 3131
```
---
## π§ͺ Diagnostic Loop (when something is just *weird*)
1. **Is the service up?** `systemctl --user is-active `
2. **What\u2019s the last error?** `journalctl --user -u -n 50 --no-pager`
3. **Is 9Router answering?** `curl -s http://192.168.51.115:20128/v1/models | head`
4. **Is GBrain healthy?** `gbrain get_health`
5. **Telegram not responding?** Restart the agent\u2019s Telegram bot unit (table above).
6. **Still down?** Restart the agent unit, wait 10s, re-check health.
---
## π©Ί Common Symptoms β Fix
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| Hermes chat dead | gateway crashed | `systemctl --user restart hermes-gateway.service` (external shell) |
| OpenClaw 18790 no response | gateway down / plugin conflict | `openclaw doctor --fix` then restart `openclaw-gateway.service` |
| Models 401 across all agents | 9Router key expired | re-export key to `9router.txt` + `systemctl --user restart 9router.service` |
| Codex/Claude Telegram silent | bot service exited | `systemctl --user restart codex-telegram-bot.service` / `claude-code-telegram.service` |
| Agent says \u201cno memory/tools\u201d | GBrain MCP down | restart `gbrain serve --http --port 3131` |
| OpenCode session crash on create | SQLite `seq` NOT NULL | trigger + cron watchdog (root fix), then relaunch CLI |
---
## π Security Notes
- All restart commands are **user-level** (`--user`); no root needed.
- 9Router key is injected via `EnvironmentFile=`, never committed.
- Restart from an **external shell** for Hermes gateway (never self-restart).
- Telegram bot tokens live in `~/.hermes/.env` / `hermes_telegram.txt` as `` placeholders here.
---
## β
Summary
Most of the crew is one `systemctl --user restart ` away. The per-session CLIs (Pi, OpenCode, Codex, Claude Code) are relaunched in a fresh terminal, and their Telegram gateways have dedicated units. When in doubt: check `is-active`, read `journalctl`, ping 9Router, check GBrain, then bounce the unit. π
---
*No API keys, tokens, or secrets are included β every value is a `` you supply locally. Built with Astro, GitHub, and Cloudflare Pages.*