Installation & Runtime Setup

What to install, which environment variables to set, and how to start every process.

Hard prerequisites

• Python 3.10+ (tokenizer + dataclasses + asyncio.to_thread).
• A running Docker daemon. The sandbox uses the Docker SDK; on macOS it auto-detects OrbStack / Docker Desktop sockets (sandbox/docker.py:13-54).
• An OpenAI-compatible LLM endpoint reachable from the agent (llama.cpp, vLLM, Ollama). Default expectation: http://127.0.0.1:8080.
• Required: a Tor SOCKS proxy for outbound traffic (TOR_PROXY=socks5h://127.0.0.1:9050). Ghost Agent fails closed if it is unreachable — see Anonymity & Tor routing.
• Optional: a Postgres database for the postgres_admin tool; GPU servers for image generation / vision.

Python dependencies

Pinned in requirements.txt at the repo root. Headline packages:

Environment variables

The CLI reads arguments first, then falls back to environment variables. See CLI Reference for argument-by-argument detail.

Filesystem layout at runtime

$GHOST_HOME/
├── sandbox/                      # bind-mounted into container as /workspace
│   ├── acquired_skills/          # generated tool .py files
│   │   ├── <name>.py             # (name strictly validated — see tools/acquired_skills)
│   │   ├── retired/              # auto-archived degraded skills
│   │   └── skills_registry.json
│   └── ... user/agent files
├── trajectories/                 # Stage-1 distill log (day-partitioned JSONL)
│   └── YYYY-MM-DD/
│       └── session-<sid>.jsonl   # redacted turn-level traces
└── system/
    ├── ghost-agent.log
    ├── eval/
    │   └── baseline.json         # frozen SuiteResult (scripts/eval_baseline.py)
    ├── optim/
    │   └── <signature>.json      # tuned GEPA instructions (scripts/run_gepa.py)
    └── memory/
        ├── chroma.sqlite3        # ChromaDB persistent store
        ├── knowledge_graph.db    # Graph triplets (SQLite)
        ├── episodic_memory.db    # Episodes (SQLite)
        ├── projects.db           # Project / task / artifact / event log
        ├── memory_journal.json   # Short-term work queue
        ├── user_profile.json     # Profile facts
        ├── skills_playbook.json  # Lessons (verified + utility-ranked)
        ├── adaptive_threshold.json
        ├── contradiction_log.json
        ├── self_play_frontier.json + .lock
        ├── library_index.json    # Names of ingested documents
        └── scratchpad.db         # LRU + TTL working memory

Boot sequence

From main.py:399-440 and the lifespan async context (main.py:140-398):

1. Parse CLI flags + env vars.
2. Make sure GHOST_HOME/sandbox and GHOST_HOME/system/memory exist.
3. setup_logging() wires colourised structured logs (utils/logging.py).
4. load_tokenizer() grabs / downloads the Qwen3 tokenizer for token budgeting.
5. Build GhostContext with Scratchpad, MemoryJournal, SkillMemory, FrontierTracker.
6. Construct the FastAPI app and attach the lifespan coroutine.
7. Lifespan startup:
   • LLMClient with foreground + swarm/worker/visual/coding/image_gen pools.
   • DockerSandbox.ensure_running() in a worker thread.
   • ProjectStore always (for user-driven multi-session projects).
   • Memory stores (vector, graph, profile, episodes, contradiction log, adaptive threshold) — skipped under --no-memory.
   • APScheduler for user-defined cron tasks.
   • MemoryBus, Verifier, UncertaintyTracker.
   • Optional MCTSReasoner + HypothesisTester when --deep-reason.
   • GhostAgent + biological watchdog daemon task.
8. Lifespan shutdown: cancel watchdog, stop scheduler, close LLM client, stop sandbox container (kept around with remove=False for fast resume).
9. uvicorn.run(app, host, port, log_config=None).

Process commands

# core agent
python -m src.ghost_agent.main \
    --upstream-url "http://127.0.0.1:8080" \
    --host 0.0.0.0 --port 8000 --verbose

# web UI proxy (separate port)
python interface/server.py --agent-log /var/log/ghost-agent.log

# Slack bot (Socket Mode)
SLACK_BOT_TOKEN=xoxb-... SLACK_APP_TOKEN=xapp-... \
  python interface/externals/slack_bot/main.py \
    --log-file /var/log/ghost-slack-bot.err

# Voice server (Jetson / RPi GPU box)
python interface/externals/tts_stt/voice_server.py

# Image gen server (GPU box)
python interface/externals/image_generation/img_gen_server.py

Pre-provisioned sandbox image (optional, recommended)

The runtime sandbox wrapper can provision Chromium + apt deps + the full pip stack on first boot, but this takes 2–3 minutes the first time and can silently rot if committed before --with-deps was added. The repeatable path is to build a proper image:

scripts/build_sandbox_image.sh
# → builds ghost-agent-base:latest (~2 GB first run; ~5 min on warm cache)
# → runs a Chromium launch smoke test at the end

The image bakes apt system packages, the deep-learning pip stack, and playwright install --with-deps chromium at build time — self-play + browser tasks start instantly. The runtime gate checks both the /root/.supercharged.v2 marker AND that the Chromium binary is actually on disk; if the marker is present but the binary is missing (the exact silent-failure mode the v2 bump exists to catch), a full reinstall fires on next ensure_running.

If you skip this step, the fallback inline provisioning still produces a correct image; it just takes longer on the first turn.

Test runner

pytest                                   # full suite (asyncio_mode=auto)
pytest tests/test_agent_planning.py      # single file
pytest -k "memory and not slack"         # filter
black src interface tests
pylint src/ghost_agent