Scaffold SimpleX Orchestrate: supervisor over official binaries
A standalone control-plane app that spawns and drives the official SimpleX binaries (never modifies simplex source). Validated against simplex-chat built from source (stable v6.5.4, GHC 9.6.3). - CLAUDE.md: architecture notes mined from the upstream docs (WebSocket bot API, per-profile DBs, directory/broadcast bot config) - supervisor/: process registry + port allocation (supervisor.py), corrId/cmd<->resp WebSocket client (ws_client.py), binary locator (binaries.py), FastAPI front with REST control + /events stream (server.py) - smoke_test.py: Pattern-1 handshake (spawn simplex-chat -p, create+read user) — PASS - group_test.py: two accounts, invitation connect + group invite/join, verified membership over the real SMP network — PASS - build_chat.sh / install_ghc.sh: reproducible toolchain + from-source build Key finding: fresh DB prompts for a display name on stdin; spawn with --create-bot-display-name to start the WebSocket server non-interactively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
11
.gitignore
vendored
Normal file
11
.gitignore
vendored
Normal file
@@ -0,0 +1,11 @@
|
||||
# Upstream simplex-chat checkout (cloned separately; never modified/committed here)
|
||||
simplex-chat/
|
||||
# Built / downloaded binaries
|
||||
bin/
|
||||
# Python env and caches
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
# Runtime per-profile databases + logs
|
||||
data/
|
||||
*.log
|
||||
124
CLAUDE.md
Normal file
124
CLAUDE.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# SimpleX Orchestrate — project notes for Claude
|
||||
|
||||
Goal: a web-managed **orchestrator** that runs the **official SimpleX binaries** (instead of
|
||||
re-implementing bot logic against the FFI `libsimplex` SDK, as the older `simplex-manager`
|
||||
prototype did). The manager spawns/supervises real SimpleX processes, drives the interactive
|
||||
ones over WebSocket, and reads the autonomous ones' output.
|
||||
|
||||
Reference checkout: `./simplex-chat/` (cloned from https://github.com/simplex-chat/simplex-chat).
|
||||
All facts below are taken from that repo's own docs — cited inline. There is **no** CLAUDE.md /
|
||||
AGENTS.md in the upstream repo; the authoritative interface docs are:
|
||||
- `simplex-chat/bots/README.md` — bot architecture & WebSocket API
|
||||
- `simplex-chat/bots/api/COMMANDS.md`, `EVENTS.md`, `TYPES.md` — full command/event/type reference
|
||||
- `simplex-chat/docs/CLI.md` — terminal client / DB / server flags
|
||||
- `simplex-chat/apps/*/README.md` + `src/.../Options.hs` — the official bot binaries
|
||||
- Official TypeScript SDK: `simplex-chat/packages/simplex-chat-client/typescript/`
|
||||
|
||||
---
|
||||
|
||||
## Two integration patterns (this is the core design decision)
|
||||
|
||||
**1. Interactive accounts → drive the CLI over WebSocket** (`bots/README.md`)
|
||||
- Run the terminal client as a local WebSocket server: `simplex-chat -p 5225`
|
||||
- Our process connects over WebSocket and sends JSON commands / receives events.
|
||||
- This is how custom bots (TS/Python/Rust) and our manager talk to a profile.
|
||||
- Same command strings as the app's *Settings → Developer tools → Chat Console*.
|
||||
|
||||
**2. Autonomous official bots → run the prebuilt Haskell binary directly**
|
||||
- `simplex-directory-service` and `simplex-broadcast-bot` are standalone executables built on
|
||||
the chat core. They run their **own** logic, with their **own** DB + config, no WebSocket.
|
||||
- The manager's job for these is lifecycle (spawn/stop/monitor/logs) + reading their output.
|
||||
- **Use these instead of reimplementing directory/broadcast in Python** — they're battle-tested
|
||||
and include captcha, moderation, approval, periodic re-checks, website generation, etc.
|
||||
|
||||
---
|
||||
|
||||
## WebSocket protocol (pattern 1) — `bots/README.md`
|
||||
|
||||
- `simplex-chat -p <port>` starts the WS server. **localhost only, NO authentication** — keep the
|
||||
bot process on the same machine and firewall the port. Messages are unencrypted.
|
||||
- Command: `{"corrId":"<unique>", "cmd":"<command string>"}`
|
||||
- Response: `{"corrId":"<same>", "resp":{"type":"<tag>", ...}}` (matched by corrId)
|
||||
- Event: `{"resp":{"type":"<tag>", ...}}` (no corrId — connections, received messages, etc.)
|
||||
- `NewChatItems` arrives both as a command response (after `APISendMessages`) and as an event.
|
||||
- Parser must **ignore unknown event types / union tags / extra fields** (forward-compat rule).
|
||||
- Minimal bot loop: handle `NewChatItems` event → reply with `APISendMessages`.
|
||||
- Network usage per command is documented: `no` / `interactive` / `background`.
|
||||
|
||||
## Per-profile databases — `docs/CLI.md`
|
||||
|
||||
- `simplex-chat -d <prefix>` → creates `<prefix>_v1_chat.db` and `<prefix>_v1_agent.db`.
|
||||
- Default data dir `~/.simplex` (`%APPDATA%/simplex` on Windows).
|
||||
- **One process = one DB prefix.** A single DB *can* hold multiple bot profiles via
|
||||
`/create bot [files=on] <name> [<bio>]`, but the simple model is one process per profile.
|
||||
- SMP servers: `-s smp://<fp>@host`. Tor: `-x` or `--socks-proxy=:9050`. `simplex-chat -h` for all.
|
||||
|
||||
## Bot profile setup — `bots/README.md` (needs app/CLI v6.4.3+)
|
||||
|
||||
- Mark a profile as a bot: `peerType: "bot"`. Set via:
|
||||
- CLI flags `--create-bot-display-name`, `--create-bot-allow-files` on first start, or
|
||||
- `/create bot [files=on] '<name>' [<bio>]`, or
|
||||
- `APIUpdateProfile` (also sets command menus).
|
||||
- Command menus: `/set bot commands '<label>':/'<keyword>[ <params>]', '<label>':{...nested...}`
|
||||
(e.g. directory's `'Show your groups':/list`). Per-contact overrides via `APISetContactPrefs`.
|
||||
- **Business address** (support): a special group chat per connecting customer; inherits the bot's
|
||||
preferences/commands. See `docs/BUSINESS.md`.
|
||||
|
||||
---
|
||||
|
||||
## Official bot: Directory service — `apps/simplex-directory-service/` (`Directory/Options.hs`)
|
||||
|
||||
Run the `simplex-directory-service` binary; it manages registration + search and **writes the
|
||||
website files itself**. Replaces the Python directory reimplementation in the old prototype.
|
||||
|
||||
Key CLI options:
|
||||
- `--super-users CONTACT_ID:DISPLAY_NAME,...` (required) — who can approve/manage listings
|
||||
- `--admin-users CONTACT_ID:DISPLAY_NAME,...` — directory managers
|
||||
- `--owners-group GROUP_ID:DISPLAY_NAME` — owners of listed groups auto-invited
|
||||
- `--directory-file PATH` — append-only log holding directory state
|
||||
- `--web-folder PATH` — **where static web assets / group listing files are written** (this is what
|
||||
feeds the directory website; our old `web/<bot>/data/listing.json` was a hand-rolled stand-in)
|
||||
- `--service-name "SimpleX Directory"` — bot display name (default shown)
|
||||
- `--captcha-generator EXE` / `--voice-captcha-generator EXE` — anti-spam
|
||||
- `--blocked-words-file` / `--blocked-fragments-file` / `--blocked-extenstion-rules` /
|
||||
`--name-spelling-file` / `--profile-name-limit` — moderation
|
||||
- `--link-check-interval SECONDS` (default 1800) — periodic re-check of public group links
|
||||
- `--run-cli` — interactive CLI mode
|
||||
- `--migrate-directory-file <check|import|export|listing>` — `listing` = `saveGroupListingFiles`
|
||||
regenerates the website listing files from the log
|
||||
- plus core chat options (DB file, SMP servers, Tor) shared with the CLI
|
||||
- Registration flow = **owner adds the bot to their group**; listing stays pending until a
|
||||
super-user approves (matches what we scoped for the prototype).
|
||||
|
||||
## Official bot: Broadcast — `apps/simplex-broadcast-bot/`
|
||||
|
||||
- `--display-name`, `--publishers CONTACT_ID:DISPLAY_NAME,...`, `--welcome`, `--prohibited`
|
||||
(+ core DB/SMP options). Re-broadcasts messages from publishers to all connected contacts.
|
||||
|
||||
## Other official bot apps
|
||||
- `apps/simplex-bot` / `simplex-bot-advanced` — minimal Haskell bot examples
|
||||
- `apps/simplex-support-bot` — support/business example
|
||||
- `apps/simplex-chat` — the terminal CLI itself
|
||||
|
||||
---
|
||||
|
||||
## Getting the binaries
|
||||
- Install script: `curl -o- https://raw.githubusercontent.com/simplex-chat/simplex-chat/stable/install.sh | bash`
|
||||
→ `simplex-chat` on PATH (CLI only).
|
||||
- Prebuilt binaries: GitHub Releases (per stable version).
|
||||
- Build from source (`docs/CLI.md`): GHC 9.6.3 + cabal 3.10.1.0, or
|
||||
`DOCKER_BUILDKIT=1 docker build --output ~/.local/bin .`. Bot apps build via
|
||||
`cabal install <exe>` (e.g. `simplex-directory-service`, `simplex-broadcast-bot`).
|
||||
- **Build the stable branch**, not master.
|
||||
|
||||
## Proposed orchestrator architecture (to flesh out)
|
||||
- Manager (web UI + supervisor) keeps a registry of profiles. Each profile has a kind:
|
||||
- `cli` (interactive: user/echo/support/custom) → spawn `simplex-chat -p <port> -d data/<name>`,
|
||||
connect WS, proxy commands; one port per profile.
|
||||
- `directory` → spawn `simplex-directory-service` with `--directory-file`, `--web-folder`,
|
||||
`--super-users`, DB; supervise + serve its web folder.
|
||||
- `broadcast` → spawn `simplex-broadcast-bot` with publishers/welcome; supervise.
|
||||
- Web frontend talks only to the manager; the manager owns process lifecycle, port allocation,
|
||||
log capture, and (for directory) serving the generated web folder.
|
||||
- Security: bind all CLI WS ports to localhost; never expose unauthenticated WS publicly.
|
||||
</content>
|
||||
14
build_chat.sh
Executable file
14
build_chat.sh
Executable file
@@ -0,0 +1,14 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
export PATH="$HOME/.ghcup/bin:$PATH"
|
||||
cd /home/user/code/simplex-orchestrate/simplex-chat
|
||||
echo "=== toolchain ==="; ghc --version; cabal --version
|
||||
echo "=== cabal update ==="; cabal update
|
||||
echo "=== building exe:simplex-chat (long) ==="
|
||||
cabal build exe:simplex-chat
|
||||
BIN="$(cabal list-bin exe:simplex-chat)"
|
||||
echo "=== built: $BIN ==="
|
||||
cp "$BIN" /home/user/code/simplex-orchestrate/bin/simplex-chat
|
||||
chmod +x /home/user/code/simplex-orchestrate/bin/simplex-chat
|
||||
/home/user/code/simplex-orchestrate/bin/simplex-chat --help >/dev/null 2>&1 && echo "binary runs OK"
|
||||
echo "=== BUILD COMPLETE ==="
|
||||
125
group_test.py
Normal file
125
group_test.py
Normal file
@@ -0,0 +1,125 @@
|
||||
"""Two-account integration test over the WebSocket interface:
|
||||
|
||||
alice ──invitation──▶ bob (1-time link connect → contacts)
|
||||
alice creates group, invites bob → bob joins → verify membership both sides.
|
||||
|
||||
Exercises the real async + SMP-network path (not just local commands).
|
||||
Run (needs ./bin/simplex-chat): .venv/bin/python group_test.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from supervisor.supervisor import Supervisor # noqa: E402
|
||||
|
||||
events: dict[str, list[dict]] = {"alice": [], "bob": []}
|
||||
|
||||
|
||||
async def on_event(name: str, resp: dict) -> None:
|
||||
events.setdefault(name, []).append(resp)
|
||||
|
||||
|
||||
async def wait_event(name: str, pred, timeout: float, label: str) -> dict:
|
||||
start = time.time()
|
||||
i = 0
|
||||
while time.time() - start < timeout:
|
||||
while i < len(events[name]):
|
||||
e = events[name][i]; i += 1
|
||||
if pred(e):
|
||||
return e
|
||||
await asyncio.sleep(0.3)
|
||||
raise TimeoutError(f"[{name}] timed out waiting for {label}")
|
||||
|
||||
|
||||
async def poll(coro_factory, pred, timeout: float, label: str):
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
val = await coro_factory()
|
||||
if pred(val):
|
||||
return val
|
||||
await asyncio.sleep(1.0)
|
||||
raise TimeoutError(f"timed out polling for {label}")
|
||||
|
||||
|
||||
def invitation_link(resp: dict) -> str:
|
||||
inv = resp.get("connLinkInvitation") or {}
|
||||
return inv.get("connShortLink") or inv.get("connFullLink", "")
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
sup = Supervisor(data_dir="data", base_port=5500, on_event=on_event)
|
||||
ok = False
|
||||
try:
|
||||
print("→ starting two accounts (alice, bob) …")
|
||||
await sup.start_cli("alice")
|
||||
await sup.start_cli("bob")
|
||||
for n in ("alice", "bob"):
|
||||
r = await sup.send(n, "/_start")
|
||||
print(f" {n} /_start -> {r.get('type')}")
|
||||
|
||||
uidA = (await sup.send("alice", "/user"))["user"]["userId"]
|
||||
uidB = (await sup.send("bob", "/user"))["user"]["userId"]
|
||||
print(f" alice uid={uidA} bob uid={uidB}")
|
||||
|
||||
# 1. connect via 1-time invitation
|
||||
inv = await sup.send("alice", f"/_connect {uidA}")
|
||||
link = invitation_link(inv)
|
||||
print(f"→ alice invitation ({inv.get('type')}): {link[:55]}…")
|
||||
cb = await sup.send("bob", f"/connect {link}")
|
||||
print(f"→ bob /connect -> {cb.get('type')}")
|
||||
|
||||
print("→ waiting for contactConnected (SMP handshake) …")
|
||||
await wait_event("alice", lambda e: e.get("type") == "contactConnected", 120, "alice contactConnected")
|
||||
print(" ✓ alice: contact connected")
|
||||
await wait_event("bob", lambda e: e.get("type") == "contactConnected", 120, "bob contactConnected")
|
||||
print(" ✓ bob: contact connected")
|
||||
|
||||
# 2. alice creates a group and invites bob
|
||||
contacts = (await sup.send("alice", f"/_contacts {uidA}")).get("contacts", [])
|
||||
bob_cid = contacts[0]["contactId"]
|
||||
print(f"→ bob is contactId={bob_cid} on alice")
|
||||
gr = await sup.send("alice", f'/_group {uidA} ' + json.dumps({"displayName": "testgrp", "fullName": ""}))
|
||||
gid = gr["groupInfo"]["groupId"]
|
||||
print(f"→ group created gid={gid} ({gr.get('type')})")
|
||||
add = await sup.send("alice", f"/_add #{gid} {bob_cid} member")
|
||||
print(f"→ alice /_add bob -> {add.get('type')}")
|
||||
|
||||
# 3. bob sees the pending invite, then joins
|
||||
async def bob_groups():
|
||||
return (await sup.send("bob", f"/_groups {uidB}")).get("groups", [])
|
||||
bgroups = await poll(bob_groups, lambda gs: len(gs) > 0, 120, "bob receives group invite")
|
||||
bg = bgroups[0]
|
||||
print(f" ✓ bob sees group #{bg['groupId']} status={bg['membership']['memberStatus']}")
|
||||
j = await sup.send("bob", f"/_join #{bg['groupId']}")
|
||||
print(f"→ bob /_join -> {j.get('type')}")
|
||||
|
||||
# 4. verify membership on both sides
|
||||
print("→ waiting for member to connect in group …")
|
||||
await wait_event(
|
||||
"alice",
|
||||
lambda e: e.get("type") in ("memberConnected", "joinedGroupMember", "connectedToGroupMember"),
|
||||
120, "alice memberConnected",
|
||||
)
|
||||
print(" ✓ alice: member connected event")
|
||||
|
||||
async def alice_member_count():
|
||||
ag = (await sup.send("alice", f"/_groups {uidA}")).get("groups", [])
|
||||
return ag[0]["groupSummary"]["currentMembers"] if ag else 0
|
||||
count = await poll(alice_member_count, lambda c: c >= 2, 60, "group reaches 2 members")
|
||||
mem = (await sup.send("alice", f"/_members #{gid}"))["group"]["members"]
|
||||
print(f" alice group members: {[(m['localDisplayName'], m['memberStatus']) for m in mem]}")
|
||||
print(f" alice group currentMembers = {count}")
|
||||
ok = count >= 2
|
||||
finally:
|
||||
await sup.stop_all()
|
||||
|
||||
print("\nRESULT:", "PASS — invite/join verified end to end" if ok else "INCOMPLETE")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
11
install_ghc.sh
Executable file
11
install_ghc.sh
Executable file
@@ -0,0 +1,11 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
export BOOTSTRAP_HASKELL_NONINTERACTIVE=1
|
||||
export BOOTSTRAP_HASKELL_GHC_VERSION=9.6.3
|
||||
export BOOTSTRAP_HASKELL_CABAL_VERSION=3.10.1.0
|
||||
export BOOTSTRAP_HASKELL_INSTALL_NO_STACK=1
|
||||
export BOOTSTRAP_HASKELL_ADJUST_BASHRC=0
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh
|
||||
echo "=== DONE ==="
|
||||
"$HOME/.ghcup/bin/ghc" --version
|
||||
"$HOME/.ghcup/bin/cabal" --version
|
||||
3
requirements.txt
Normal file
3
requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
websockets
|
||||
67
smoke_test.py
Normal file
67
smoke_test.py
Normal file
@@ -0,0 +1,67 @@
|
||||
"""Pattern-1 smoke test: spawn a real `simplex-chat -p` process, drive it over the
|
||||
WebSocket corrId/cmd↔resp handshake, and confirm a structured response comes back.
|
||||
|
||||
Validates the supervisor + ws_client against a real binary. Local commands only
|
||||
(/user, /_create user, /users) — no network needed.
|
||||
|
||||
Run (with ./bin/simplex-chat present): python smoke_test.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from supervisor.supervisor import Supervisor # noqa: E402
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
events: list[tuple[str, str]] = []
|
||||
|
||||
async def on_event(name: str, resp: dict) -> None:
|
||||
events.append((name, resp.get("type", "?")))
|
||||
|
||||
sup = Supervisor(data_dir="data", base_port=5400, on_event=on_event)
|
||||
ok = False
|
||||
try:
|
||||
print("→ spawning simplex-chat -p (profile 'smoke') …")
|
||||
m = await sup.start_cli("smoke")
|
||||
print(f" process pid={m.proc.pid} port={m.port}, WebSocket connected")
|
||||
|
||||
print("→ /user (show active user)")
|
||||
r = await sup.send("smoke", "/user")
|
||||
print(" resp.type =", r.get("type"))
|
||||
|
||||
if r.get("type") != "activeUser":
|
||||
new_user = {
|
||||
"profile": {"displayName": "smoke", "fullName": ""},
|
||||
"pastTimestamp": False,
|
||||
"userChatRelay": False,
|
||||
}
|
||||
print("→ /_create user")
|
||||
rc = await sup.send("smoke", "/_create user " + json.dumps(new_user))
|
||||
print(" resp.type =", rc.get("type"))
|
||||
r = await sup.send("smoke", "/user")
|
||||
print(" /user resp.type =", r.get("type"))
|
||||
|
||||
user = r.get("user") or {}
|
||||
print(" active user displayName =", user.get("localDisplayName"))
|
||||
|
||||
rl = await sup.send("smoke", "/users")
|
||||
print("→ /users resp.type =", rl.get("type"),
|
||||
"count =", len(rl.get("users", []) if isinstance(rl, dict) else []))
|
||||
|
||||
ok = r.get("type") == "activeUser"
|
||||
finally:
|
||||
await sup.stop("smoke")
|
||||
print("→ stopped process")
|
||||
|
||||
print(" events observed:", events[:5])
|
||||
print("\nRESULT:", "PASS — corrId/cmd↔resp handshake works" if ok else "FAIL")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
9
supervisor/__init__.py
Normal file
9
supervisor/__init__.py
Normal file
@@ -0,0 +1,9 @@
|
||||
"""SimpleX Orchestrate supervisor — spawns and drives the official SimpleX binaries.
|
||||
|
||||
It never modifies or rebuilds simplex-chat; it only invokes prebuilt binaries.
|
||||
"""
|
||||
|
||||
from .supervisor import Managed, Supervisor
|
||||
from .ws_client import SimplexWSClient
|
||||
|
||||
__all__ = ["Supervisor", "Managed", "SimplexWSClient"]
|
||||
24
supervisor/binaries.py
Normal file
24
supervisor/binaries.py
Normal file
@@ -0,0 +1,24 @@
|
||||
"""Locate the official SimpleX binaries — never built/modified here, only invoked.
|
||||
|
||||
Resolution order: $SIMPLEX_BIN dir, then ./bin/, then PATH. Place prebuilt
|
||||
binaries (simplex-chat, simplex-directory-service, simplex-broadcast-bot) in ./bin/
|
||||
or install via the upstream install script. We never compile or alter them.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
BIN_DIR = Path(os.environ.get("SIMPLEX_BIN", Path(__file__).resolve().parent.parent / "bin"))
|
||||
|
||||
KNOWN = ("simplex-chat", "simplex-directory-service", "simplex-broadcast-bot")
|
||||
|
||||
|
||||
def binary_path(name: str) -> str:
|
||||
local = BIN_DIR / name
|
||||
if local.exists():
|
||||
return str(local)
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
return found
|
||||
raise FileNotFoundError(f"{name!r} not found in {BIN_DIR} or PATH")
|
||||
90
supervisor/server.py
Normal file
90
supervisor/server.py
Normal file
@@ -0,0 +1,90 @@
|
||||
"""Front-facing API the website talks to.
|
||||
|
||||
The browser:
|
||||
- opens a WebSocket to /events to receive live chat events from every profile
|
||||
(each event is tagged with the originating profile name);
|
||||
- uses REST to spawn/stop profiles and to send commands to cli profiles.
|
||||
|
||||
This process sits between the website and the SimpleX binaries; it is the only
|
||||
thing that touches the binaries. Run: uvicorn supervisor.server:app
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
|
||||
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
|
||||
|
||||
from .supervisor import Supervisor
|
||||
|
||||
app = FastAPI(title="SimpleX Orchestrate")
|
||||
|
||||
_browser_clients: set[WebSocket] = set()
|
||||
|
||||
|
||||
async def _broadcast(name: str, event: dict) -> None:
|
||||
dead = []
|
||||
for ws in _browser_clients:
|
||||
try:
|
||||
await ws.send_json({"profile": name, "event": event})
|
||||
except Exception:
|
||||
dead.append(ws)
|
||||
for ws in dead:
|
||||
_browser_clients.discard(ws)
|
||||
|
||||
|
||||
sup = Supervisor(on_event=_broadcast)
|
||||
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def _shutdown() -> None:
|
||||
await sup.stop_all()
|
||||
|
||||
|
||||
# ── Control (REST) ──────────────────────────────────────────────────────────────
|
||||
@app.get("/profiles")
|
||||
async def profiles() -> dict:
|
||||
return {"profiles": sup.list()}
|
||||
|
||||
|
||||
@app.post("/profiles/{name}/start-cli")
|
||||
async def start_cli(name: str) -> dict:
|
||||
await sup.start_cli(name)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@app.post("/profiles/{name}/start-directory")
|
||||
async def start_directory(name: str, body: dict) -> dict:
|
||||
await sup.start_directory(name, body["super_users"], body["web_folder"])
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@app.post("/profiles/{name}/start-broadcast")
|
||||
async def start_broadcast(name: str, body: dict) -> dict:
|
||||
await sup.start_broadcast(name, body["display_name"], body["publishers"])
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@app.post("/profiles/{name}/cmd")
|
||||
async def cmd(name: str, body: dict) -> dict:
|
||||
"""Send a raw chat command string to a cli profile (e.g. '/_send @1 text hi')."""
|
||||
return {"resp": await sup.send(name, body["cmd"])}
|
||||
|
||||
|
||||
@app.post("/profiles/{name}/stop")
|
||||
async def stop(name: str) -> dict:
|
||||
await sup.stop(name)
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
# ── Live events (WebSocket) ──────────────────────────────────────────────────────
|
||||
@app.websocket("/events")
|
||||
async def events(ws: WebSocket) -> None:
|
||||
await ws.accept()
|
||||
_browser_clients.add(ws)
|
||||
try:
|
||||
while True:
|
||||
await ws.receive_text() # keep the socket open; ignore inbound
|
||||
except WebSocketDisconnect:
|
||||
pass
|
||||
finally:
|
||||
_browser_clients.discard(ws)
|
||||
153
supervisor/supervisor.py
Normal file
153
supervisor/supervisor.py
Normal file
@@ -0,0 +1,153 @@
|
||||
"""Process supervisor: spawns and tracks the official SimpleX binaries.
|
||||
|
||||
Two kinds of managed process (see ../CLAUDE.md):
|
||||
- cli : `simplex-chat -p PORT -d PREFIX` → we hold a WebSocket and stream commands
|
||||
- directory : `simplex-directory-service ...` → autonomous; lifecycle + read its web-folder
|
||||
- broadcast : `simplex-broadcast-bot ...` → autonomous; lifecycle only
|
||||
|
||||
The DB belongs to each process; we never open it ourselves while the process runs.
|
||||
|
||||
NOTE: exact flag spellings for the autonomous bots should be confirmed with
|
||||
`<binary> --help` once binaries are in ./bin — the directory flags here follow
|
||||
apps/simplex-directory-service/src/Directory/Options.hs.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from .binaries import binary_path
|
||||
from .ws_client import SimplexWSClient
|
||||
|
||||
# (profile_name, event_record) -> None
|
||||
EventSink = Callable[[str, dict], Awaitable[None]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Managed:
|
||||
name: str
|
||||
kind: str # 'cli' | 'directory' | 'broadcast'
|
||||
proc: asyncio.subprocess.Process
|
||||
port: int | None = None
|
||||
client: SimplexWSClient | None = None
|
||||
|
||||
|
||||
class Supervisor:
|
||||
def __init__(self, data_dir: str = "data", base_port: int = 5300, on_event: EventSink | None = None):
|
||||
self.data_dir = Path(data_dir)
|
||||
self.data_dir.mkdir(parents=True, exist_ok=True)
|
||||
self._next_port = base_port
|
||||
self._procs: dict[str, Managed] = {}
|
||||
self._on_event = on_event
|
||||
|
||||
def _alloc_port(self) -> int:
|
||||
port = self._next_port
|
||||
self._next_port += 1
|
||||
return port
|
||||
|
||||
def list(self) -> list[dict]:
|
||||
return [
|
||||
{"name": m.name, "kind": m.kind, "port": m.port, "running": m.proc.returncode is None}
|
||||
for m in self._procs.values()
|
||||
]
|
||||
|
||||
# ── Pattern 1: interactive CLI (driven over WebSocket) ──────────────────────
|
||||
async def start_cli(self, name: str, display_name: str | None = None,
|
||||
allow_files: bool = True, extra_args: tuple[str, ...] = ()) -> Managed:
|
||||
if name in self._procs:
|
||||
return self._procs[name]
|
||||
port = self._alloc_port()
|
||||
prefix = str(self.data_dir / name)
|
||||
# On a fresh DB the CLI prompts for a display name on stdin and won't start
|
||||
# the WS server. --create-bot-display-name creates the profile non-interactively
|
||||
# (no-op once the DB exists, so it's safe to always pass).
|
||||
args = ["-p", str(port), "-d", prefix, "--create-bot-display-name", display_name or name]
|
||||
if allow_files:
|
||||
args.append("--create-bot-allow-files")
|
||||
args += list(extra_args)
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
binary_path("simplex-chat"), *args,
|
||||
stdin=asyncio.subprocess.DEVNULL,
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
client = SimplexWSClient(port, on_event=lambda r: self._emit(name, r))
|
||||
await self._connect_with_retry(client)
|
||||
m = Managed(name=name, kind="cli", proc=proc, port=port, client=client)
|
||||
self._procs[name] = m
|
||||
return m
|
||||
|
||||
async def _connect_with_retry(self, client: SimplexWSClient, attempts: int = 40, delay: float = 0.25) -> None:
|
||||
for _ in range(attempts):
|
||||
try:
|
||||
await client.connect()
|
||||
return
|
||||
except OSError:
|
||||
await asyncio.sleep(delay) # WS server not up yet
|
||||
raise RuntimeError("simplex-chat websocket did not come up")
|
||||
|
||||
async def send(self, name: str, cmd: str) -> dict:
|
||||
m = self._procs.get(name)
|
||||
if not m or not m.client:
|
||||
raise RuntimeError(f"{name!r} is not a running cli profile")
|
||||
return await m.client.send_cmd(cmd)
|
||||
|
||||
# ── Pattern 2: autonomous official bots (lifecycle + config-at-launch) ──────
|
||||
async def start_directory(self, name: str, super_users: str, web_folder: str,
|
||||
extra_args: tuple[str, ...] = ()) -> Managed:
|
||||
if name in self._procs:
|
||||
return self._procs[name]
|
||||
prefix = str(self.data_dir / name)
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
binary_path("simplex-directory-service"),
|
||||
"-d", prefix,
|
||||
"--super-users", super_users, # CONTACT_ID:NAME,...
|
||||
"--directory-file", f"{prefix}_directory.log", # append-only state log
|
||||
"--web-folder", web_folder, # bot writes listing files here
|
||||
*extra_args,
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
m = Managed(name=name, kind="directory", proc=proc) # no WS — autonomous
|
||||
self._procs[name] = m
|
||||
return m
|
||||
|
||||
async def start_broadcast(self, name: str, display_name: str, publishers: str,
|
||||
extra_args: tuple[str, ...] = ()) -> Managed:
|
||||
if name in self._procs:
|
||||
return self._procs[name]
|
||||
prefix = str(self.data_dir / name)
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
binary_path("simplex-broadcast-bot"),
|
||||
"-d", prefix,
|
||||
"--display-name", display_name,
|
||||
"--publishers", publishers, # CONTACT_ID:NAME,...
|
||||
*extra_args,
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
m = Managed(name=name, kind="broadcast", proc=proc)
|
||||
self._procs[name] = m
|
||||
return m
|
||||
|
||||
# ── Lifecycle ───────────────────────────────────────────────────────────────
|
||||
async def stop(self, name: str) -> None:
|
||||
m = self._procs.pop(name, None)
|
||||
if not m:
|
||||
return
|
||||
if m.client:
|
||||
await m.client.close()
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
m.proc.terminate()
|
||||
with contextlib.suppress(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(m.proc.wait(), 5)
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
if m.proc.returncode is None:
|
||||
m.proc.kill()
|
||||
|
||||
async def stop_all(self) -> None:
|
||||
for name in list(self._procs):
|
||||
await self.stop(name)
|
||||
|
||||
async def _emit(self, name: str, resp: dict) -> None:
|
||||
if self._on_event:
|
||||
await self._on_event(name, resp)
|
||||
73
supervisor/ws_client.py
Normal file
73
supervisor/ws_client.py
Normal file
@@ -0,0 +1,73 @@
|
||||
"""WebSocket client for a `simplex-chat -p <port>` process (Pattern 1 control).
|
||||
|
||||
Protocol (simplex-chat/bots/README.md):
|
||||
- send {"corrId": "<unique>", "cmd": "<command string>"}
|
||||
- response{"corrId": "<same>", "resp": {"type": ..., ...}} (matched by corrId)
|
||||
- event {"resp": {"type": ..., ...}} (no corrId)
|
||||
|
||||
We keep one long-lived connection per profile. Responses resolve futures keyed by
|
||||
corrId; events (no corrId) are pushed to an async callback.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import itertools
|
||||
import json
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
import websockets
|
||||
|
||||
EventHandler = Callable[[dict], Awaitable[None]]
|
||||
|
||||
|
||||
class SimplexWSClient:
|
||||
def __init__(self, port: int, on_event: EventHandler | None = None):
|
||||
self.url = f"ws://localhost:{port}/"
|
||||
self._on_event = on_event
|
||||
self._ws: websockets.WebSocketClientProtocol | None = None
|
||||
self._corr = itertools.count(1)
|
||||
self._pending: dict[str, asyncio.Future] = {}
|
||||
self._reader: asyncio.Task | None = None
|
||||
|
||||
async def connect(self) -> None:
|
||||
# max_size=None: chat responses (chat lists, profiles w/ avatars) can be large.
|
||||
self._ws = await websockets.connect(self.url, max_size=None)
|
||||
self._reader = asyncio.create_task(self._read_loop())
|
||||
|
||||
async def _read_loop(self) -> None:
|
||||
assert self._ws is not None
|
||||
async for raw in self._ws:
|
||||
try:
|
||||
msg = json.loads(raw)
|
||||
except Exception:
|
||||
continue
|
||||
corr = msg.get("corrId")
|
||||
resp = msg.get("resp")
|
||||
fut = self._pending.pop(corr, None) if corr else None
|
||||
if fut is not None:
|
||||
if not fut.done():
|
||||
fut.set_result(resp)
|
||||
elif resp is not None and self._on_event is not None:
|
||||
# Forward-compat: never fail on unknown event types (bots/README.md)
|
||||
try:
|
||||
await self._on_event(resp)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def send_cmd(self, cmd: str, timeout: float = 30.0) -> dict:
|
||||
"""Send a chat command string, await its matching response record."""
|
||||
if self._ws is None:
|
||||
raise RuntimeError("not connected")
|
||||
corr = str(next(self._corr))
|
||||
fut: asyncio.Future = asyncio.get_event_loop().create_future()
|
||||
self._pending[corr] = fut
|
||||
await self._ws.send(json.dumps({"corrId": corr, "cmd": cmd}))
|
||||
try:
|
||||
return await asyncio.wait_for(fut, timeout)
|
||||
finally:
|
||||
self._pending.pop(corr, None)
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._reader:
|
||||
self._reader.cancel()
|
||||
if self._ws:
|
||||
await self._ws.close()
|
||||
Reference in New Issue
Block a user