Scaffold SimpleX Orchestrate: supervisor over official binaries
A standalone control-plane app that spawns and drives the official SimpleX binaries (never modifies simplex source). Validated against simplex-chat built from source (stable v6.5.4, GHC 9.6.3). - CLAUDE.md: architecture notes mined from the upstream docs (WebSocket bot API, per-profile DBs, directory/broadcast bot config) - supervisor/: process registry + port allocation (supervisor.py), corrId/cmd<->resp WebSocket client (ws_client.py), binary locator (binaries.py), FastAPI front with REST control + /events stream (server.py) - smoke_test.py: Pattern-1 handshake (spawn simplex-chat -p, create+read user) — PASS - group_test.py: two accounts, invitation connect + group invite/join, verified membership over the real SMP network — PASS - build_chat.sh / install_ghc.sh: reproducible toolchain + from-source build Key finding: fresh DB prompts for a display name on stdin; spawn with --create-bot-display-name to start the WebSocket server non-interactively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
11
.gitignore
vendored
Normal file
11
.gitignore
vendored
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
# Upstream simplex-chat checkout (cloned separately; never modified/committed here)
|
||||||
|
simplex-chat/
|
||||||
|
# Built / downloaded binaries
|
||||||
|
bin/
|
||||||
|
# Python env and caches
|
||||||
|
.venv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
# Runtime per-profile databases + logs
|
||||||
|
data/
|
||||||
|
*.log
|
||||||
124
CLAUDE.md
Normal file
124
CLAUDE.md
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
# SimpleX Orchestrate — project notes for Claude
|
||||||
|
|
||||||
|
Goal: a web-managed **orchestrator** that runs the **official SimpleX binaries** (instead of
|
||||||
|
re-implementing bot logic against the FFI `libsimplex` SDK, as the older `simplex-manager`
|
||||||
|
prototype did). The manager spawns/supervises real SimpleX processes, drives the interactive
|
||||||
|
ones over WebSocket, and reads the autonomous ones' output.
|
||||||
|
|
||||||
|
Reference checkout: `./simplex-chat/` (cloned from https://github.com/simplex-chat/simplex-chat).
|
||||||
|
All facts below are taken from that repo's own docs — cited inline. There is **no** CLAUDE.md /
|
||||||
|
AGENTS.md in the upstream repo; the authoritative interface docs are:
|
||||||
|
- `simplex-chat/bots/README.md` — bot architecture & WebSocket API
|
||||||
|
- `simplex-chat/bots/api/COMMANDS.md`, `EVENTS.md`, `TYPES.md` — full command/event/type reference
|
||||||
|
- `simplex-chat/docs/CLI.md` — terminal client / DB / server flags
|
||||||
|
- `simplex-chat/apps/*/README.md` + `src/.../Options.hs` — the official bot binaries
|
||||||
|
- Official TypeScript SDK: `simplex-chat/packages/simplex-chat-client/typescript/`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Two integration patterns (this is the core design decision)
|
||||||
|
|
||||||
|
**1. Interactive accounts → drive the CLI over WebSocket** (`bots/README.md`)
|
||||||
|
- Run the terminal client as a local WebSocket server: `simplex-chat -p 5225`
|
||||||
|
- Our process connects over WebSocket and sends JSON commands / receives events.
|
||||||
|
- This is how custom bots (TS/Python/Rust) and our manager talk to a profile.
|
||||||
|
- Same command strings as the app's *Settings → Developer tools → Chat Console*.
|
||||||
|
|
||||||
|
**2. Autonomous official bots → run the prebuilt Haskell binary directly**
|
||||||
|
- `simplex-directory-service` and `simplex-broadcast-bot` are standalone executables built on
|
||||||
|
the chat core. They run their **own** logic, with their **own** DB + config, no WebSocket.
|
||||||
|
- The manager's job for these is lifecycle (spawn/stop/monitor/logs) + reading their output.
|
||||||
|
- **Use these instead of reimplementing directory/broadcast in Python** — they're battle-tested
|
||||||
|
and include captcha, moderation, approval, periodic re-checks, website generation, etc.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WebSocket protocol (pattern 1) — `bots/README.md`
|
||||||
|
|
||||||
|
- `simplex-chat -p <port>` starts the WS server. **localhost only, NO authentication** — keep the
|
||||||
|
bot process on the same machine and firewall the port. Messages are unencrypted.
|
||||||
|
- Command: `{"corrId":"<unique>", "cmd":"<command string>"}`
|
||||||
|
- Response: `{"corrId":"<same>", "resp":{"type":"<tag>", ...}}` (matched by corrId)
|
||||||
|
- Event: `{"resp":{"type":"<tag>", ...}}` (no corrId — connections, received messages, etc.)
|
||||||
|
- `NewChatItems` arrives both as a command response (after `APISendMessages`) and as an event.
|
||||||
|
- Parser must **ignore unknown event types / union tags / extra fields** (forward-compat rule).
|
||||||
|
- Minimal bot loop: handle `NewChatItems` event → reply with `APISendMessages`.
|
||||||
|
- Network usage per command is documented: `no` / `interactive` / `background`.
|
||||||
|
|
||||||
|
## Per-profile databases — `docs/CLI.md`
|
||||||
|
|
||||||
|
- `simplex-chat -d <prefix>` → creates `<prefix>_v1_chat.db` and `<prefix>_v1_agent.db`.
|
||||||
|
- Default data dir `~/.simplex` (`%APPDATA%/simplex` on Windows).
|
||||||
|
- **One process = one DB prefix.** A single DB *can* hold multiple bot profiles via
|
||||||
|
`/create bot [files=on] <name> [<bio>]`, but the simple model is one process per profile.
|
||||||
|
- SMP servers: `-s smp://<fp>@host`. Tor: `-x` or `--socks-proxy=:9050`. `simplex-chat -h` for all.
|
||||||
|
|
||||||
|
## Bot profile setup — `bots/README.md` (needs app/CLI v6.4.3+)
|
||||||
|
|
||||||
|
- Mark a profile as a bot: `peerType: "bot"`. Set via:
|
||||||
|
- CLI flags `--create-bot-display-name`, `--create-bot-allow-files` on first start, or
|
||||||
|
- `/create bot [files=on] '<name>' [<bio>]`, or
|
||||||
|
- `APIUpdateProfile` (also sets command menus).
|
||||||
|
- Command menus: `/set bot commands '<label>':/'<keyword>[ <params>]', '<label>':{...nested...}`
|
||||||
|
(e.g. directory's `'Show your groups':/list`). Per-contact overrides via `APISetContactPrefs`.
|
||||||
|
- **Business address** (support): a special group chat per connecting customer; inherits the bot's
|
||||||
|
preferences/commands. See `docs/BUSINESS.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Official bot: Directory service — `apps/simplex-directory-service/` (`Directory/Options.hs`)
|
||||||
|
|
||||||
|
Run the `simplex-directory-service` binary; it manages registration + search and **writes the
|
||||||
|
website files itself**. Replaces the Python directory reimplementation in the old prototype.
|
||||||
|
|
||||||
|
Key CLI options:
|
||||||
|
- `--super-users CONTACT_ID:DISPLAY_NAME,...` (required) — who can approve/manage listings
|
||||||
|
- `--admin-users CONTACT_ID:DISPLAY_NAME,...` — directory managers
|
||||||
|
- `--owners-group GROUP_ID:DISPLAY_NAME` — owners of listed groups auto-invited
|
||||||
|
- `--directory-file PATH` — append-only log holding directory state
|
||||||
|
- `--web-folder PATH` — **where static web assets / group listing files are written** (this is what
|
||||||
|
feeds the directory website; our old `web/<bot>/data/listing.json` was a hand-rolled stand-in)
|
||||||
|
- `--service-name "SimpleX Directory"` — bot display name (default shown)
|
||||||
|
- `--captcha-generator EXE` / `--voice-captcha-generator EXE` — anti-spam
|
||||||
|
- `--blocked-words-file` / `--blocked-fragments-file` / `--blocked-extenstion-rules` /
|
||||||
|
`--name-spelling-file` / `--profile-name-limit` — moderation
|
||||||
|
- `--link-check-interval SECONDS` (default 1800) — periodic re-check of public group links
|
||||||
|
- `--run-cli` — interactive CLI mode
|
||||||
|
- `--migrate-directory-file <check|import|export|listing>` — `listing` = `saveGroupListingFiles`
|
||||||
|
regenerates the website listing files from the log
|
||||||
|
- plus core chat options (DB file, SMP servers, Tor) shared with the CLI
|
||||||
|
- Registration flow = **owner adds the bot to their group**; listing stays pending until a
|
||||||
|
super-user approves (matches what we scoped for the prototype).
|
||||||
|
|
||||||
|
## Official bot: Broadcast — `apps/simplex-broadcast-bot/`
|
||||||
|
|
||||||
|
- `--display-name`, `--publishers CONTACT_ID:DISPLAY_NAME,...`, `--welcome`, `--prohibited`
|
||||||
|
(+ core DB/SMP options). Re-broadcasts messages from publishers to all connected contacts.
|
||||||
|
|
||||||
|
## Other official bot apps
|
||||||
|
- `apps/simplex-bot` / `simplex-bot-advanced` — minimal Haskell bot examples
|
||||||
|
- `apps/simplex-support-bot` — support/business example
|
||||||
|
- `apps/simplex-chat` — the terminal CLI itself
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Getting the binaries
|
||||||
|
- Install script: `curl -o- https://raw.githubusercontent.com/simplex-chat/simplex-chat/stable/install.sh | bash`
|
||||||
|
→ `simplex-chat` on PATH (CLI only).
|
||||||
|
- Prebuilt binaries: GitHub Releases (per stable version).
|
||||||
|
- Build from source (`docs/CLI.md`): GHC 9.6.3 + cabal 3.10.1.0, or
|
||||||
|
`DOCKER_BUILDKIT=1 docker build --output ~/.local/bin .`. Bot apps build via
|
||||||
|
`cabal install <exe>` (e.g. `simplex-directory-service`, `simplex-broadcast-bot`).
|
||||||
|
- **Build the stable branch**, not master.
|
||||||
|
|
||||||
|
## Proposed orchestrator architecture (to flesh out)
|
||||||
|
- Manager (web UI + supervisor) keeps a registry of profiles. Each profile has a kind:
|
||||||
|
- `cli` (interactive: user/echo/support/custom) → spawn `simplex-chat -p <port> -d data/<name>`,
|
||||||
|
connect WS, proxy commands; one port per profile.
|
||||||
|
- `directory` → spawn `simplex-directory-service` with `--directory-file`, `--web-folder`,
|
||||||
|
`--super-users`, DB; supervise + serve its web folder.
|
||||||
|
- `broadcast` → spawn `simplex-broadcast-bot` with publishers/welcome; supervise.
|
||||||
|
- Web frontend talks only to the manager; the manager owns process lifecycle, port allocation,
|
||||||
|
log capture, and (for directory) serving the generated web folder.
|
||||||
|
- Security: bind all CLI WS ports to localhost; never expose unauthenticated WS publicly.
|
||||||
|
</content>
|
||||||
14
build_chat.sh
Executable file
14
build_chat.sh
Executable file
@@ -0,0 +1,14 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
export PATH="$HOME/.ghcup/bin:$PATH"
|
||||||
|
cd /home/user/code/simplex-orchestrate/simplex-chat
|
||||||
|
echo "=== toolchain ==="; ghc --version; cabal --version
|
||||||
|
echo "=== cabal update ==="; cabal update
|
||||||
|
echo "=== building exe:simplex-chat (long) ==="
|
||||||
|
cabal build exe:simplex-chat
|
||||||
|
BIN="$(cabal list-bin exe:simplex-chat)"
|
||||||
|
echo "=== built: $BIN ==="
|
||||||
|
cp "$BIN" /home/user/code/simplex-orchestrate/bin/simplex-chat
|
||||||
|
chmod +x /home/user/code/simplex-orchestrate/bin/simplex-chat
|
||||||
|
/home/user/code/simplex-orchestrate/bin/simplex-chat --help >/dev/null 2>&1 && echo "binary runs OK"
|
||||||
|
echo "=== BUILD COMPLETE ==="
|
||||||
125
group_test.py
Normal file
125
group_test.py
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
"""Two-account integration test over the WebSocket interface:
|
||||||
|
|
||||||
|
alice ──invitation──▶ bob (1-time link connect → contacts)
|
||||||
|
alice creates group, invites bob → bob joins → verify membership both sides.
|
||||||
|
|
||||||
|
Exercises the real async + SMP-network path (not just local commands).
|
||||||
|
Run (needs ./bin/simplex-chat): .venv/bin/python group_test.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from supervisor.supervisor import Supervisor # noqa: E402
|
||||||
|
|
||||||
|
events: dict[str, list[dict]] = {"alice": [], "bob": []}
|
||||||
|
|
||||||
|
|
||||||
|
async def on_event(name: str, resp: dict) -> None:
|
||||||
|
events.setdefault(name, []).append(resp)
|
||||||
|
|
||||||
|
|
||||||
|
async def wait_event(name: str, pred, timeout: float, label: str) -> dict:
|
||||||
|
start = time.time()
|
||||||
|
i = 0
|
||||||
|
while time.time() - start < timeout:
|
||||||
|
while i < len(events[name]):
|
||||||
|
e = events[name][i]; i += 1
|
||||||
|
if pred(e):
|
||||||
|
return e
|
||||||
|
await asyncio.sleep(0.3)
|
||||||
|
raise TimeoutError(f"[{name}] timed out waiting for {label}")
|
||||||
|
|
||||||
|
|
||||||
|
async def poll(coro_factory, pred, timeout: float, label: str):
|
||||||
|
start = time.time()
|
||||||
|
while time.time() - start < timeout:
|
||||||
|
val = await coro_factory()
|
||||||
|
if pred(val):
|
||||||
|
return val
|
||||||
|
await asyncio.sleep(1.0)
|
||||||
|
raise TimeoutError(f"timed out polling for {label}")
|
||||||
|
|
||||||
|
|
||||||
|
def invitation_link(resp: dict) -> str:
|
||||||
|
inv = resp.get("connLinkInvitation") or {}
|
||||||
|
return inv.get("connShortLink") or inv.get("connFullLink", "")
|
||||||
|
|
||||||
|
|
||||||
|
async def main() -> int:
|
||||||
|
sup = Supervisor(data_dir="data", base_port=5500, on_event=on_event)
|
||||||
|
ok = False
|
||||||
|
try:
|
||||||
|
print("→ starting two accounts (alice, bob) …")
|
||||||
|
await sup.start_cli("alice")
|
||||||
|
await sup.start_cli("bob")
|
||||||
|
for n in ("alice", "bob"):
|
||||||
|
r = await sup.send(n, "/_start")
|
||||||
|
print(f" {n} /_start -> {r.get('type')}")
|
||||||
|
|
||||||
|
uidA = (await sup.send("alice", "/user"))["user"]["userId"]
|
||||||
|
uidB = (await sup.send("bob", "/user"))["user"]["userId"]
|
||||||
|
print(f" alice uid={uidA} bob uid={uidB}")
|
||||||
|
|
||||||
|
# 1. connect via 1-time invitation
|
||||||
|
inv = await sup.send("alice", f"/_connect {uidA}")
|
||||||
|
link = invitation_link(inv)
|
||||||
|
print(f"→ alice invitation ({inv.get('type')}): {link[:55]}…")
|
||||||
|
cb = await sup.send("bob", f"/connect {link}")
|
||||||
|
print(f"→ bob /connect -> {cb.get('type')}")
|
||||||
|
|
||||||
|
print("→ waiting for contactConnected (SMP handshake) …")
|
||||||
|
await wait_event("alice", lambda e: e.get("type") == "contactConnected", 120, "alice contactConnected")
|
||||||
|
print(" ✓ alice: contact connected")
|
||||||
|
await wait_event("bob", lambda e: e.get("type") == "contactConnected", 120, "bob contactConnected")
|
||||||
|
print(" ✓ bob: contact connected")
|
||||||
|
|
||||||
|
# 2. alice creates a group and invites bob
|
||||||
|
contacts = (await sup.send("alice", f"/_contacts {uidA}")).get("contacts", [])
|
||||||
|
bob_cid = contacts[0]["contactId"]
|
||||||
|
print(f"→ bob is contactId={bob_cid} on alice")
|
||||||
|
gr = await sup.send("alice", f'/_group {uidA} ' + json.dumps({"displayName": "testgrp", "fullName": ""}))
|
||||||
|
gid = gr["groupInfo"]["groupId"]
|
||||||
|
print(f"→ group created gid={gid} ({gr.get('type')})")
|
||||||
|
add = await sup.send("alice", f"/_add #{gid} {bob_cid} member")
|
||||||
|
print(f"→ alice /_add bob -> {add.get('type')}")
|
||||||
|
|
||||||
|
# 3. bob sees the pending invite, then joins
|
||||||
|
async def bob_groups():
|
||||||
|
return (await sup.send("bob", f"/_groups {uidB}")).get("groups", [])
|
||||||
|
bgroups = await poll(bob_groups, lambda gs: len(gs) > 0, 120, "bob receives group invite")
|
||||||
|
bg = bgroups[0]
|
||||||
|
print(f" ✓ bob sees group #{bg['groupId']} status={bg['membership']['memberStatus']}")
|
||||||
|
j = await sup.send("bob", f"/_join #{bg['groupId']}")
|
||||||
|
print(f"→ bob /_join -> {j.get('type')}")
|
||||||
|
|
||||||
|
# 4. verify membership on both sides
|
||||||
|
print("→ waiting for member to connect in group …")
|
||||||
|
await wait_event(
|
||||||
|
"alice",
|
||||||
|
lambda e: e.get("type") in ("memberConnected", "joinedGroupMember", "connectedToGroupMember"),
|
||||||
|
120, "alice memberConnected",
|
||||||
|
)
|
||||||
|
print(" ✓ alice: member connected event")
|
||||||
|
|
||||||
|
async def alice_member_count():
|
||||||
|
ag = (await sup.send("alice", f"/_groups {uidA}")).get("groups", [])
|
||||||
|
return ag[0]["groupSummary"]["currentMembers"] if ag else 0
|
||||||
|
count = await poll(alice_member_count, lambda c: c >= 2, 60, "group reaches 2 members")
|
||||||
|
mem = (await sup.send("alice", f"/_members #{gid}"))["group"]["members"]
|
||||||
|
print(f" alice group members: {[(m['localDisplayName'], m['memberStatus']) for m in mem]}")
|
||||||
|
print(f" alice group currentMembers = {count}")
|
||||||
|
ok = count >= 2
|
||||||
|
finally:
|
||||||
|
await sup.stop_all()
|
||||||
|
|
||||||
|
print("\nRESULT:", "PASS — invite/join verified end to end" if ok else "INCOMPLETE")
|
||||||
|
return 0 if ok else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(asyncio.run(main()))
|
||||||
11
install_ghc.sh
Executable file
11
install_ghc.sh
Executable file
@@ -0,0 +1,11 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
export BOOTSTRAP_HASKELL_NONINTERACTIVE=1
|
||||||
|
export BOOTSTRAP_HASKELL_GHC_VERSION=9.6.3
|
||||||
|
export BOOTSTRAP_HASKELL_CABAL_VERSION=3.10.1.0
|
||||||
|
export BOOTSTRAP_HASKELL_INSTALL_NO_STACK=1
|
||||||
|
export BOOTSTRAP_HASKELL_ADJUST_BASHRC=0
|
||||||
|
curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh
|
||||||
|
echo "=== DONE ==="
|
||||||
|
"$HOME/.ghcup/bin/ghc" --version
|
||||||
|
"$HOME/.ghcup/bin/cabal" --version
|
||||||
3
requirements.txt
Normal file
3
requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
fastapi
|
||||||
|
uvicorn[standard]
|
||||||
|
websockets
|
||||||
67
smoke_test.py
Normal file
67
smoke_test.py
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
"""Pattern-1 smoke test: spawn a real `simplex-chat -p` process, drive it over the
|
||||||
|
WebSocket corrId/cmd↔resp handshake, and confirm a structured response comes back.
|
||||||
|
|
||||||
|
Validates the supervisor + ws_client against a real binary. Local commands only
|
||||||
|
(/user, /_create user, /users) — no network needed.
|
||||||
|
|
||||||
|
Run (with ./bin/simplex-chat present): python smoke_test.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from supervisor.supervisor import Supervisor # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
async def main() -> int:
|
||||||
|
events: list[tuple[str, str]] = []
|
||||||
|
|
||||||
|
async def on_event(name: str, resp: dict) -> None:
|
||||||
|
events.append((name, resp.get("type", "?")))
|
||||||
|
|
||||||
|
sup = Supervisor(data_dir="data", base_port=5400, on_event=on_event)
|
||||||
|
ok = False
|
||||||
|
try:
|
||||||
|
print("→ spawning simplex-chat -p (profile 'smoke') …")
|
||||||
|
m = await sup.start_cli("smoke")
|
||||||
|
print(f" process pid={m.proc.pid} port={m.port}, WebSocket connected")
|
||||||
|
|
||||||
|
print("→ /user (show active user)")
|
||||||
|
r = await sup.send("smoke", "/user")
|
||||||
|
print(" resp.type =", r.get("type"))
|
||||||
|
|
||||||
|
if r.get("type") != "activeUser":
|
||||||
|
new_user = {
|
||||||
|
"profile": {"displayName": "smoke", "fullName": ""},
|
||||||
|
"pastTimestamp": False,
|
||||||
|
"userChatRelay": False,
|
||||||
|
}
|
||||||
|
print("→ /_create user")
|
||||||
|
rc = await sup.send("smoke", "/_create user " + json.dumps(new_user))
|
||||||
|
print(" resp.type =", rc.get("type"))
|
||||||
|
r = await sup.send("smoke", "/user")
|
||||||
|
print(" /user resp.type =", r.get("type"))
|
||||||
|
|
||||||
|
user = r.get("user") or {}
|
||||||
|
print(" active user displayName =", user.get("localDisplayName"))
|
||||||
|
|
||||||
|
rl = await sup.send("smoke", "/users")
|
||||||
|
print("→ /users resp.type =", rl.get("type"),
|
||||||
|
"count =", len(rl.get("users", []) if isinstance(rl, dict) else []))
|
||||||
|
|
||||||
|
ok = r.get("type") == "activeUser"
|
||||||
|
finally:
|
||||||
|
await sup.stop("smoke")
|
||||||
|
print("→ stopped process")
|
||||||
|
|
||||||
|
print(" events observed:", events[:5])
|
||||||
|
print("\nRESULT:", "PASS — corrId/cmd↔resp handshake works" if ok else "FAIL")
|
||||||
|
return 0 if ok else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(asyncio.run(main()))
|
||||||
9
supervisor/__init__.py
Normal file
9
supervisor/__init__.py
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
"""SimpleX Orchestrate supervisor — spawns and drives the official SimpleX binaries.
|
||||||
|
|
||||||
|
It never modifies or rebuilds simplex-chat; it only invokes prebuilt binaries.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .supervisor import Managed, Supervisor
|
||||||
|
from .ws_client import SimplexWSClient
|
||||||
|
|
||||||
|
__all__ = ["Supervisor", "Managed", "SimplexWSClient"]
|
||||||
24
supervisor/binaries.py
Normal file
24
supervisor/binaries.py
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
"""Locate the official SimpleX binaries — never built/modified here, only invoked.
|
||||||
|
|
||||||
|
Resolution order: $SIMPLEX_BIN dir, then ./bin/, then PATH. Place prebuilt
|
||||||
|
binaries (simplex-chat, simplex-directory-service, simplex-broadcast-bot) in ./bin/
|
||||||
|
or install via the upstream install script. We never compile or alter them.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
BIN_DIR = Path(os.environ.get("SIMPLEX_BIN", Path(__file__).resolve().parent.parent / "bin"))
|
||||||
|
|
||||||
|
KNOWN = ("simplex-chat", "simplex-directory-service", "simplex-broadcast-bot")
|
||||||
|
|
||||||
|
|
||||||
|
def binary_path(name: str) -> str:
|
||||||
|
local = BIN_DIR / name
|
||||||
|
if local.exists():
|
||||||
|
return str(local)
|
||||||
|
found = shutil.which(name)
|
||||||
|
if found:
|
||||||
|
return found
|
||||||
|
raise FileNotFoundError(f"{name!r} not found in {BIN_DIR} or PATH")
|
||||||
90
supervisor/server.py
Normal file
90
supervisor/server.py
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
"""Front-facing API the website talks to.
|
||||||
|
|
||||||
|
The browser:
|
||||||
|
- opens a WebSocket to /events to receive live chat events from every profile
|
||||||
|
(each event is tagged with the originating profile name);
|
||||||
|
- uses REST to spawn/stop profiles and to send commands to cli profiles.
|
||||||
|
|
||||||
|
This process sits between the website and the SimpleX binaries; it is the only
|
||||||
|
thing that touches the binaries. Run: uvicorn supervisor.server:app
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import contextlib
|
||||||
|
|
||||||
|
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
|
||||||
|
|
||||||
|
from .supervisor import Supervisor
|
||||||
|
|
||||||
|
app = FastAPI(title="SimpleX Orchestrate")
|
||||||
|
|
||||||
|
_browser_clients: set[WebSocket] = set()
|
||||||
|
|
||||||
|
|
||||||
|
async def _broadcast(name: str, event: dict) -> None:
|
||||||
|
dead = []
|
||||||
|
for ws in _browser_clients:
|
||||||
|
try:
|
||||||
|
await ws.send_json({"profile": name, "event": event})
|
||||||
|
except Exception:
|
||||||
|
dead.append(ws)
|
||||||
|
for ws in dead:
|
||||||
|
_browser_clients.discard(ws)
|
||||||
|
|
||||||
|
|
||||||
|
sup = Supervisor(on_event=_broadcast)
|
||||||
|
|
||||||
|
|
||||||
|
@app.on_event("shutdown")
|
||||||
|
async def _shutdown() -> None:
|
||||||
|
await sup.stop_all()
|
||||||
|
|
||||||
|
|
||||||
|
# ── Control (REST) ──────────────────────────────────────────────────────────────
|
||||||
|
@app.get("/profiles")
|
||||||
|
async def profiles() -> dict:
|
||||||
|
return {"profiles": sup.list()}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/profiles/{name}/start-cli")
|
||||||
|
async def start_cli(name: str) -> dict:
|
||||||
|
await sup.start_cli(name)
|
||||||
|
return {"ok": True}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/profiles/{name}/start-directory")
|
||||||
|
async def start_directory(name: str, body: dict) -> dict:
|
||||||
|
await sup.start_directory(name, body["super_users"], body["web_folder"])
|
||||||
|
return {"ok": True}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/profiles/{name}/start-broadcast")
|
||||||
|
async def start_broadcast(name: str, body: dict) -> dict:
|
||||||
|
await sup.start_broadcast(name, body["display_name"], body["publishers"])
|
||||||
|
return {"ok": True}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/profiles/{name}/cmd")
|
||||||
|
async def cmd(name: str, body: dict) -> dict:
|
||||||
|
"""Send a raw chat command string to a cli profile (e.g. '/_send @1 text hi')."""
|
||||||
|
return {"resp": await sup.send(name, body["cmd"])}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/profiles/{name}/stop")
|
||||||
|
async def stop(name: str) -> dict:
|
||||||
|
await sup.stop(name)
|
||||||
|
return {"ok": True}
|
||||||
|
|
||||||
|
|
||||||
|
# ── Live events (WebSocket) ──────────────────────────────────────────────────────
|
||||||
|
@app.websocket("/events")
|
||||||
|
async def events(ws: WebSocket) -> None:
|
||||||
|
await ws.accept()
|
||||||
|
_browser_clients.add(ws)
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
await ws.receive_text() # keep the socket open; ignore inbound
|
||||||
|
except WebSocketDisconnect:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
_browser_clients.discard(ws)
|
||||||
153
supervisor/supervisor.py
Normal file
153
supervisor/supervisor.py
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
"""Process supervisor: spawns and tracks the official SimpleX binaries.
|
||||||
|
|
||||||
|
Two kinds of managed process (see ../CLAUDE.md):
|
||||||
|
- cli : `simplex-chat -p PORT -d PREFIX` → we hold a WebSocket and stream commands
|
||||||
|
- directory : `simplex-directory-service ...` → autonomous; lifecycle + read its web-folder
|
||||||
|
- broadcast : `simplex-broadcast-bot ...` → autonomous; lifecycle only
|
||||||
|
|
||||||
|
The DB belongs to each process; we never open it ourselves while the process runs.
|
||||||
|
|
||||||
|
NOTE: exact flag spellings for the autonomous bots should be confirmed with
|
||||||
|
`<binary> --help` once binaries are in ./bin — the directory flags here follow
|
||||||
|
apps/simplex-directory-service/src/Directory/Options.hs.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import contextlib
|
||||||
|
from collections.abc import Awaitable, Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .binaries import binary_path
|
||||||
|
from .ws_client import SimplexWSClient
|
||||||
|
|
||||||
|
# (profile_name, event_record) -> None
|
||||||
|
EventSink = Callable[[str, dict], Awaitable[None]]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Managed:
|
||||||
|
name: str
|
||||||
|
kind: str # 'cli' | 'directory' | 'broadcast'
|
||||||
|
proc: asyncio.subprocess.Process
|
||||||
|
port: int | None = None
|
||||||
|
client: SimplexWSClient | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Supervisor:
|
||||||
|
def __init__(self, data_dir: str = "data", base_port: int = 5300, on_event: EventSink | None = None):
|
||||||
|
self.data_dir = Path(data_dir)
|
||||||
|
self.data_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._next_port = base_port
|
||||||
|
self._procs: dict[str, Managed] = {}
|
||||||
|
self._on_event = on_event
|
||||||
|
|
||||||
|
def _alloc_port(self) -> int:
|
||||||
|
port = self._next_port
|
||||||
|
self._next_port += 1
|
||||||
|
return port
|
||||||
|
|
||||||
|
def list(self) -> list[dict]:
|
||||||
|
return [
|
||||||
|
{"name": m.name, "kind": m.kind, "port": m.port, "running": m.proc.returncode is None}
|
||||||
|
for m in self._procs.values()
|
||||||
|
]
|
||||||
|
|
||||||
|
# ── Pattern 1: interactive CLI (driven over WebSocket) ──────────────────────
|
||||||
|
async def start_cli(self, name: str, display_name: str | None = None,
|
||||||
|
allow_files: bool = True, extra_args: tuple[str, ...] = ()) -> Managed:
|
||||||
|
if name in self._procs:
|
||||||
|
return self._procs[name]
|
||||||
|
port = self._alloc_port()
|
||||||
|
prefix = str(self.data_dir / name)
|
||||||
|
# On a fresh DB the CLI prompts for a display name on stdin and won't start
|
||||||
|
# the WS server. --create-bot-display-name creates the profile non-interactively
|
||||||
|
# (no-op once the DB exists, so it's safe to always pass).
|
||||||
|
args = ["-p", str(port), "-d", prefix, "--create-bot-display-name", display_name or name]
|
||||||
|
if allow_files:
|
||||||
|
args.append("--create-bot-allow-files")
|
||||||
|
args += list(extra_args)
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
binary_path("simplex-chat"), *args,
|
||||||
|
stdin=asyncio.subprocess.DEVNULL,
|
||||||
|
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||||
|
)
|
||||||
|
client = SimplexWSClient(port, on_event=lambda r: self._emit(name, r))
|
||||||
|
await self._connect_with_retry(client)
|
||||||
|
m = Managed(name=name, kind="cli", proc=proc, port=port, client=client)
|
||||||
|
self._procs[name] = m
|
||||||
|
return m
|
||||||
|
|
||||||
|
async def _connect_with_retry(self, client: SimplexWSClient, attempts: int = 40, delay: float = 0.25) -> None:
|
||||||
|
for _ in range(attempts):
|
||||||
|
try:
|
||||||
|
await client.connect()
|
||||||
|
return
|
||||||
|
except OSError:
|
||||||
|
await asyncio.sleep(delay) # WS server not up yet
|
||||||
|
raise RuntimeError("simplex-chat websocket did not come up")
|
||||||
|
|
||||||
|
async def send(self, name: str, cmd: str) -> dict:
|
||||||
|
m = self._procs.get(name)
|
||||||
|
if not m or not m.client:
|
||||||
|
raise RuntimeError(f"{name!r} is not a running cli profile")
|
||||||
|
return await m.client.send_cmd(cmd)
|
||||||
|
|
||||||
|
# ── Pattern 2: autonomous official bots (lifecycle + config-at-launch) ──────
|
||||||
|
async def start_directory(self, name: str, super_users: str, web_folder: str,
|
||||||
|
extra_args: tuple[str, ...] = ()) -> Managed:
|
||||||
|
if name in self._procs:
|
||||||
|
return self._procs[name]
|
||||||
|
prefix = str(self.data_dir / name)
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
binary_path("simplex-directory-service"),
|
||||||
|
"-d", prefix,
|
||||||
|
"--super-users", super_users, # CONTACT_ID:NAME,...
|
||||||
|
"--directory-file", f"{prefix}_directory.log", # append-only state log
|
||||||
|
"--web-folder", web_folder, # bot writes listing files here
|
||||||
|
*extra_args,
|
||||||
|
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||||
|
)
|
||||||
|
m = Managed(name=name, kind="directory", proc=proc) # no WS — autonomous
|
||||||
|
self._procs[name] = m
|
||||||
|
return m
|
||||||
|
|
||||||
|
async def start_broadcast(self, name: str, display_name: str, publishers: str,
|
||||||
|
extra_args: tuple[str, ...] = ()) -> Managed:
|
||||||
|
if name in self._procs:
|
||||||
|
return self._procs[name]
|
||||||
|
prefix = str(self.data_dir / name)
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
binary_path("simplex-broadcast-bot"),
|
||||||
|
"-d", prefix,
|
||||||
|
"--display-name", display_name,
|
||||||
|
"--publishers", publishers, # CONTACT_ID:NAME,...
|
||||||
|
*extra_args,
|
||||||
|
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT,
|
||||||
|
)
|
||||||
|
m = Managed(name=name, kind="broadcast", proc=proc)
|
||||||
|
self._procs[name] = m
|
||||||
|
return m
|
||||||
|
|
||||||
|
# ── Lifecycle ───────────────────────────────────────────────────────────────
|
||||||
|
async def stop(self, name: str) -> None:
|
||||||
|
m = self._procs.pop(name, None)
|
||||||
|
if not m:
|
||||||
|
return
|
||||||
|
if m.client:
|
||||||
|
await m.client.close()
|
||||||
|
with contextlib.suppress(ProcessLookupError):
|
||||||
|
m.proc.terminate()
|
||||||
|
with contextlib.suppress(asyncio.TimeoutError):
|
||||||
|
await asyncio.wait_for(m.proc.wait(), 5)
|
||||||
|
with contextlib.suppress(ProcessLookupError):
|
||||||
|
if m.proc.returncode is None:
|
||||||
|
m.proc.kill()
|
||||||
|
|
||||||
|
async def stop_all(self) -> None:
|
||||||
|
for name in list(self._procs):
|
||||||
|
await self.stop(name)
|
||||||
|
|
||||||
|
async def _emit(self, name: str, resp: dict) -> None:
|
||||||
|
if self._on_event:
|
||||||
|
await self._on_event(name, resp)
|
||||||
73
supervisor/ws_client.py
Normal file
73
supervisor/ws_client.py
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
"""WebSocket client for a `simplex-chat -p <port>` process (Pattern 1 control).
|
||||||
|
|
||||||
|
Protocol (simplex-chat/bots/README.md):
|
||||||
|
- send {"corrId": "<unique>", "cmd": "<command string>"}
|
||||||
|
- response{"corrId": "<same>", "resp": {"type": ..., ...}} (matched by corrId)
|
||||||
|
- event {"resp": {"type": ..., ...}} (no corrId)
|
||||||
|
|
||||||
|
We keep one long-lived connection per profile. Responses resolve futures keyed by
|
||||||
|
corrId; events (no corrId) are pushed to an async callback.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import itertools
|
||||||
|
import json
|
||||||
|
from collections.abc import Awaitable, Callable
|
||||||
|
|
||||||
|
import websockets
|
||||||
|
|
||||||
|
EventHandler = Callable[[dict], Awaitable[None]]
|
||||||
|
|
||||||
|
|
||||||
|
class SimplexWSClient:
|
||||||
|
def __init__(self, port: int, on_event: EventHandler | None = None):
|
||||||
|
self.url = f"ws://localhost:{port}/"
|
||||||
|
self._on_event = on_event
|
||||||
|
self._ws: websockets.WebSocketClientProtocol | None = None
|
||||||
|
self._corr = itertools.count(1)
|
||||||
|
self._pending: dict[str, asyncio.Future] = {}
|
||||||
|
self._reader: asyncio.Task | None = None
|
||||||
|
|
||||||
|
async def connect(self) -> None:
|
||||||
|
# max_size=None: chat responses (chat lists, profiles w/ avatars) can be large.
|
||||||
|
self._ws = await websockets.connect(self.url, max_size=None)
|
||||||
|
self._reader = asyncio.create_task(self._read_loop())
|
||||||
|
|
||||||
|
async def _read_loop(self) -> None:
|
||||||
|
assert self._ws is not None
|
||||||
|
async for raw in self._ws:
|
||||||
|
try:
|
||||||
|
msg = json.loads(raw)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
corr = msg.get("corrId")
|
||||||
|
resp = msg.get("resp")
|
||||||
|
fut = self._pending.pop(corr, None) if corr else None
|
||||||
|
if fut is not None:
|
||||||
|
if not fut.done():
|
||||||
|
fut.set_result(resp)
|
||||||
|
elif resp is not None and self._on_event is not None:
|
||||||
|
# Forward-compat: never fail on unknown event types (bots/README.md)
|
||||||
|
try:
|
||||||
|
await self._on_event(resp)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def send_cmd(self, cmd: str, timeout: float = 30.0) -> dict:
|
||||||
|
"""Send a chat command string, await its matching response record."""
|
||||||
|
if self._ws is None:
|
||||||
|
raise RuntimeError("not connected")
|
||||||
|
corr = str(next(self._corr))
|
||||||
|
fut: asyncio.Future = asyncio.get_event_loop().create_future()
|
||||||
|
self._pending[corr] = fut
|
||||||
|
await self._ws.send(json.dumps({"corrId": corr, "cmd": cmd}))
|
||||||
|
try:
|
||||||
|
return await asyncio.wait_for(fut, timeout)
|
||||||
|
finally:
|
||||||
|
self._pending.pop(corr, None)
|
||||||
|
|
||||||
|
async def close(self) -> None:
|
||||||
|
if self._reader:
|
||||||
|
self._reader.cancel()
|
||||||
|
if self._ws:
|
||||||
|
await self._ws.close()
|
||||||
Reference in New Issue
Block a user