Architecture & Transport Protocol
OpenAgent bridges Instinct (which reasons in the cloud) to your local macOS machine without custom cloud servers, reverse proxies, or open firewall ports. It uses WhatsApp Desktop as a secure, end-to-end encrypted transport.
End-to-End Execution Flow
The JARVIS_CALL Transport Envelope
WhatsApp Desktop automatically processes chat text with its own Markdown parser: words surrounded by asterisks become bold (`*bold*`), underscores become italics (`_italics_`), and tildes become strikethrough (`~strike~`). When an LLM sends code or JSON with regex patterns, WhatsApp's parser frequently mangles payloads.
To guarantee pristine integrity, OpenAgent wraps all tool invocations inside a base64 transport envelope:
{
"tool": "edit",
"args": {
"path": "src/server.ts",
"old_string": "const PORT = 3000;",
"new_string": "const PORT = 8080;"
}
}Fail-Closed Chat Header Assertions
Because OpenAgent executes real shell commands and browser automation on your machine, preventing unauthorized execution is essential. OpenAgent implements strict fail-closed safety by default:
Header Verification
Before parsing any message, OpenAgent inspects WhatsApp's UI tree to confirm the selected conversation title matches your configured phone number (`+16508702892`).
Rejection on Ambiguity
If the window is hidden, another chat is clicked, or the phone number cannot be verified with certainty, execution is refused immediately with an audit log.
Offline Speech Pipeline & Echo Suppression
The voice architecture enables continuous conversation without cloud audio latency or microphone surveillance:
- Vosk Wake-Word Engine: Runs an offline 40MB acoustic model locally. Constantly monitors microphone audio buffer for "Wake up, Jarvis" using near-zero CPU.
- SoX Voice Activity Detection: Captures audio until a natural pause of 2.0 seconds (`BRIDGE_VOICE_SILENCE_SECONDS`), then automatically cuts and packages the clip.
- whisper.cpp Local STT: When configured with `BRIDGE_SEND_MODE=text`, runs 4-bit quantized Whisper models locally on Apple Silicon Metal or Intel AVX.
- Echo Guard: Automatically suppresses microphone input during Jarvis voice note playback so Jarvis never interrupts or loops over its own speech.
Bun + TypeScript Developer Harness
Rather than relying solely on Python subprocessing, OpenAgent offloads high-speed developer tasks to an isolated Bun TypeScript daemon (`src/tools/browser-harness` and `src/core/dispatcher.py`):