MCP Server

sajou includes an MCP (Model Context Protocol) server that lets AI agents interact with the visual choreographer programmatically. Any MCP-compatible client — Claude Code, Claude Desktop, or custom agents — can read scene state, compose choreographies, place entities, write GLSL shaders, and control uniforms in real-time.

Architecture

The MCP server is the source of truth for scene state. It runs as a standalone Node.js process with an in-memory state store. The scene-builder (browser) is a view/edit client that syncs with the server.

                 ┌─────────────────────────────┐
                 │   sajou MCP server           │
                 │   (@sajou/mcp-server)        │
                 │                              │
                 │   In-memory state store      │
                 │   REST API  (/api/*)         │
                 │   SSE streams                │
                 │   MCP stdio / HTTP (/mcp)    │
                 └──────┬──────────┬────────────┘
                        │          │
              ┌─────────┘          └──────────┐
              ▼                               ▼
     Browser (scene-builder)           AI Agent (Claude)
     connects via HTTP + SSE           connects via MCP
     view + edit scenes                compose scenes

The server works standalone — agents can compose scenes without any browser open. When a browser connects, it syncs bidirectionally: manual edits in the browser are pushed to the server, and agent commands from the server appear live in the browser.

Signal relay

The MCP server also acts as a signal relay for Claude Code. When Claude Code hooks fire, the sajou-emit CLI posts signals to the server, which broadcasts them to all connected browsers via SSE:

Claude Code hook → sajou-emit → POST /api/signal → MCP server
                                                        ↓
                              Browser ← EventSource(/__signals__/stream)

This works in both dev mode (Vite proxies to the server) and production (Tauri/static builds connect directly via __SERVER_URL__). The browser discovers and auto-connects to this endpoint as the local:claude-code signal source.

Installation

The server is published on npm. No need to clone the repo.

bash

npx -y @sajou/mcp-server --http

This starts the server on port 3001 (default). Specify a custom port:

bash

npx -y @sajou/mcp-server --http 3000

Running with a client

The scene-builder (browser client) connects to the server via HTTP. In development:

Terminal 1 — server:

bash

npx -y @sajou/mcp-server --http 3000

Terminal 2 — client:

bash

# From the sajou repo
cd tools/scene-builder && pnpm dev:vite

The Vite dev server proxies /api/* to http://localhost:3000 (configurable via SAJOU_SERVER env var). Open http://localhost:5175 in your browser.

The scene-builder can also run without a server (offline/Tauri mode) — it falls back to local IndexedDB storage.

Startup flow

Browser restores local state from IndexedDB
initServerConnection() probes server via GET /api/state/full (2s timeout)
If server has state → overwrites local stores with server data → starts sync + commands
If server is empty → starts sync (pushes local state on first push) + commands
If server unreachable → enters local mode (IDB only), starts reconnect with exponential backoff
SSE connection established for live state-change notifications

Connection status

The help bar (bottom-right corner) shows a server connection indicator:

Dot	Status	Meaning
Green	`connected`	Server reachable, bidirectional sync active
Gray + "local"	`local`	Server not found, IndexedDB only
Amber pulsing	`reconnecting`	Was connected, retrying with backoff (5s → 60s cap)

Click the dot to open a popover with:

Status line and last contact timestamp
Editable server URL — change the target server at runtime (persisted in localStorage). Empty = Vite proxy (default). Useful when the default port is taken.
Connection log — timestamped events (connected, lost, retrying, reconnected…)

State sync

Bidirectional sync between browser and server:

Push (browser → server): state-sync.ts pushes full state snapshot via POST /api/state/push (debounced 300ms)
Pull (server → browser): command-consumer.ts listens on /__commands__/stream SSE for state-change events, then re-fetches /api/state/full
Feedback prevention: isApplyingServerState flag suppresses pushes while applying server state

MCP client configuration

Claude Code / Claude Desktop

Add to your MCP config (~/.claude/claude_desktop_config.json or project .mcp.json):

json

{
  "mcpServers": {
    "sajou": {
      "command": "npx",
      "args": ["-y", "@sajou/mcp-server"]
    }
  }
}

This starts the server in stdio mode (MCP protocol over stdin/stdout). The server also accepts --http to run in HTTP mode with REST API and SSE.

Development (from the repo)

json

{
  "mcpServers": {
    "sajou": {
      "command": "pnpm",
      "args": ["--filter", "@sajou/mcp-server", "start"]
    }
  }
}

Entry points

Command	Mode	Use case
`npx -y @sajou/mcp-server`	stdio	Claude Code / Claude Desktop MCP integration
`npx -y @sajou/mcp-server --http`	HTTP (port 3001)	Standalone server with REST API + SSE + MCP HTTP
`npx -y @sajou/mcp-server --http 8080`	HTTP (custom port)	Same, custom port

Tools

The MCP server exposes 20+ tools organized into five categories.

Read tools — scene inspection

These tools let the agent understand what's currently on stage.

Tool	Description
`describe_scene`	Comprehensive human-readable summary of the entire scene — entities, choreographies, signal sources, bindings, wiring. The primary entry point for understanding scene state.
`get_scene_state`	Raw scene state — all placed entities with positions, visibility, layers, routes, dimensions, and editor mode.
`get_choreographies`	List all choreographies with trigger signal types, conditions, step types, and wiring info.
`get_shaders`	All GLSL shaders with full source code, uniforms, object groups, and pass count.
`get_sketches`	All sketches (p5.js / Three.js) with source code and parameters.
`map_signals`	View current signal-to-choreography wiring (read-only).

Write tools — scene composition

These tools let the agent build and modify scenes.

Tool	Description
`place_entity`	Place an entity on the scene at a given position. Supports scale, rotation, layer, z-index, animation state, and semantic ID (Actor ID) for choreography targeting.
`create_choreography`	Create a choreography — a sequence of animation steps triggered by a signal. Supports all actions: `move`, `fly`, `flash`, `spawn`, `destroy`, `wait`, `playSound`, `setAnimation`, `parallel`, `onArrive`, `onInterrupt`.
`create_binding`	Bind a choreography to an entity property (position, rotation, opacity, animation state) with optional mapping and transitions.
`create_wire`	Wire connections in the patch bay across three layers: signal→signal-type, signal-type→choreographer, choreographer→theme/shader.
`remove_item`	Remove entities, choreographies, bindings, wires, or signal sources. Cleans up dependent connections.

Entity management tools — inspect and control actors

These tools let the agent discover, inspect, and control individual entities. Use them to pilot actors at runtime — move characters, change states, adjust visibility — without creating a full choreography.

Tool	Description
`list_entities`	Roster of all placed entities with their semanticId, position, animation state, and key properties. Filter by `actors` (with semanticId), `decoration` (without), or `all`. This is the starting point — call it to learn what you can target.
`get_entity`	Full state of a single entity by semanticId or instance id. Returns position, visual state, topology, speech bubble config, and all active bindings targeting this entity.
`update_entity`	Partial update of entity properties — position (x, y), scale, rotation, opacity, visibility, layer, z-order, flip. Only provided fields change. Target by semanticId (preferred) or id.
`set_entity_state`	Shortcut to change an entity's animation state. The simplest way to "activate" something: `set_entity_state('door-kitchen', 'open')`, `set_entity_state('agent-1', 'walk')`.

Actor vs decoration: entities with a semanticId are actors — they have a name that choreographies and agents can target. Entities without a semanticId are passive decoration. Use list_entities(filter: 'actors') to see only targetable entities.

When to use update_entity vs choreography: update_entity applies changes immediately (teleport, snap). Choreographies animate changes over time (walk, fade). Use update_entity for setup and instant adjustments; use choreographies for narrative animations.

Shader tools — GPU effects

These tools let the agent create and control GLSL shaders.

Tool	Description
`create_shader`	Create a fragment/vertex shader with uniforms. Supports `@ui` controls (slider, color, toggle, xy), virtual object grouping (`@object`), and multi-pass feedback.
`update_shader`	Update an existing shader's code, uniforms, name, or pass count. Partial updates — only provided fields change.
`set_uniform`	Set a uniform value in real-time. Supports float, int, bool, vec2, vec3, vec4.
`get_shaders`	Read all shader definitions (also listed under read tools).

Sketch tools — p5.js + Three.js

These tools let the agent create and control live-coded sketches.

Tool	Description
`create_sketch`	Create a sketch (p5.js or Three.js mode) with source code and param annotations.
`update_sketch`	Update a sketch's source, name, mode, or params.
`set_sketch_param`	Set a sketch param value in real-time (e.g. `speed: 2.5`).

Runtime tools — signals

Tool	Description
`emit_signal`	Emit a signal to the scene. Triggers any choreographies wired to that signal type.

Resources — agent guides

The MCP server exposes resources that agents can read on demand via the MCP resource protocol (list_resources / read_resource). These are static reference guides that teach the agent how to work with sajou.

Resource	URI	Description
Entity Naming Guide	`sajou://guide/entity-naming`	Naming conventions for semantic IDs, positions, routes, zones. Read before placing entities.
Choreography Patterns	`sajou://guide/choreography-patterns`	7 common animation patterns (arrive-work-leave, flash, state toggle, parallel, etc.) with tool call structure.
Signal Protocol	`sajou://guide/signal-protocol`	Well-known signal types with payloads, custom types, MIDI, conditional triggers.

Usage: agents call list_resources to discover available guides, then read_resource(uri: "sajou://guide/entity-naming") to read one. This is optional context — agents can work without reading guides, but the guides significantly improve naming consistency and choreography quality.

State sync

The server maintains bidirectional state sync between the browser and external tools:

Client push: the browser pushes scene state to the server on every change (debounced 300ms)
Server push: external tools (MCP agents, REST API) mutate state on the server; changes are broadcast to the browser via SSE (/__commands__/stream)
SSE fallback: if the SSE stream disconnects, the browser falls back to polling every 500ms

Command delivery flow

MCP tool call ──┐
REST API POST ──┤──> Server mutates state
                │         │
                │         ▼
                │    /__commands__/stream (SSE)
                │         │
                │         ▼
                │    Browser applies command
                │         │
                │         ▼
                │    POST /api/commands/ack
                └──> Server prunes queue

REST API endpoints

Read (query state)

Endpoint	Method	Description
`/api/scene/state`	GET	Full scene state (entities, positions, layers, routes, lighting, particles)
`/api/choreographies`	GET	All choreographies with wiring metadata
`/api/bindings`	GET	All entity property bindings
`/api/wiring`	GET	Full wiring graph (signal → signal-type → choreographer → shader)
`/api/signals/sources`	GET	Connected signal sources (local + remote)
`/api/shaders`	GET	All shaders with source code and uniforms
`/api/p5`	GET	All sketches with source and params
`/api/discover/local`	GET	Probe local services (Claude Code, OpenClaw, LM Studio, Ollama)

Write (mutate scene)

Endpoint	Method	Description
`/api/scene/entities`	POST	Add, remove, or update entities
`/api/choreographies`	POST	Add, remove, or update choreographies
`/api/bindings`	POST	Add or remove bindings
`/api/wiring`	POST	Add or remove wire connections
`/api/signals/sources`	POST	Add or remove signal sources
`/api/shaders`	POST	Create a shader
`/api/shaders/:id`	PUT	Update an existing shader
`/api/shaders/:id`	DELETE	Remove a shader
`/api/shaders/:id/uniforms`	POST	Set a uniform value in real-time
`/api/p5`	POST	Create a sketch
`/api/p5/:id`	PUT	Update an existing sketch
`/api/p5/:id`	DELETE	Remove a sketch
`/api/p5/:id/params`	POST	Set a sketch param value in real-time
`/api/signal`	POST	Emit a signal (triggers wired choreographies)

Example workflow

Here's a complete example of an AI agent building a scene from scratch.

Step 1: Understand the scene

Tool: describe_scene
→ "Empty scene. No entities, no choreographies, no wiring."

Step 2: Place entities

Tool: place_entity
  entityId: "peon", x: 200, y: 300, semanticId: "worker"

Tool: place_entity
  entityId: "forge", x: 500, y: 300, semanticId: "forge"

Step 3: Create a choreography

Tool: create_choreography
  on: "task_dispatch"
  steps: [
    { "action": "move", "entity": "worker", "target": "forge", "duration": 1200 },
    { "action": "onArrive", "steps": [
      { "action": "flash", "entity": "forge", "params": { "color": "gold" } }
    ]}
  ]

Step 4: Wire and trigger

Tool: create_wire
  fromZone: "signal-type", fromId: "task_dispatch"
  toZone: "choreographer", toId: "<choreography-id>"

Tool: emit_signal
  type: "task_dispatch"
  payload: { "task": "gather_resources" }

The worker moves to the forge and a flash fires on arrival — composed entirely by the AI agent.

Step 5: Inspect and control actors

Tool: list_entities
  filter: "actors"
→ [{ semanticId: "worker", activeState: "idle", position: {x: 200, y: 300} },
   { semanticId: "forge",  activeState: "idle", position: {x: 500, y: 300} }]

Tool: get_entity
  semanticId: "worker"
→ Full entity state + active bindings

Tool: set_entity_state
  semanticId: "worker", state: "work"
→ Worker switches to 'work' animation

Tool: update_entity
  semanticId: "worker", opacity: 0.5, scale: 1.2
→ Worker becomes semi-transparent and slightly larger

Entity management tools give the agent direct control over actors for setup, debugging, and instant adjustments outside of choreographies.

For naming conventions (semantic IDs, positions, routes), see the entity naming guide. For common choreography recipes, see the choreography patterns guide.

MCP Server ​

Architecture ​

Signal relay ​

Installation ​

Running with a client ​

Startup flow ​

Connection status ​

State sync ​

MCP client configuration ​

Claude Code / Claude Desktop ​

Development (from the repo) ​

Entry points ​

Tools ​

Read tools — scene inspection ​

Write tools — scene composition ​

Entity management tools — inspect and control actors ​

Shader tools — GPU effects ​

Sketch tools — p5.js + Three.js ​

Runtime tools — signals ​

Resources — agent guides ​

State sync ​

Command delivery flow ​

REST API endpoints ​

Read (query state) ​

Write (mutate scene) ​

Example workflow ​

Step 1: Understand the scene ​

Step 2: Place entities ​

Step 3: Create a choreography ​

Step 4: Wire and trigger ​

Step 5: Inspect and control actors ​

MCP Server

Architecture

Signal relay

Installation

Running with a client

Startup flow

Connection status

State sync

MCP client configuration

Claude Code / Claude Desktop

Development (from the repo)

Entry points

Tools

Read tools — scene inspection

Write tools — scene composition

Entity management tools — inspect and control actors

Shader tools — GPU effects

Sketch tools — p5.js + Three.js

Runtime tools — signals

Resources — agent guides

State sync

Command delivery flow

REST API endpoints

Read (query state)

Write (mutate scene)

Example workflow

Step 1: Understand the scene

Step 2: Place entities

Step 3: Create a choreography

Step 4: Wire and trigger

Step 5: Inspect and control actors