WebX Docs
← Catalog Raw Markdown Open console

WebX WebSocket Protocol Documentation

This document describes the WebSocket protocol used by the WebX Server for real-time chat and browser automation streaming.

Overview

The WebX Server exposes two main WebSocket endpoints:

The server runs on port 8081 by default for WebSocket connections.

REST API Endpoints

Before connecting to WebSockets, you typically interact with tasks via REST API:

Start a Task

POST /api/tasks
Content-Type: application/json

{
    "task": "Navigate to example.com and click the login button"
}

Response:
{
    "session_id": "abc123",
    "status": "starting"
}

Get Task Status

GET /api/tasks/{session_id}

Response:
{
    "session_id": "abc123",
    "status": "running",
    "task": "Navigate to example.com and click the login button",
    "created_at": "2024-01-15T10:30:00Z",
    "completed_at": null
}

Stop a Task

POST /api/tasks/{session_id}/stop

Response:
{
    "session_id": "abc123",
    "status": "cancelled",
    "task": "Navigate to example.com and click the login button",
    "created_at": "2024-01-15T10:30:00Z",
    "completed_at": "2024-01-15T10:32:00Z"
}

Chat WebSocket Protocol

Connection

ws://localhost:8081/ws/chat?session_id=demo123

The session_id parameter is optional and defaults to "default" if not provided.

Client → Server Messages

User Message

{
    "type": "chat.user_msg",
    "session": "demo123",
    "text": "Navigate to google.com and search for 'rust programming'"
}

Server → Client Messages

Assistant Message Delta (Streaming)

{
    "type": "chat.delta",
    "delta": "I'll help you navigate to Google and search for 'rust programming'. Let me start by..."
}

Chat Completion

{
    "type": "chat.done"
}

Chat Error

{
    "type": "chat.error",
    "message": "Failed to process request: timeout"
}

Connection Management

Agent WebSocket Protocol

Connection

ws://localhost:8081/ws/agent?session_id=abc123

The session_id parameter is required and should match an active task session.

Message Types (Server → Client Only)

Agent Started

{
    "type": "agent.started",
    "session_id": "abc123",
    "task": "Navigate to example.com and click the login button"
}

Agent Observation

{
    "type": "agent.observe",
    "summary": "On Google homepage with search box visible",
    "url": "https://www.google.com"
}

Agent Action

{
    "type": "agent.action",
    "step": 3,
    "action": {
        "type": "click",
        "selector": "#search-button"
    }
}

Other action types:

{
    "type": "agent.action",
    "step": 4,
    "action": {
        "type": "type",
        "selector": "#search-input",
        "text": "rust programming"
    }
}

{
    "type": "agent.action",
    "step": 5,
    "action": {
        "type": "navigate",
        "url": "https://example.com"
    }
}

Action Result

{
    "type": "agent.result",
    "step": 3,
    "success": true,
    "notes": "Successfully clicked search button"
}

Screenshot Metadata

{
    "type": "agent.screenshot",
    "step": 3,
    "mime": "image/png",
    "id": "shot_0003"
}

Note: Screenshot metadata is followed immediately by a binary frame containing the PNG image data.

Agent Completion

{
    "type": "agent.finish",
    "success": true,
    "reason": "Task completed successfully - reached search results page"
}

Agent Error

{
    "type": "agent.error",
    "message": "Element not found: #login-button"
}

Screenshot Handling

Screenshots are sent as a two-part message:

  1. Text frame: JSON metadata with type: "agent.screenshot"
  2. Binary frame: PNG image data

Example client handling:

websocket.onmessage = (event) => {
    if (typeof event.data === 'string') {
        const msg = JSON.parse(event.data);
        if (msg.type === 'agent.screenshot') {
            console.log(`Screenshot ${msg.id} for step ${msg.step}`);
            // Next message will be binary with image data
        }
    } else if (event.data instanceof ArrayBuffer) {
        // Handle screenshot binary data
        const imageBlob = new Blob([event.data], { type: 'image/png' });
        // Process or display the image
    }
};

Connection Management

Error Handling

HTTP Errors

WebSocket Errors

Example Workflows

1. Start Task and Monitor Progress

1. POST /api/tasks → get session_id
2. Connect to /ws/agent?session_id=xxx
3. Listen for agent events until agent.finish or agent.error
4. GET /api/tasks/xxx for final status

2. Interactive Chat Session

1. Connect to /ws/chat?session_id=demo
2. Send chat.user_msg with task description
3. Stream chat.delta responses until chat.done
4. Optionally start automation task with the planned approach

3. Real-time Monitoring with Screenshots

1. Start task via REST API
2. Connect to agent WebSocket
3. Display agent events in real-time
4. Save binary screenshot frames as they arrive
5. Show completion status

Security Considerations

Client Libraries

Rust

See examples/websocket/agent_ws_client.rs and examples/websocket/chat_ws_client.rs

TypeScript/JavaScript

See clients/ts/ directory for example implementation

cURL Examples

Start a task:

curl -X POST http://localhost:8081/api/tasks \
  -H "Content-Type: application/json" \
  -d '{"task": "Navigate to example.com"}'

Get task status:

curl http://localhost:8081/api/tasks/abc123

Configuration

Server Configuration

Client Configuration