# WebX WebSocket Protocol Documentation

This document describes the WebSocket protocol used by the WebX Server for real-time chat and browser automation streaming.

## Overview

The WebX Server exposes two main WebSocket endpoints:
- **Chat WebSocket** (`/ws/chat`): For real-time LLM chat interactions
- **Agent WebSocket** (`/ws/agent`): For streaming browser automation events

The server runs on port **8081** by default for WebSocket connections.

## REST API Endpoints

Before connecting to WebSockets, you typically interact with tasks via REST API:

### Start a Task
```
POST /api/tasks
Content-Type: application/json

{
    "task": "Navigate to example.com and click the login button"
}

Response:
{
    "session_id": "abc123",
    "status": "starting"
}
```

### Get Task Status
```
GET /api/tasks/{session_id}

Response:
{
    "session_id": "abc123",
    "status": "running",
    "task": "Navigate to example.com and click the login button",
    "created_at": "2024-01-15T10:30:00Z",
    "completed_at": null
}
```

### Stop a Task
```
POST /api/tasks/{session_id}/stop

Response:
{
    "session_id": "abc123",
    "status": "cancelled",
    "task": "Navigate to example.com and click the login button",
    "created_at": "2024-01-15T10:30:00Z",
    "completed_at": "2024-01-15T10:32:00Z"
}
```

## Chat WebSocket Protocol

### Connection
```
ws://localhost:8081/ws/chat?session_id=demo123
```

The `session_id` parameter is optional and defaults to "default" if not provided.

### Client → Server Messages

#### User Message
```json
{
    "type": "chat.user_msg",
    "session": "demo123",
    "text": "Navigate to google.com and search for 'rust programming'"
}
```

### Server → Client Messages

#### Assistant Message Delta (Streaming)
```json
{
    "type": "chat.delta",
    "delta": "I'll help you navigate to Google and search for 'rust programming'. Let me start by..."
}
```

#### Chat Completion
```json
{
    "type": "chat.done"
}
```

#### Chat Error
```json
{
    "type": "chat.error",
    "message": "Failed to process request: timeout"
}
```

### Connection Management
- **Heartbeat**: Server sends ping frames every 25 seconds
- **Timeout**: Connection closed after 3 missed pongs
- **Reconnection**: Client should implement exponential backoff

## Agent WebSocket Protocol

### Connection
```
ws://localhost:8081/ws/agent?session_id=abc123
```

The `session_id` parameter is **required** and should match an active task session.

### Message Types (Server → Client Only)

#### Agent Started
```json
{
    "type": "agent.started",
    "session_id": "abc123",
    "task": "Navigate to example.com and click the login button"
}
```

#### Agent Observation
```json
{
    "type": "agent.observe",
    "summary": "On Google homepage with search box visible",
    "url": "https://www.google.com"
}
```

#### Agent Action
```json
{
    "type": "agent.action",
    "step": 3,
    "action": {
        "type": "click",
        "selector": "#search-button"
    }
}
```

Other action types:
```json
{
    "type": "agent.action",
    "step": 4,
    "action": {
        "type": "type",
        "selector": "#search-input",
        "text": "rust programming"
    }
}
```

```json
{
    "type": "agent.action",
    "step": 5,
    "action": {
        "type": "navigate",
        "url": "https://example.com"
    }
}
```

#### Action Result
```json
{
    "type": "agent.result",
    "step": 3,
    "success": true,
    "notes": "Successfully clicked search button"
}
```

#### Screenshot Metadata
```json
{
    "type": "agent.screenshot",
    "step": 3,
    "mime": "image/png",
    "id": "shot_0003"
}
```

**Note**: Screenshot metadata is followed immediately by a **binary frame** containing the PNG image data.

#### Agent Completion
```json
{
    "type": "agent.finish",
    "success": true,
    "reason": "Task completed successfully - reached search results page"
}
```

#### Agent Error
```json
{
    "type": "agent.error",
    "message": "Element not found: #login-button"
}
```

### Screenshot Handling

Screenshots are sent as a two-part message:
1. **Text frame**: JSON metadata with `type: "agent.screenshot"`
2. **Binary frame**: PNG image data

Example client handling:
```javascript
websocket.onmessage = (event) => {
    if (typeof event.data === 'string') {
        const msg = JSON.parse(event.data);
        if (msg.type === 'agent.screenshot') {
            console.log(`Screenshot ${msg.id} for step ${msg.step}`);
            // Next message will be binary with image data
        }
    } else if (event.data instanceof ArrayBuffer) {
        // Handle screenshot binary data
        const imageBlob = new Blob([event.data], { type: 'image/png' });
        // Process or display the image
    }
};
```

### Connection Management
- **Heartbeat**: Server sends ping frames every 25 seconds
- **Timeout**: Connection closed after 3 missed pongs
- **Backpressure**: 128 message buffer per client; oldest messages dropped when full
- **Cleanup**: Automatic unsubscription when client disconnects

## Error Handling

### HTTP Errors
- `400 Bad Request`: Invalid request format or missing required fields
- `404 Not Found`: Session not found
- `500 Internal Server Error`: Server-side error during task execution

### WebSocket Errors
- Connection failures should trigger exponential backoff reconnection
- Parse errors indicate protocol violations
- Timeout errors occur when heartbeat fails

## Example Workflows

### 1. Start Task and Monitor Progress
```
1. POST /api/tasks → get session_id
2. Connect to /ws/agent?session_id=xxx
3. Listen for agent events until agent.finish or agent.error
4. GET /api/tasks/xxx for final status
```

### 2. Interactive Chat Session
```
1. Connect to /ws/chat?session_id=demo
2. Send chat.user_msg with task description
3. Stream chat.delta responses until chat.done
4. Optionally start automation task with the planned approach
```

### 3. Real-time Monitoring with Screenshots
```
1. Start task via REST API
2. Connect to agent WebSocket
3. Display agent events in real-time
4. Save binary screenshot frames as they arrive
5. Show completion status
```

## Security Considerations

- **Authentication**: Required in production. Use `x-api-key`,
  `x-bb-api-key`, `Authorization: Bearer`, or the `?token=` fallback for browser
  WebSocket clients. CDP/live capability tokens are signed, expiring,
  session-bound, and protocol-scoped.
- **Rate limiting**: The production ingress applies connection, request-rate,
  and burst limits; WebX also enforces project browser-time quotas and a global
  concurrent-session ceiling.
- **Origins**: Production CORS is restricted by `WEBX_ALLOWED_ORIGINS`; it does
  not use an allow-all policy.
- **Input validation**: Task size, launch configuration, extension archives,
  proxy credentials, callback targets, and session ownership are validated
  before execution.

## Client Libraries

### Rust
See `examples/websocket/agent_ws_client.rs` and `examples/websocket/chat_ws_client.rs`

### TypeScript/JavaScript
See `clients/ts/` directory for example implementation

### cURL Examples

Start a task:
```bash
curl -X POST http://localhost:8081/api/tasks \
  -H "Content-Type: application/json" \
  -d '{"task": "Navigate to example.com"}'
```

Get task status:
```bash
curl http://localhost:8081/api/tasks/abc123
```

## Configuration

### Server Configuration
- **Port**: Default 8081 (configurable via `--addr` flag)
- **Log Level**: Set `RUST_LOG=webx_server=debug` for detailed logging
- **Environment**: Load `.env` file for configuration

### Client Configuration
- **Reconnection**: Implement exponential backoff (1s, 2s, 4s, 8s, 16s max)
- **Buffering**: Handle message backpressure gracefully
- **Timeouts**: Set appropriate connection and read timeouts