# WebX — User Guide

WebX is a verified browser-automation service: you give it a natural-language task, it drives a real Chrome browser with an LLM agent, validates its own work with an independent LLM judge, and streams every step to you over WebSocket. It also offers a browser-as-a-service Sessions API.

This guide documents the **live deployment** at:

> **Base URL:** `https://webx.agentslab.host`
> **WebSocket:** `wss://webx.agentslab.host`

Everything below — every response body, event payload, and screenshot — was captured from this deployment on 2026-06-12.

---

## 1. Architecture

![Architecture](images/architecture.svg)

The service runs on a microk8s cluster as a single `webx-server` pod (Rust/axum). With `WEBX_ORCHESTRATOR=process`, it launches one isolated Chrome process per session from the pinned browser bundled in the server image; there is no shared Chrome sidecar. TLS is terminated by the NGINX ingress with an auto-renewing Let's Encrypt certificate. Planning and validation use the configured OpenAI-compatible model endpoint.

---

## 2. Authentication

All `/api/*`, `/ws`, `/live`, and `/connect` routes require an API key. Only
`GET /health` and `GET /ready` are public (the Kubernetes liveness/readiness
probes).

Three ways to present the key, checked in this order:

| Method | Example |
|---|---|
| `x-api-key` header (recommended) | `curl -H "x-api-key: $WEBX_TOKEN" …` |
| `Authorization: Bearer` | `curl -H "Authorization: Bearer $WEBX_TOKEN" …` |
| `?token=` query parameter | `wss://…/ws?session_id=…&token=$WEBX_TOKEN` |

The query form exists because browser `WebSocket` clients cannot set headers. Keys are compared in constant time; per-project keys can be minted via `POST /api/keys` when a durable store is configured.

Without a valid key you get `401`:

```console
$ curl -s -o /dev/null -w "%{http_code}" https://webx.agentslab.host/api/engines
401
```

> The master token for this deployment is stored on the server at `/root/webx-api-token` (not in git). Rotate it by regenerating the `webx-env` Secret — see §9.

---

## 3. Quickstart

```bash
TOKEN=$(ssh -i ~/.ssh/poc_server_new root@65.108.78.80 \
  'microk8s kubectl -n webx get secret webx-env -o jsonpath="{.data.WEBX_API_TOKEN}" | base64 -d')

# 1. Is it up?
curl https://webx.agentslab.host/health
# OK

# 2. What engines are available?
curl -s -H "x-api-key: $TOKEN" https://webx.agentslab.host/api/engines | jq
```

```json
[
  {
    "name": "chromium",
    "supports_screencast": true,
    "supports_fetch_auth": true,
    "supports_screenshot": true,
    "supports_pdf": true,
    "supports_proxy": true,
    "supports_persistent_context": true,
    "supports_extensions": true,
    "supports_selenium": true,
    "supports_real_browser_fingerprint": true,
    "supports_long_sessions": true,
    "stateless": false,
    "perception_modes": ["webmcp", "axtree", "dom", "pixels"],
    "cost_score": 100,
    "cold_start_ms": 2500
  }
]
```

```bash
# 3. Run an agent task
curl -s -X POST -H "x-api-key: $TOKEN" -H "Content-Type: application/json" \
  -d '{"task": "Go to https://example.com and report the main heading text on the page."}' \
  https://webx.agentslab.host/api/tasks
# {"session_id":"c2910342-9963-4575-b9a7-7368bea9c165","status":"starting"}

# 4. Poll until terminal
curl -s -H "x-api-key: $TOKEN" \
  https://webx.agentslab.host/api/tasks/c2910342-9963-4575-b9a7-7368bea9c165 | jq
```

```json
{
  "session_id": "c2910342-9963-4575-b9a7-7368bea9c165",
  "status": "completed",
  "task": "Go to https://example.com and report the main heading text on the page.",
  "created_at": "2026-06-12T21:06:19.013409031Z",
  "completed_at": "2026-06-12T21:06:27.629785460Z"
}
```

This task completed in **8 seconds, 1 agent step**: the server seed-navigates to the URL named in the task, the LLM reads the page and answers, and the validator (a second, independent LLM pass) confirms the answer before the run is marked successful.

---

## 4. Agent task lifecycle

![Task flow](images/task-flow.svg)

### 4.1 Start a task

```
POST /api/tasks
{"task": "<natural language instruction>", "session_id": "<optional, reuse an id>"}
→ {"session_id": "…", "status": "starting"}
```

For asynchronous automation, attach a per-run callback and your own correlation
id:

```json
{
  "task": "Go to https://example.com and report the main heading.",
  "callback": "https://automation.example.com/webhooks/webx",
  "workflow_id": "order-42/run-7"
}
```

The callback is sent only after the terminal status and evidence are persisted.
Its JSON includes `event`, `session_id`, `task_id`, `workflow_id`, `status`,
`task`, `answer`, and timing. WebX retries a failed delivery twice and records
every attempt as a `webhook_delivery` audit row. Callback URLs must use HTTPS,
cannot contain credentials or fragments, cannot resolve to private/reserved
networks, are DNS-pinned for delivery, and cannot redirect. When
`WEBX_CALLBACK_SIGNING_SECRET` is configured, WebX adds `X-WebX-Token`,
`X-WebX-Timestamp`, and `X-WebX-Signature: sha256=<HMAC-SHA256 of the exact
body>`.

### 4.2 Watch it live over WebSocket

Connect to the unified socket and you receive every agent event in real time:

```python
import asyncio, json, websockets

async def watch(sid, token):
    url = f"wss://webx.agentslab.host/ws?session_id={sid}&token={token}"
    async with websockets.connect(url) as ws:
        while True:
            e = json.loads(await ws.recv())
            print(e["type"], e.get("message", e.get("url", "")))
            if e["type"] == "agent.finish":
                return e
```

Real event sequence captured from this deployment:

```json
{"type": "connection.ready", "session_id": "0b7893e3-…", "timestamp": "2026-06-12T21:04:24.155845881+00:00"}
{"type": "agent.started",  "session_id": "0b7893e3-…", "task": "Go to https://example.com and report the main heading text on the page."}
{"type": "agent.observe",  "session_id": "0b7893e3-…", "url": "about:blank"}
{"type": "agent.action",   "session_id": "0b7893e3-…", "action": {"type": "navigate", "url": "https://example.com"}}
{"type": "agent.result",   "session_id": "0b7893e3-…", "success": true, "message": "Navigated to 'https://example.com'"}
{"type": "agent.finish",   "session_id": "0b7893e3-…", "success": true, "message": "Real WebX agent execution completed - 1 steps, duration: 11s"}
```

Event types you will see:

| Event | Meaning |
|---|---|
| `connection.ready` | Socket accepted; events for your `session_id` will follow |
| `agent.started` | Task accepted, agent loop starting |
| `agent.observe` | One perception pass (current URL attached) |
| `agent.action` | An action the agent is executing (navigate, click, fill, …) |
| `agent.result` | Outcome of an action or of the whole run (`success` flag) |
| `agent.finish` | Terminal event with step count and duration |

### 4.3 Stop a task

```
POST /api/tasks/{session_id}/stop
```

Cancellation is cooperative: the agent stops at the next step boundary and the browser is torn down immediately.

---

## 5. Live view — watch the browser as the agent works

`GET /live?sessionId=…&token=…` upgrades to a WebSocket that streams the session's screen.

**Protocol:** one JSON text preamble, then **binary JPEG frames** (one message per frame):

```json
{"format": "jpeg", "sessionId": "c2910342-…", "type": "live"}
```

```python
async with websockets.connect(f"wss://webx.agentslab.host/live?sessionId={sid}&token={token}") as ws:
    preamble = json.loads(await ws.recv())          # text frame
    while True:
        frame = await ws.recv()                     # bytes = one JPEG image
        if isinstance(frame, bytes):
            display(frame)
```

A frame captured from a real run, while the agent was reading example.com:

![Live view screenshot — example.com rendered in an isolated deployed Chrome process](images/screenshot-liveview.jpg)

Frames are sent on page damage, so a static page produces few frames — render the latest one you have.

---

## 5.5 Playground / admin portal

`https://webx.agentslab.host/admin` — a built-in playground + operator console (served by the server itself; the page is a static shell and every data call requires either your API key or a tenant-scoped browser identity session established through OIDC/SAML).

![Playground — light theme](images/screenshot-admin.png)

Light and dark themes (toggle in the header, follows your OS preference by default, `?theme=dark|light` to force):

![Playground — dark theme](images/screenshot-admin-dark.png)

What it does:

- **Playground composer**: type a task (or pick an example chip), hit Run, and watch the browser pane and the live agent-event stream side by side
- **Sessions table** (auto-refreshing, backed by the durable store so history survives restarts): status, task, timings; filter by status
- **Watch** any running session — embeds the `/live` JPEG screencast
- **Stop** a running session (cooperative cancel, immediate browser teardown)
- **Playback / logs tab** — the per-session recorder's network + console timeline for any session, including finished ones
- **Receipts tab** — the session's hash-chained authorization ledger
- **Attribution tab** — typed failure cause for failed runs
- **Pending confirmations** — approve/deny membrane-gated actions awaiting an operator
- Deep-link with `?session=<id>` to open a session's detail directly; header links to the Grafana dashboard

![Admin portal — session detail with live view and log playback](images/screenshot-admin-detail.png)

---

## 5.7 One-shot Fetch API

`POST /api/fetch` renders a URL in a real, JS-capable browser and returns it in the formats you ask for — no session to manage. This is the stateless "give me this page" primitive (like Hyperbrowser Fetch / Steel `/scrape`).

```bash
curl -s -X POST -H "x-api-key: $TOKEN" -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["text","title","links","screenshot"]}' \
  https://webx.agentslab.host/api/fetch
```

```json
{
  "url": "https://example.com/",
  "title": "Example Domain",
  "text": "Example Domain This domain is for use in documentation examples…",
  "links": [{"url": "https://www.iana.org/domains/example", "text": "More information..."}],
  "screenshot_base64": "iVBORw0KGgo…"
}
```

| Field | Meaning |
|---|---|
| `url` (required) | Absolute `http(s)` URL to render |
| `formats` | Any of `text`, `html`, `links`, `title`, `screenshot`. Omitted → `text`+`links`+`title` |
| `wait_ms` | Extra settle time after load for JS-heavy pages (capped at 15s) |

A fetch provisions a short-lived isolated browser, drives it, and tears it down within the request, so it shares the global concurrency bound (`429` when at capacity).

## 5.8 Multi-tenant projects, API keys, and isolation

- `POST /api/keys` `{ "project_id": "acme" }` → issues a scoped key (`wx_…`, shown once). Gated behind the master token.
- `POST /api/keys/revoke` `{ "api_key": "wx_…" }` (or `{ "key_hash": "…" }`) → revokes a key immediately. Rotation = revoke + issue.
- **Isolation is enforced**: a project key only sees sessions created under its own project. `GET /api/sessions`, `/api/sessions/:id`, `/logs`, `/receipts`, `/attribution`, `/spend`, stop, and release are all scoped — a cross-tenant id returns `404` (indistinguishable from "absent"). The **master token is admin** and sees every project.

## 5.9 Enterprise identity lifecycle

- **OIDC**: `GET /api/identity/oidc/:project/start` starts an auth-code flow with state, nonce, PKCE, and redirect allowlisting. `GET /api/identity/oidc/:project/callback` validates issuer/audience/signature before creating a short-lived `HttpOnly; Secure; SameSite=Lax` browser session.
- **OIDC secrets**: tenant OIDC `client_secret` values are envelope-encrypted at rest with `WEBX_EVIDENCE_ENCRYPTION_KEY` under an `identity:<project>` scope. The config APIs and admin UI only return a stable redacted marker plus a configured flag; resubmitting that marker (or leaving the field blank) preserves the stored secret.
- **SAML**: `GET /api/identity/saml/:project/metadata` serves SP metadata, `GET /api/identity/saml/:project/login` starts the signed AuthnRequest flow, and `POST /api/identity/saml/:project/acs` validates the signed response/assertion, destination/audience/`InResponseTo`, and time windows before creating the same browser session type.
- **SCIM**: `POST /api/identity/projects/:project/scim/tokens` mints a tenant-scoped provisioning bearer token (stored hashed). Use it against `/scim/v2/:project/Users` and `/scim/v2/:project/Groups` for idempotent provisioning, PATCH-based lifecycle updates, and deprovisioning that revokes browser sessions (and any identity-bound API keys).
- **Console support**: the **Identity** tab in `/admin` edits tenant OIDC/SAML/claims settings, shows live user/group/session/token counts, and exposes recent identity audit events plus one-shot SCIM token creation.

## 5.9 Observability, replay, and human-in-the-loop

| Endpoint | What it gives you |
|---|---|
| `GET /api/sessions/:id/trajectory` | The agent's step timeline — the persisted observe/act/result events, in order (the "what did the agent do" trace) |
| `GET /api/sessions/:id/replay` | Captured screencast frames (base64 JPEG, oldest-first) for post-run replay |
| `GET /replay/:id` | A standalone HTML **replay player** (scrub, play, speed) — open `https://webx.agentslab.host/replay/<session>?token=<key>` |
| `GET /api/sessions/:id/logs` | Per-session network + console capture (filter with `?kind=`) |

**Replay** is captured from the live screencast, so any session you watch (the playground opens `/live` automatically when a task runs) is recorded and replayable afterward. Frames are retained past the run; memory is bounded by keeping the most recent 64 sessions.

**Human takeover** — `/live` is bidirectional. Send a JSON text frame on the same socket to drive the page:

```json
{"type": "mouse", "action": "click", "x": 420, "y": 230}
{"type": "scroll", "x": 200, "y": 200, "deltaY": 300}
{"type": "key", "text": "hello"}
{"type": "key", "key": "Enter"}
```

Mouse coordinates are page pixels. In the playground, click **Take over** on the Browser pane and your clicks/keystrokes are sent through — for logins, 2FA, or captchas the agent can't handle.

## 5.10 Usage metering and quotas

| Endpoint | What it does |
|---|---|
| `GET /api/billing/usage` | Per-project browser-time totals (`browser_seconds`, `browser_hours`, session count). Master sees all projects; a project key sees only its own |
| `POST /api/billing/quota` `{ "project_id": "acme", "max_browser_hours": 10 }` | Set (or clear with `null`) a project's browser-hours cap (master only) |

Browser time is metered on every session's finalize. A project at/over its cap is rejected at session create with `429 quota_exceeded`.

## 6. Sessions API — browser-as-a-service

![Session flow](images/session-flow.svg)

For when you want the browser without the agent:

```
POST /api/sessions
{"timeout_ms": 120000, "keep_alive": false, "headless": true,
 "viewport_width": 1280, "viewport_height": 800, "user_agent": null, "proxy_server": null}
```

Real response:

```json
{
  "session_id": "2f4484f1-21a2-47e4-80fc-b14320109d36",
  "status": "running",
  "connect_path": "/connect?sessionId=2f4484f1-21a2-47e4-80fc-b14320109d36",
  "cdp_ws_url": "ws://localhost:9223/devtools/page/87D28712…",
  "browser_ws_url": "ws://localhost:9223/devtools/browser/635371e2-…"
}
```

| Endpoint | Purpose |
|---|---|
| `POST /api/sessions` | Create a browser session |
| `GET /api/sessions` | List sessions |
| `GET /api/sessions/:id` | Session detail |
| `POST /api/sessions/:id/release` | Release the browser and finalize the session |
| `GET /api/sessions/:id/logs` | Per-session structured log entries |
| `GET /api/sessions/:id/attribution` | Typed failure cause (`webx why`) for a *terminal failed* session |

### 6.1 `/connect` (raw CDP relay) and orchestrator tiers

✅ **On this deployment, `/connect` is available** because
`WEBX_ORCHESTRATOR=process`. Each session owns an isolated Chrome process, so
the authenticated session owner may attach a raw CDP client directly with the
`connect_path` returned by `POST /api/sessions`.

Deployments that run the shared-browser context tier deliberately reject
`/connect` with `409`:

```json
{
  "error": "gateway_requires_process_tier",
  "message": "Raw CDP relay is only available under the process orchestrator tier (set WEBX_ORCHESTRATOR=process). The shared-browser tier cannot expose an isolated browser-level CDP endpoint."
}
```

That is an isolation guard, not a fault. A context-tier deployment keeps many
sessions inside one shared Chrome, so exposing the browser-level CDP socket
would leak `Target.*`/`Browser.*` access across tenants. Use `/live` (§5) for
visuals under the context tier.

---

## 7. Verified execution: receipts, spend, confirmations

Every session carries a tamper-evident, hash-chained receipt ledger:

```bash
curl -s -H "x-api-key: $TOKEN" https://webx.agentslab.host/api/sessions/$SID/receipts | jq
```

```json
{
  "mandate_id": "c2910342-9963-4575-b9a7-7368bea9c165",
  "budget_limit_minor": 0,
  "entries": [],
  "verified": true,
  "error": null
}
```

`verified: true` means the chain re-verified on read. Related endpoints:

| Endpoint | Purpose |
|---|---|
| `GET /api/sessions/:id/receipts` | Hash-chained ledger of approved gated actions |
| `POST /api/sessions/:id/spend` | Budget-bounded spend under the session mandate (`{"amount_minor": 500, "payload": {…}}`); enforced against `WEBX_BUDGET_MINOR`, recomputed from the chain on restart |
| `GET /api/confirmations` | Pending operator confirmations for membrane-gated actions |
| `POST /api/confirmations/:id` | `{"approve": true|false}` — the blocked agent resumes with the decision |

Plus operational insight endpoints:

| Endpoint | Purpose |
|---|---|
| `GET /api/engines` | Engine capability profiles |
| `GET /api/insights/config` | Run-config bandit insights |
| `POST /api/contexts` / `GET /api/contexts` / `DELETE /api/contexts/:id` | Persistent browser profiles |
| `POST /api/keys` | Issue a per-project API key (returns the raw key **once**) |

Browser-session create responses include `engine`, `routing_reason`, and
redacted `routing_attempts`. Capability requirements supplied by the caller are
merged with proxy/profile/extension/isolation requirements inferred from the
launch configuration. See [multi-engine routing](MULTI_ENGINE_ROUTING.md).

---

## 8. Error reference

| Status | When | What to do |
|---|---|---|
| `401 {"error":"unauthorized"}` | Missing/invalid API key | Send the key via `x-api-key`, Bearer, or `?token=` |
| `400 missing sessionId` | `/live` or `/connect` without `sessionId` | Add the query param |
| `404 unknown session` | Session released, timed out, or never existed | Sessions expire after their `timeout_ms` (default 10 min) |
| `404` on `/attribution` | Session is not terminal-failed | Attribution is computed for failed runs |
| `409 gateway_requires_process_tier` | `/connect` on a shared-browser/context-tier deployment | Use `/live`, or redeploy with `WEBX_ORCHESTRATOR=process` |
| `500 internal_error` | Unhandled failure starting a task | Check pod logs (§9) |

Task statuses: `starting → running → completed | failed | cancelled | timed_out`.

---

## 9. Operations (this deployment)

**Where things live on the server** (`ssh -i ~/.ssh/poc_server_new root@65.108.78.80`):

| Artifact | Path |
|---|---|
| Source tree (rsynced) | `/root/webx-src` |
| Image build log | `/root/webx-build.log` |
| K8s manifest | `/root/webx-single-node.yaml` |
| Master API token | Kubernetes Secret `webx-env`, key `WEBX_API_TOKEN` |

**Supported container architecture:** WebX release images are `linux/amd64`
only. The pinned Chrome-for-Testing `linux64` archives and checksum-verified
Debian `_amd64.deb` libraries are x86_64; Linux ARM64 is not supported. Both
release Dockerfiles pin every stage, so ARM64 Docker Desktop builds use amd64
emulation instead of mixing architectures.

**Deploy a new version:**

```bash
REV="$(git rev-parse --short=7 HEAD)"
IMAGE_TAG="localhost:32000/webx-server:${REV}-fullchrome"
docker build --platform linux/amd64 -f Dockerfile.ext-engine \
  -t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
IMAGE="$(docker image inspect "${IMAGE_TAG}" --format '{{index .RepoDigests 0}}')"
WEBX_RELEASE_REVISION="${REV}" WEBX_RELEASE_IMAGE="${IMAGE}" \
  ops/render-webx-single-node.sh > /root/webx-single-node.yaml
microk8s kubectl apply -f /root/webx-single-node.yaml
microk8s kubectl -n webx rollout status deploy/webx-server
```

**Logs / status:**

```bash
microk8s kubectl -n webx get pods
microk8s kubectl -n webx logs deploy/webx-server -c webx-server --tail=100
```

**Configuration** comes from the `webx-env` and `webx-infrastructure` Secrets
plus explicit non-secret deployment settings. `webx-env` must carry
`WEBX_API_TOKEN`, `WEBX_EVIDENCE_ENCRYPTION_KEY`,
`WEBX_RECEIPT_SIGNING_KEY_ID`, `WEBX_RECEIPT_SIGNING_KEY`, and the provider
credentials. The current process tier sets `WEBX_ORCHESTRATOR=process` and uses
the full Chrome-for-Testing binary at `/opt/chrome/chrome-linux64/chrome`; this
is required for extension sessions. `WEBX_CDP_HTTP` is only used by the
shared-browser tier.

Other knobs the server honors: `WEBX_ORCHESTRATOR`, `WEBX_MAX_CONCURRENT_SESSIONS`, `WEBX_SESSION_TIMEOUT_MS`, `WEBX_RUN_MAX_STEPS`, `WEBX_RUN_MAX_TIMEOUT_SECONDS`, `WEBX_BUDGET_MINOR`, `WEBX_MEMBRANE`, `WEBX_CONFIRM_*`, `WEBX_DB_URL`, `WEBX_ALLOWED_ORIGINS`, `WEBX_ENGINES`, `WEBX_PROFILES_DIR`, `WEBX_LOG_*`, `WEBX_DISCONNECT_GRACE_MS`, `WEBX_CALLBACK_SIGNING_SECRET`, `WEBX_DISK_CHECK_PATH`, `WEBX_DISK_MIN_FREE_BYTES`, `WEBX_DISK_MIN_FREE_PERCENT`, `WEBX_WORKFLOW_MEMORY`, `WEBX_DRIFT_JUDGE`, `WEBX_ATTRIBUTION_JUDGE`.

Run requests may select a model on the effective provider, but a per-run
provider switch is rejected because the request does not carry provider
credentials. Per-run USD budgets are also rejected until model-token metering
is wired end to end. Use the enforced `max_steps` and `timeout_seconds` fields;
the deployment ceilings above prevent callers from disabling or inflating
those guards.

**Rotate the API token:**

```bash
NEW=$(openssl rand -hex 24); echo -n "$NEW" > /root/webx-api-token
microk8s kubectl -n webx patch secret webx-env -p "{\"stringData\":{\"WEBX_API_TOKEN\":\"$NEW\"}}"
microk8s kubectl -n webx rollout restart deploy/webx-server
```

**Server quirk:** outbound port 80 is blocked on this host. The full-Chrome
Dockerfile uses verified HTTPS downloads with pinned checksums and copies the
CA bundle into the runtime image. Keep those constraints if you edit it.

**Persistence — everything is on the `webx-data` PVC (`/data`, microk8s-hostpath), durable across pod restarts:**

| Artifact | Where | Backed by |
|---|---|---|
| Sessions, logs, agent trajectory, receipts, usage, quotas, API keys, workflow memory | `/data/webx.db` (`WEBX_DB_URL=sqlite:///data/webx.db`) | SQLite, 6 tables |
| Session **replay** frames | `/data/artifacts/replay/<session>/NNNNNN.jpg` (`WEBX_ARTIFACTS_DIR`) | files on the PVC (off-heap, survive restart; oldest 128 sessions retained) |
| Agent **screenshots** | `/data/screenshots` (`WEBX_SCREENSHOT_DIR`) | files on the PVC |
| Persistent **context profiles** | `/data/profiles` (`WEBX_PROFILES_DIR`) | files on the PVC (real Chrome `--user-data-dir`) |

`GET /api/tasks/:id` falls back to the store for sessions that predate the current pod, and replay survives a pod restart (verified). The deployment uses `strategy: Recreate` so two pods never share the volume. Going multi-replica would mean moving the DB to the cluster's PostgreSQL and the blobs to the cluster's MinIO (S3) — the artifact paths are env-configurable for exactly that.

**Metrics & dashboards:**
- The server exposes Prometheus metrics on a side-channel listener (`WEBX_METRICS_ADDR`, default `0.0.0.0:9091`) — in-cluster only via the `webx-metrics` ClusterIP Service, never through the public ingress.
- Metrics: `webx_http_requests_total{method,path,status}`, `webx_http_request_duration_seconds{path}`, `webx_tasks_started_total`, `webx_tasks_finished_total{status}`, `webx_active_sessions`.
- The cluster Prometheus (`monitoring` ns) scrapes job `webx` every 15s; the **WebX Overview** dashboard lives in the existing Grafana at `https://dash.agentslab.host` (request rate, p95 latency, task outcomes, active sessions, 5xx, scrape health).

**Known limitations:**
- Single replica (SQLite store requires it — see above).
- Prometheus alert rules cover availability, server-error ratio, capacity,
  latency, and outbound callback/integration failures. An operator-owned
  Alertmanager receiver is still required for paging.

---

## 10. Verification record

The full consumer-side verification (scripts + captured outputs) lives in [`docs/verification/`](verification/):

| File | What it is |
|---|---|
| `consumer_test3.py` | Final end-to-end consumer test (task + live capture + policy checks) |
| `ws_events.json` | Real captured WebSocket event stream |
| `verification_report.json` | Machine-readable summary of the final verified run |

Verified on 2026-06-12 against `https://webx.agentslab.host`:

- ✅ `GET /health` → `200 OK` (unauthenticated, used by k8s probes)
- ✅ Auth gating: `401` without key; `x-api-key`, Bearer, and `?token=` all accepted
- ✅ `POST /api/tasks` → agent navigated, answered "Example Domain", validator confirmed — **completed in 8 s / 1 step**
- ✅ REST status reaches `completed` with `completed_at` set
- ✅ `/ws` event stream: `connection.ready → agent.started → agent.observe → agent.action → agent.result → agent.finish`
- ✅ `/live` JPEG screencast (screenshot above is a real captured frame)
- ✅ `GET /api/sessions/:id/receipts` → `verified: true`
- ✅ `/connect` accepted under the process tier with the signed session URL
- ✅ TLS: Let's Encrypt cert, auto-renewed by cert-manager (expires 2026-09-10, renews ~30 days prior)