> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gravixlayer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Human in the loop (HITL)

> Watch the live screen together, take control from the agent, and hand it back

A live view is a shared session, not a private feed. Every connected client — agent and humans — appears in a **presence roster** with a labeled cursor, a **control mode** decides whose input lands, and a takeover is one call away on both the desktop and the browser.

CLI and cURL examples use `$RT` for a `desktop-small` runtime and `$BID` for a browser ID ([Screen](https://docs.gravixlayer.ai/documentation/agentruntime/computer-use/screen.md), [Browser](https://docs.gravixlayer.ai/documentation/agentruntime/browser-use/browser.md)).

## Who is watching

A view connection's identity comes from its token and its URL parameters:

* **Role** — the `browser_url` control token connects as `control`; a `watch`-role token (`vt=`, minted by `view`/`view_url` with `role="watch"`) connects as `watch` and can never inject input.
* **`name=`** — the label rendered next to the cursor (form-urlencoded, 32 chars max).
* **`color=`** — the cursor's CSS color, e.g. `%23ff0055` for `#ff0055` (form-urlencoded, 16 chars max).
* **`agent=1`** — marks the entry as an agent-side cursor.

Appended to a watch URL: `&vt=<view_token>&name=Ope&color=%23ff0055`.

The platform agent holds the reserved entry `_agent` — name `Agent`, color `#008145` — and its cursor tracks real dispatched input, not a viewer's guess, so every watcher sees what the agent is actually pointing at.

## What the socket sends

Video frames are binary; presence and control travel as JSON text frames on the same socket:

```json theme={null}
{"t":"hello","you":"v1","role":"control","presence":[{"id":"v1","name":"Ope","color":"#ff0055","agent":false,"x":400,"y":300,"p":false}]}
{"t":"presence","op":"join","peer":{"id":"v2","name":"Rae","color":"#1e6fff","agent":false,"x":0,"y":0,"p":false}}
{"t":"presence","op":"cursor","id":"v2","x":512,"y":288,"p":false}
{"t":"presence","op":"cursor","id":"_agent","x":640,"y":120,"p":true}
{"t":"control","mode":"human"}
{"t":"presence","op":"leave","id":"v2"}
```

A new client gets `hello` first — its id, its role, and the full roster — then the replayed `control` mode, then live `join`/`cursor`/`leave` traffic. Its own input doubles as its cursor report, so watch-role cursors broadcast too.

## Take over, hand back

`view_control` sets who may inject input. The status calls report the mode and the connected roster (`control_mode`, `viewers`) so an orchestrator can tell a human is watching without being a viewer.

| Mode | Agent input | Control-token viewer input |
| - | - | - |
| `shared` (default) | injected | injected |
| `human` | **rejected** (`400` — reads still work) | injected |
| `agent` | injected | dropped at the socket |

Take over with `human`, hand back with `shared` (or `agent` to lock viewers out). A rejected agent call fails immediately — it does not queue — so an agent should catch the error and retry once the mode changes back.

```python Python theme={null}
from gravixlayer import GravixLayer

client = GravixLayer()
sandbox = client.runtime.create(template="desktop-small")
sandbox.desktop.enable()

# a human clicks while the agent is paused
sandbox.desktop.set_control_mode("human")

st = sandbox.desktop.status()
print(st["control_mode"], st["viewers"])  # human [{id, name, color, agent, x, y, p}, ...]

# hand the session back
sandbox.desktop.set_control_mode("shared")
sandbox.kill()
```

```typescript TypeScript theme={null}
import { GravixLayer } from 'gravixlayer';

const client = new GravixLayer();
const sandbox = await client.runtime.create({ template: 'desktop-small' });
await sandbox.desktop.enable();

// a human clicks while the agent is paused
await sandbox.desktop.viewControl('human');

const st = await sandbox.desktop.status();
console.log(st.controlMode, st.viewers); // human [{id, name, color, agent, x, y, p}, ...]

// hand the session back
await sandbox.desktop.viewControl('shared');
await sandbox.kill();
```

```bash CLI theme={null}
gravixlayer runtime desktop view-control "$RT" human
gravixlayer runtime desktop status "$RT"        # control_mode + viewers
gravixlayer runtime desktop view-control "$RT" shared
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/view/control" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"mode":"human"}'

curl -sS "$API/agents/runtime/$RT/display/status" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" | jq '{control_mode, viewers}'
```

## Same contract on the browser

The browser plane is identical: `view` mints a live URL for a browser's tab, presence and takeover work the same way, and `browser.status` exposes `control_mode`/`viewers`. Under `human`, agent **actions** (click, type, navigate…) reject while **reads** (`evaluate`, screenshots, DOM queries) still return — a human can drive while the agent keeps observing.

```python Python theme={null}
sandbox.browser.open(url="https://example.com")
bid = "b1"
sandbox.browser.set_control_mode(bid, "human")
sandbox.browser.status(bid)          # {"control_mode": "human", "viewers": [...]}
sandbox.browser.set_control_mode(bid, "shared")
```

```typescript TypeScript theme={null}
const info = await sandbox.browser.open({ url: 'https://example.com' });
const bid = info.browserId;
await sandbox.browser.viewControl(bid, 'human');
console.log(await sandbox.browser.status(bid)); // { controlMode: 'human', viewers: [...] }
await sandbox.browser.viewControl(bid, 'shared');
```

```bash CLI theme={null}
gravixlayer runtime browser view-control "$RT" human --browser "$BID"
gravixlayer runtime browser view-control "$RT" shared --browser "$BID"
```

```bash cURL theme={null}
curl -sS -X POST "$API/agents/runtime/$RT/browser/$BID/view/control" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"mode":"human"}'
```

## Endpoints

| Method | Path | Description | | |
| - | - | - | - | - |
| POST | `/v1/agents/runtime/{runtime_id}/display/view` | Mint a live-watch URL (\`role=control | watch\`) | |
| POST | `/v1/agents/runtime/{runtime_id}/display/view/control` | Set takeover mode (\`shared | human | agent\`) |
| GET | `/v1/agents/runtime/{runtime_id}/display/status` | `control_mode` + `viewers` roster | | |
| POST | `/v1/agents/runtime/{runtime_id}/browser/{bid}/view` | Mint a live-watch URL for a browser tab | | |
| POST | `/v1/agents/runtime/{runtime_id}/browser/{bid}/view/control` | Set takeover mode | | |
| GET | `/v1/agents/runtime/{runtime_id}/browser/{bid}/status` | `control_mode` + `viewers` roster | | |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.