> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gravixlayer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Control the pointer

> Click, type, and use the clipboard on the screen

Click, type, and use the clipboard on the screen. Start the display first (`sandbox.desktop.enable()`). A click lands at the coordinates you pass, or at the current pointer if you omit them.

CLI and cURL examples use `$RT` with the display already started ([Screen](https://docs.gravixlayer.ai/documentation/agentruntime/computer-use/screen.md#start-the-display)).

## Move and click

`click` takes `x`, `y`, an optional `button` (`left`/`middle`/`right`) and
`click_count` (`2` is a double-click); an unset axis keeps the current
pointer position. `down`/`up` press and release a button — between them the
pointer can `move` for a manual drag; `drag` does the same in one call,
interpolating from the current position (or `x`/`y`) to `to_x`/`to_y` over
`steps` inside `duration_ms`. `scroll` sends `dx`/`dy` wheel deltas — the
wheel lands under the pointer, so pass `x`/`y` to position it first (an
omitted axis keeps the current one) or `move` to aim it. `modifiers`
(`["ctrl","shift"]`, …) are held during the action.

```python Python theme={null}
from gravixlayer import GravixLayer

client = GravixLayer()
sandbox = client.runtime.create(template="desktop-small")
sandbox.desktop.enable()

sandbox.desktop.click(640, 420)                    # left click at x,y
sandbox.desktop.click(640, 420, button="right")
sandbox.desktop.double_click(200, 100)             # convenience: click_count=2
sandbox.desktop.move(320, 240)
sandbox.desktop.scroll(dy=-480)                    # wheel delta (dx/dy)
sandbox.desktop.drag(500, 300, x=200, y=200)       # from x,y to to_x,to_y
sandbox.kill()
```

```typescript TypeScript theme={null}
import { GravixLayer } from 'gravixlayer';

const client = new GravixLayer();
const sandbox = await client.runtime.create({ template: 'desktop-small' });
await sandbox.desktop.enable();

await sandbox.desktop.click(640, 420);             // left click at x,y
await sandbox.desktop.click(640, 420, 'right');
await sandbox.desktop.doubleClick(200, 100);       // convenience: clickCount=2
await sandbox.desktop.move(320, 240);
await sandbox.desktop.scroll(0, -480);             // wheel delta (dx, dy)
await sandbox.desktop.drag(500, 300, 'left', { from: { x: 200, y: 200 } });
await sandbox.kill();
```

```bash CLI theme={null}
gravixlayer runtime desktop mouse "$RT" click --x 640 --y 420 --button left
gravixlayer runtime desktop mouse "$RT" click --x 200 --y 100 --click-count 2
gravixlayer runtime desktop mouse "$RT" move --x 320 --y 240
gravixlayer runtime desktop mouse "$RT" scroll --dy=-480    # use = for negative values
gravixlayer runtime desktop mouse "$RT" drag --x 200 --y 200 --to-x 500 --to-y 300
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/mouse/click" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"x":640,"y":420,"button":"left"}'

curl -sS -X POST "$API/agents/runtime/$RT/display/mouse/scroll" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"dx":0,"dy":-480}'

curl -sS -X POST "$API/agents/runtime/$RT/display/mouse/drag" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"x":200,"y":200,"to_x":500,"to_y":300,"button":"left"}'
```

| Route | Body |
| - | - |
| `mouse/click` | `x`, `y`, `button` (default `left`), `click_count`, `modifiers` |
| `mouse/down` / `mouse/up` | `button`, optional `x`, `y`, `modifiers` |
| `mouse/drag` | `to_x`, `to_y`, `button`; optional `x`, `y` (start — defaults to the pointer), `steps`, `duration_ms`, `modifiers` |
| `mouse/move` | `x`, `y` |
| `mouse/scroll` | `dx`, `dy` — optional `x`, `y` position the pointer first; an omitted axis keeps the current one |

Python also exposes `left_click`, `double_click`, `right_click`,
`middle_click`, `mouse_press`, `mouse_release`, and `move_mouse` as named
conveniences; TypeScript has `leftClick`, `doubleClick`, `rightClick`,
`middleClick`, `mousePress`, `mouseRelease`, and `moveMouse`.

**Screenshot coordinates.** `coordinate_space="screenshot"`
(`coordinateSpace: 'screenshot'`, `--coordinate-space screenshot`) maps
coordinates measured in delivered screenshot pixels back to display space —
use it on input calls after a fitted `screenshot(max_width=…, max_height=…)`
so you can click directly on the image you gave the model.

## Type on the screen

`type` sends a string as keystrokes (`delay_ms` spaces them out;
`mode="paste"` stages the text on the clipboard and sends `ctrl+v` —
faster and lossless for large blocks). `press` sends one key name —
`Enter`, `Tab`, `Escape`, `F1`–`F12`, arrows, or a single character;
`hotkey` sends a chord whose modifiers hold while the rest press.
`desktop.keys` lists every accepted name. Python's `write` is `type`
without shadowing the builtin.

```python Python theme={null}
from gravixlayer import GravixLayer

client = GravixLayer()
sandbox = client.runtime.create(template="desktop-small")
sandbox.desktop.enable()

sandbox.desktop.write("user@example.com")          # same as type()
sandbox.desktop.press("Enter")
sandbox.desktop.hotkey(["ctrl", "l"])              # focus the URL bar
sandbox.kill()
```

```typescript TypeScript theme={null}
import { GravixLayer } from 'gravixlayer';

const client = new GravixLayer();
const sandbox = await client.runtime.create({ template: 'desktop-small' });
await sandbox.desktop.enable();

await sandbox.desktop.write('user@example.com');   // same as type()
await sandbox.desktop.press('Enter');
await sandbox.desktop.hotkey(['ctrl', 'l']);       // focus the URL bar
await sandbox.kill();
```

```bash CLI theme={null}
gravixlayer runtime desktop keyboard "$RT" type --value "user@example.com"
gravixlayer runtime desktop keyboard "$RT" type --value "$(cat big.txt)" --mode paste
gravixlayer runtime desktop keyboard "$RT" press --value Enter
gravixlayer runtime desktop keyboard "$RT" hotkey --keys ctrl,l
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/keyboard/type" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"user@example.com"}'

curl -sS -X POST "$API/agents/runtime/$RT/display/keyboard/press" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"key":"Enter"}'

curl -sS -X POST "$API/agents/runtime/$RT/display/keyboard/hotkey" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"keys":["ctrl","l"]}'
```

| Route | Body |
| - | - |
| `keyboard/type` | `text`, optional `delay_ms` between keystrokes, `mode` (`keys` default; `paste` via clipboard) |
| `keyboard/press` | `key` — `Enter`, `Tab`, `Escape`, `F1`–`F12`, arrows, or a single character; optional `modifiers` |
| `keyboard/hotkey` | `keys` — e.g. `["ctrl","shift","t"]`; optional `modifiers` |

## Copy and paste

The desktop clipboard is shared with every window on the display — set text
or a PNG image and pasting inside the sandbox yields it, or read back what
the session put there. `clipboard` accepts `op: "get" | "set"` with a `text`
payload, or `png` (base64) for an image payload. Python returns the image
decoded as `bytes`; TypeScript returns/takes the base64 string.

```python Python theme={null}
from gravixlayer import GravixLayer

client = GravixLayer()
sandbox = client.runtime.create(template="desktop-small")
sandbox.desktop.enable()

sandbox.desktop.set_clipboard("seed text")
print(sandbox.desktop.get_clipboard())       # seed text

png = sandbox.desktop.clipboard_image()      # PNG bytes, b"" when empty
sandbox.desktop.set_clipboard_image(png)     # put it back
sandbox.kill()
```

```typescript TypeScript theme={null}
import { GravixLayer } from 'gravixlayer';

const client = new GravixLayer();
const sandbox = await client.runtime.create({ template: 'desktop-small' });
await sandbox.desktop.enable();

await sandbox.desktop.setClipboard('seed text');
console.log(await sandbox.desktop.getClipboard()); // seed text

const png = await sandbox.desktop.getClipboardImage(); // base64 PNG, "" when empty
await sandbox.desktop.setClipboardImage(png);          // put it back
await sandbox.kill();
```

```bash CLI theme={null}
gravixlayer runtime desktop clipboard "$RT" set --text "seed text"
gravixlayer runtime desktop clipboard "$RT" get
gravixlayer runtime desktop clipboard "$RT" get --png-out clip.png   # image payload
gravixlayer runtime desktop clipboard "$RT" set --png clip.png
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/clipboard" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"op":"set","text":"seed text"}'

curl -sS -X POST "$API/agents/runtime/$RT/display/clipboard" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"op":"get"}'                    # -> {"text": "seed text"} or {"png": "<base64>"}
```

## Take one step

`act` runs a batch of inputs in order, waits for the screen to settle, and
returns the observation — `{settled, rev, screenshot}`. Each action is an
input body with a `kind` — `click`, `down`, `up`, `drag`, `move`, `scroll`,
`type`, `press`, `hotkey`, `keydown`, `keyup`, `set_clipboard`, or `wait` —
either nested
(`{"kind":"click","mouse":{"x":10,"y":20}}`) or the flat per-verb shape the
input methods take (`{"kind":"click","x":10,"y":20,"button":"left"}`);
`{"kind":"wait","ms":300}` pauses the batch. `observe` is a screenshot
options dict (`format`, `region`, `quality`, `scale_pct`, `max_width`,
`max_height`, `cursor`, `if_rev`) — omit it for no image. Feed the returned
`screenshot.rev` back as `observe.if_rev` on the next call to skip unchanged
frames.

```python Python theme={null}
import base64
from gravixlayer import GravixLayer

client = GravixLayer()
sandbox = client.runtime.create(template="desktop-small")
sandbox.desktop.enable()

r = sandbox.desktop.act(
    [
        {"kind": "click", "x": 640, "y": 420},
        {"kind": "type", "text": "user@example.com"},
        {"kind": "press", "key": "Tab"},
        {"kind": "type", "text": "hunter2"},
        {"kind": "press", "key": "Enter"},
        {"kind": "wait", "ms": 300},
    ],
    settle_ms=200,
    timeout_ms=15_000,
    observe={"format": "jpeg", "quality": 70, "max_width": 1024, "if_rev": 0},
)
print(r["settled"], r["rev"])
shot = r.get("screenshot")
if shot and shot["data"]:
    open("step.jpg", "wb").write(base64.b64decode(shot["data"]))
sandbox.kill()
```

```typescript TypeScript theme={null}
import { writeFileSync } from 'node:fs';
import { GravixLayer } from 'gravixlayer';

const client = new GravixLayer();
const sandbox = await client.runtime.create({ template: 'desktop-small' });
await sandbox.desktop.enable();

const r = await sandbox.desktop.act({
  actions: [
    { kind: 'click', x: 640, y: 420 },
    { kind: 'type', text: 'user@example.com' },
    { kind: 'press', key: 'Tab' },
    { kind: 'type', text: 'hunter2' },
    { kind: 'press', key: 'Enter' },
    { kind: 'wait', ms: 300 },
  ],
  settleMs: 200,
  timeoutMs: 15_000,
  observe: { format: 'jpeg', quality: 70, max_width: 1024 },
});
console.log(r.settled, r.rev);
if (r.screenshot?.data) {
  writeFileSync('step.jpg', Buffer.from(r.screenshot.data, 'base64'));
}
await sandbox.kill();
```

```bash CLI theme={null}
gravixlayer runtime desktop act "$RT" --actions '[
  {"kind":"click","x":640,"y":420},
  {"kind":"type","text":"user@example.com"},
  {"kind":"press","key":"Tab"},
  {"kind":"type","text":"hunter2"},
  {"kind":"press","key":"Enter"}
]' --settle-ms 200 --timeout-ms 15000 \
  --observe '{"format":"jpeg","quality":70,"max_width":1024}'
# --actions also takes a JSON file path, or "-" for stdin
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/act" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      {"kind":"click","x":640,"y":420},
      {"kind":"type","text":"user@example.com"},
      {"kind":"press","key":"Tab"},
      {"kind":"type","text":"hunter2"},
      {"kind":"press","key":"Enter"}
    ],
    "settle_ms": 200,
    "timeout_ms": 15000,
    "observe": {"format":"jpeg","quality":70,"max_width":1024}
  }' | jq '{settled, rev, has_shot: (.screenshot.data != null)}'
```

| Parameter | Type | Required | Description |
| - | - | - | - |
| `actions` | array | Yes | Input bodies in order — `kind` decides which fields apply |
| `settle_ms` | int | No | Extra pixel-quiet time required after the last change |
| `timeout_ms` | int | No | Overall budget for actions + settle |
| `observe` | object | No | Observation screenshot options (`if_rev` skips unchanged frames); omit for no image |

## Type a long string

* Coordinates are absolute display pixels; `keys` lists the exact names
  `press`/`hotkey` accept.
* `act` is the tight loop primitive — one request instead of an input call,
  a `wait_stable`, and a `screenshot`. Reuse `rev` between calls so
  unchanged frames transfer nothing.
* `type` sends synthetic keystrokes — prefer `mode="paste"` for long text;
  it is faster and immune to keyboard-layout edge cases.

## Endpoints

| Method | Path | Description |
| - | - | - |
| POST | `/v1/agents/runtime/{runtime_id}/display/mouse/click` | Click at x,y |
| POST | `/v1/agents/runtime/{runtime_id}/display/mouse/down` | Press a button |
| POST | `/v1/agents/runtime/{runtime_id}/display/mouse/up` | Release a button |
| POST | `/v1/agents/runtime/{runtime_id}/display/mouse/drag` | Drag between points |
| POST | `/v1/agents/runtime/{runtime_id}/display/mouse/move` | Move the pointer |
| POST | `/v1/agents/runtime/{runtime_id}/display/mouse/scroll` | Scroll the wheel |
| POST | `/v1/agents/runtime/{runtime_id}/display/keyboard/type` | Type a string |
| POST | `/v1/agents/runtime/{runtime_id}/display/keyboard/press` | Press one key (`Enter`, `Tab`, `F5`…) |
| POST | `/v1/agents/runtime/{runtime_id}/display/keyboard/hotkey` | Press a chord (`ctrl`, `l`) |
| POST | `/v1/agents/runtime/{runtime_id}/display/clipboard` | Get or set clipboard text / image |
| POST | `/v1/agents/runtime/{runtime_id}/display/act` | Batched agent step |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.