Skip to main content
Click, type, and use the clipboard on the screen. Start the display first (sandbox.desktop.enable()). A click lands at the coordinates you pass, or at the current pointer if you omit them. CLI and cURL examples use $RT with the display already started (Screen).

Move and click

click takes x, y, an optional button (left/middle/right) and click_count (2 is a double-click); an unset axis keeps the current pointer position. down/up press and release a button — between them the pointer can move for a manual drag; drag does the same in one call, interpolating from the current position (or x/y) to to_x/to_y over steps inside duration_ms. scroll sends dx/dy wheel deltas — the wheel lands under the pointer, so pass x/y to position it first (an omitted axis keeps the current one) or move to aim it. modifiers (["ctrl","shift"], …) are held during the action.
Python also exposes left_click, double_click, right_click, middle_click, mouse_press, mouse_release, and move_mouse as named conveniences; TypeScript has leftClick, doubleClick, rightClick, middleClick, mousePress, mouseRelease, and moveMouse. Screenshot coordinates. coordinate_space="screenshot" (coordinateSpace: 'screenshot', --coordinate-space screenshot) maps coordinates measured in delivered screenshot pixels back to display space — use it on input calls after a fitted screenshot(max_width=…, max_height=…) so you can click directly on the image you gave the model.

Type on the screen

type sends a string as keystrokes (delay_ms spaces them out; mode="paste" stages the text on the clipboard and sends ctrl+v — faster and lossless for large blocks). press sends one key name — Enter, Tab, Escape, F1–F12, arrows, or a single character; hotkey sends a chord whose modifiers hold while the rest press. desktop.keys lists every accepted name. Python’s write is type without shadowing the builtin.

Copy and paste

The desktop clipboard is shared with every window on the display — set text or a PNG image and pasting inside the sandbox yields it, or read back what the session put there. clipboard accepts op: "get" | "set" with a text payload, or png (base64) for an image payload. Python returns the image decoded as bytes; TypeScript returns/takes the base64 string.

Take one step

act runs a batch of inputs in order, waits for the screen to settle, and returns the observation — {settled, rev, screenshot}. Each action is an input body with a kind — click, down, up, drag, move, scroll, type, press, hotkey, keydown, keyup, set_clipboard, or wait — either nested ({"kind":"click","mouse":{"x":10,"y":20}}) or the flat per-verb shape the input methods take ({"kind":"click","x":10,"y":20,"button":"left"}); {"kind":"wait","ms":300} pauses the batch. observe is a screenshot options dict (format, region, quality, scale_pct, max_width, max_height, cursor, if_rev) — omit it for no image. Feed the returned screenshot.rev back as observe.if_rev on the next call to skip unchanged frames.

Type a long string

  • Coordinates are absolute display pixels; keys lists the exact names press/hotkey accept.
  • act is the tight loop primitive — one request instead of an input call, a wait_stable, and a screenshot. Reuse rev between calls so unchanged frames transfer nothing.
  • type sends synthetic keystrokes — prefer mode="paste" for long text; it is faster and immune to keyboard-layout edge cases.

Endpoints