sandbox.desktop.enable()). A click lands at the coordinates you pass, or at the current pointer if you omit them.
CLI and cURL examples use $RT with the display already started (Screen).
Move and click
click takes x, y, an optional button (left/middle/right) and
click_count (2 is a double-click); an unset axis keeps the current
pointer position. down/up press and release a button — between them the
pointer can move for a manual drag; drag does the same in one call,
interpolating from the current position (or x/y) to to_x/to_y over
steps inside duration_ms. scroll sends dx/dy wheel deltas — the
wheel lands under the pointer, so pass x/y to position it first (an
omitted axis keeps the current one) or move to aim it. modifiers
(["ctrl","shift"], …) are held during the action.
left_click, double_click, right_click,
middle_click, mouse_press, mouse_release, and move_mouse as named
conveniences; TypeScript has leftClick, doubleClick, rightClick,
middleClick, mousePress, mouseRelease, and moveMouse.
Screenshot coordinates. coordinate_space="screenshot"
(coordinateSpace: 'screenshot', --coordinate-space screenshot) maps
coordinates measured in delivered screenshot pixels back to display space —
use it on input calls after a fitted screenshot(max_width=…, max_height=…)
so you can click directly on the image you gave the model.
Type on the screen
type sends a string as keystrokes (delay_ms spaces them out;
mode="paste" stages the text on the clipboard and sends ctrl+v —
faster and lossless for large blocks). press sends one key name —
Enter, Tab, Escape, F1–F12, arrows, or a single character;
hotkey sends a chord whose modifiers hold while the rest press.
desktop.keys lists every accepted name. Python’s write is type
without shadowing the builtin.
Copy and paste
The desktop clipboard is shared with every window on the display — set text or a PNG image and pasting inside the sandbox yields it, or read back what the session put there.clipboard accepts op: "get" | "set" with a text
payload, or png (base64) for an image payload. Python returns the image
decoded as bytes; TypeScript returns/takes the base64 string.
Take one step
act runs a batch of inputs in order, waits for the screen to settle, and
returns the observation — {settled, rev, screenshot}. Each action is an
input body with a kind — click, down, up, drag, move, scroll,
type, press, hotkey, keydown, keyup, set_clipboard, or wait —
either nested
({"kind":"click","mouse":{"x":10,"y":20}}) or the flat per-verb shape the
input methods take ({"kind":"click","x":10,"y":20,"button":"left"});
{"kind":"wait","ms":300} pauses the batch. observe is a screenshot
options dict (format, region, quality, scale_pct, max_width,
max_height, cursor, if_rev) — omit it for no image. Feed the returned
screenshot.rev back as observe.if_rev on the next call to skip unchanged
frames.
Type a long string
- Coordinates are absolute display pixels;
keyslists the exact namespress/hotkeyaccept. actis the tight loop primitive — one request instead of an input call, await_stable, and ascreenshot. Reuserevbetween calls so unchanged frames transfer nothing.typesends synthetic keystrokes — prefermode="paste"for long text; it is faster and immune to keyboard-layout edge cases.