> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gravixlayer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Read the interface

> Find a control by its name and press it

Find a button, field, or menu by its name and press it, without guessing pixel coordinates. Each control gets a ref (`d1`, `d2`, …) you pass to the action.

This needs the `desktop-small` template, and the display started. On any other template these calls return an error.

## Read the interface

`a11y_snapshot` returns `{rev, elements, truncated}` — a flat digest of the
tree. Each element carries:

| Field | Meaning |
| - | - |
| `ref` | `d{n}` — stable within a snapshot revision; pass to `a11y_action` |
| `role` | Element role — `push button`, `entry`, `menu item`, … |
| `name` | Accessible name (the label a screen reader would say) |
| `value` | Current value text when the element has one |
| `x`, `y`, `w`, `h` | Screen-space bounds in display pixels |
| `states` | State names — `enabled`, `showing`, `visible`, `focused`, `editable`, … |
| `actions` | Named actions the element supports — `press`, `set_value`, … |
| `parent`, `app`, `pid`, `depth` | Parent ref, owning application, process id, depth below the scope root |

Refs are minted per snapshot revision: they are stable until the next
snapshot or until the element disappears — acting on a gone element returns
a stale-ref error, so re-snapshot after the UI changes.

Scope the snapshot instead of walking the whole desktop:

* `scope: "app"` + `app` (name substring) or `pid` narrows to one application
* `scope: "focus"` narrows to the subtree holding keyboard focus
* `roles`, `name` filter elements; `max_depth` (default 6) and `limit`
  (default 500, hard cap 5000 — the reply's `truncated` says it hit) bound
  the walk
* `visible_only: false` includes off-screen elements (default keeps only
  showing+visible)
* `diff: true` + `rev` returns only elements changed since a prior snapshot

```python Python theme={null}
from gravixlayer import GravixLayer

client = GravixLayer()
sandbox = client.runtime.create(template="desktop-small")
sandbox.desktop.enable()

snap = sandbox.desktop.a11y_snapshot(roles=["push button", "entry"])
for el in snap["elements"]:
    print(el["ref"], el["role"], el["name"], el["states"])

# Only what changed since that snapshot
delta = sandbox.desktop.a11y_snapshot(diff=True, rev=snap["rev"])
```

```typescript TypeScript theme={null}
import { GravixLayer } from 'gravixlayer';

const client = new GravixLayer();
const sandbox = await client.runtime.create({ template: 'desktop-small' });
await sandbox.desktop.enable();

const snap = await sandbox.desktop.a11ySnapshot({ roles: ['push button', 'entry'] });
for (const el of snap.elements) {
  console.log(el.ref, el.role, el.name, el.states);
}

// Only what changed since that snapshot
const delta = await sandbox.desktop.a11ySnapshot({ diff: true, rev: snap.rev });
```

```bash CLI theme={null}
gravixlayer runtime desktop a11y-snapshot "$RT" --role "push button" --role entry

# Only what changed since that snapshot
gravixlayer runtime desktop a11y-snapshot "$RT" --diff --rev "$REV"

# One application only
gravixlayer runtime desktop a11y-snapshot "$RT" --scope app --app chromium
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/a11y/snapshot" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"roles":["push button","entry"]}'

# Delta — only elements changed since a snapshot rev
curl -sS -X POST "$API/agents/runtime/$RT/display/a11y/snapshot" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"diff\":true,\"rev\":$REV}"
```

## Find a control

`a11y_find` searches the tree directly — `role` and `name` (case-insensitive
substring) combine, `ref` scopes the search below one element, `max` caps
results (default 50, hard cap 500).

```python Python theme={null}
# The Save button anywhere in the tree
hit = sandbox.desktop.a11y_find(role="push button", name="save", max=5)
ref = hit["elements"][0]["ref"]
```

```typescript TypeScript theme={null}
// The Save button anywhere in the tree
const hit = await sandbox.desktop.a11yFind({ role: 'push button', name: 'save', max: 5 });
const ref = hit.elements[0]?.ref;
```

```bash CLI theme={null}
# The Save button anywhere in the tree
gravixlayer runtime desktop a11y-find "$RT" --role "push button" --name save --max 5
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

curl -sS -X POST "$API/agents/runtime/$RT/display/a11y/find" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"role":"push button","name":"save","max":5}'
```

## Press a control

`a11y_action` performs an accessibility action on a ref →
`{ok, settled, rev}`:

| Op | Payload | Effect |
| - | - | - |
| `click` | — | Activate the element (its `press` action) |
| `focus` | — | Move keyboard focus to it |
| `invoke` | `action` (name) or `action_index` | Run a named action the element exposes |
| `set_value` | `text` or numeric `value` | Set the element's value |
| `type` | `text` | Type text into the element |

The action settles damage-driven like `act` — it waits for the screen to go
quiet after the change, bounded by `timeout_ms` (default 1500).

```python Python theme={null}
# Focus a field, type into it, then press the button
sandbox.desktop.a11y_action("focus", entry_ref)
sandbox.desktop.a11y_action("type", entry_ref, text="report.pdf")
sandbox.desktop.a11y_action("click", save_ref)
```

```typescript TypeScript theme={null}
// Focus a field, type into it, then press the button
await sandbox.desktop.a11yAction('focus', entryRef);
await sandbox.desktop.a11yAction('type', entryRef, { text: 'report.pdf' });
await sandbox.desktop.a11yAction('click', saveRef);
```

```bash CLI theme={null}
# Focus a field, type into it, then press the button
gravixlayer runtime desktop a11y-action "$RT" focus --ref "$ENTRY_REF"
gravixlayer runtime desktop a11y-action "$RT" type --ref "$ENTRY_REF" --text "report.pdf"
gravixlayer runtime desktop a11y-action "$RT" click --ref "$SAVE_REF"
```

```bash cURL theme={null}
API="https://api.gravixlayer.ai/v1"

# Focus a field, type into it, then press the button
curl -sS -X POST "$API/agents/runtime/$RT/display/a11y/action" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"op\":\"focus\",\"ref\":\"$ENTRY_REF\"}"

curl -sS -X POST "$API/agents/runtime/$RT/display/a11y/action" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"op\":\"type\",\"ref\":\"$ENTRY_REF\",\"text\":\"report.pdf\"}"

curl -sS -X POST "$API/agents/runtime/$RT/display/a11y/action" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"op\":\"click\",\"ref\":\"$SAVE_REF\"}"
```

Use a11y actions when the element is what matters — dialogs, menus, form
fields — and fall back to [pixel input](https://docs.gravixlayer.ai/documentation/agentruntime/computer-use/input.md)
for canvas, WebGL, or anything the tree cannot describe.

## Endpoints

| Method | Path | Description |
| - | - | - |
| POST | `/v1/agents/runtime/{runtime_id}/display/a11y/snapshot` | Flat digest of the accessibility tree |
| POST | `/v1/agents/runtime/{runtime_id}/display/a11y/find` | Search the tree by role/name |
| POST | `/v1/agents/runtime/{runtime_id}/display/a11y/action` | Act on an element ref |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.