Automate
Make your phone or tablet smarter with automation
Make your phone or tablet smarter with automation
A decision block that inspect the User Interface (UI) shown on screen.
If the proceed option is set to Immediately, then the UI is evaluated with what’s currently shown on screen without pausing the fiber. If set to When UI layout changed, then the fiber will pause until the UI has changed and evaluation produced a result.
The XPath expression is used to evaluate the UI, e.g. to select a specific Android UI element and/or extract a attribute from it, custom graphics as used in browsers and games may not work. The “context node” (.) is the XML document root element when using proceed Immediately, or the UI element that changed when using proceed When UI layout changed. When passing text as string literal use the xpathEncode function to ensure proper quote character escaping.
The YES path is executed when the evaluated result is a Number, a String, a Boolean true, a selected Node or a non-empty Node-set. The NO path for a Boolean false, or when using proceed Immediately, not selected a Node or an empty Node-set.
The UI is represented as an XML document, of the type as specified by the schema:
Since it includes a lot more data to be processed avoid using the Display windows schema unless access to other windows is required. Enable the Debug logging option in settings to see statistics on processing performance. Use the Inspect user interface button to preview the XML document.
The root of the XML document when using the Active window layout schema, or the contents of an <layout> element otherwise. A view element represents a UI component drawn on screen, it may contain other views (elements) that it draw. The XML element (tag) name is its UI component class name. Attribute availability depends on the type of UI component and Android version.
| Attribute | Value | Description |
|---|---|---|
@android:checkable | boolean | Can display a check mark |
@android:checked | boolean | Checked state |
@android:choiceMode | text | Choice behavior for collection (Android 5+) |
@android:clickable | boolean | Reacts to clicks |
@android:columnCount | number | Column count of collection (Android 5+) |
@android:contentDescription | text | Associated content description |
@android:contextClickable | boolean | Reacts to context clicks (Android 6+) |
@android:editable | boolean | Has an input method (Android 4.3+) |
@android:enabled | boolean | Allows interaction |
@android:error | text | Error text (Android 5+) |
@android:focusable | boolean | Can take focus |
@android:focused | boolean | Currently has focus |
@android:hint | text | Hint text (Android 8+) |
@android:id | text | Identifier name (Android 4.3+) |
@android:inputType | text | Allowed type of text input, separated by '|' (Android 4.4+) |
@android:labelFor | text | Identifier name of element it’s a label for (Android 4.3+) |
@android:labeledBy | text | Identifier name of label element (Android 4.3+) |
@android:layout_column | number | Column index of collection item (Android 5+) |
@android:layout_columnSpan | number | Column span of collection item (Android 5+) |
@android:layout_height | (number) | Height |
@android:layout_row | number | Row index of collection item (Android 5+) |
@android:layout_rowSpan | number | Row span of collection item (Android 5+) |
@android:layout_width | (number) | Width |
@android:layout_x | (number) | Horizontal position on screen |
@android:layout_y | (number) | Vertical position on screen |
@android:longClickable | boolean | Reacts to long clicks |
@android:max | number | Maximum value for range (Android 5+) |
@android:maxLength | number | Maximum length of text input (Android 5+) |
@android:min | number | Minimum value for range (Android 5+) |
@android:password | boolean | Input characters displayed as password dots |
@android:progress | number | Current value of range (Android 5+) |
@android:rowCount | number | Row count of collection (Android 5+) |
@android:scrollable | boolean | Can scroll |
@android:selected | boolean | Currently selected |
A <window> element represents a window shown on a display, it may contain other windows (<window> elements) that it own. It also always contains a single <layout> element that itself contains the view it draws. Attribute availability depends on Android version.
| Attribute | Value | Description |
|---|---|---|
@android:active | boolean | Currently active |
@android:focused | boolean | Currently has focus |
@android:id | number | Identifier value |
@android:layout_height | (number) | Height |
@android:layout_width | (number) | Width |
@android:layout_x | (number) | Horizontal position on screen |
@android:layout_y | (number) | Vertical position on screen |
@android:packageName | text | Owner app package |
@android:textLocales | text | Languages used, separated by comma (Android 14+) |
@android:title | text | Title |
@android:windowLayout | number | Z-order, higher above lower |
@android:windowType | tex | Type of window |
The root of the XML document when using the Display windows schema. The <display> element contain one or more <window> elements, representing all the windows shown on the display, as determined by the display id.
| Attribute | Value | Description |
|---|---|---|
@android:id | number | Identifier value |
@android:layout_height | (number) | Height |
@android:layout_width | (number) | Width |
@android:layout_x | (number) | Horizontal position on screen |
@android:layout_y | (number) | Vertical position on screen |
@android:label | text | Name of display |
@android:rotation | number | Degrees of rotation |
If the floating record window is unable to record an UI event, e.g. when clicking a button, then it’s likely due to the XML document changing again before the XPath expression can be generated. A workaround is to instead use the search feature (magnifying glass icon) to pick the UI element shown on screen.
The generated XPath expressions may have to be manually modified to work since it’s difficult automatically predetermine how the UI will change as the user interacts with it.
On Android 11+ the record interactions and inspect user interface windows are hidden on system Settings unless the Allow screen overlays on settings option is enable in system Developer settings.
The XML of the entire UI can be very large, therefore avoid using result type Node and Node-set that select entire elements that’s then parsed and processed using blocks/functions, instead let the XPath do most of the “processing” to limit the result to a Boolean or the value of a specific attribute.