Automate

Automate

Make your phone or tablet smarter with automation

Get it on Google Play

Interact

A decision block that interacts with the on-screen user interface.

If the proceed option is set to Immediately, then the action will be performed on the UI elements currently visible on screen, without pausing the fiber. If set to When UI element appeared, then the fiber will pause until the UI element has appeared on screen.

Only use this block as a last resort if there’s no other way of performing the desired automation.

There are two types of interactions:

  • Global actions which can be performed at any time and doesn’t require knowledge of the on-screen UI elements, the proceed option, package and XPath expression input arguments are ignored.
  • Targeted actions which are performed on a specific Android UI element, custom graphics as used in browsers and games will likely not work. The UI element is selected by evaluating the XPath expression, if specified, otherwise it’s the currently focused UI element, if any. When passing text as string literal use the xpathEncode function to ensure proper quote character escaping. Use the Record interactions button to locate and generate the XPath expression of an UI element. The “context node” (.) is the entire UI layout document of the “active” window when using proceed Immediately, or an UI element that changed when using proceed When UI element appeared.

The NO path is executed if the action is allowed but failed to be performed, and also when the UI element isn’t visible on screen when using proceed Immediately.

Use the App start or App in foreground block to ensure that the correct activity is shown on screen. A delay between sequential interactions may be needed to ensure the UI has been fully loaded and redrawn.

UI layout XML

The XML element “tag” name is the UI element view class. Attribute availability depends on type of UI element and Android version, see table below. Use the Inspect layout button to view the XML layout of a window shown on screen.

AttributeValueDescription
@android:checkablebooleanCan display a check mark
@android:checkedbooleanChecked state
@android:choiceModetextChoice behavior for collection (Android 5+)
@android:clickablebooleanReacts to clicks
@android:columnCountnumberColumn count of collection (Android 5+)
@android:contentDescriptiontextAssociated content description
@android:contextClickablebooleanReacts to context clicks (Android 6+)
@android:editablebooleanHas an input method (Android 4.3+)
@android:enabledbooleanAllows interaction
@android:errortextError text (Android 5+)
@android:focusablebooleanCan take focus
@android:focusedbooleanCurrently has focus
@android:hinttextHint text (Android 8+)
@android:idtextIdentifier name (Android 4.3+)
@android:inputTypetextAllowed type of text input, separated by '|' (Android 4.4+)
@android:labelFortextIdentifier name of element it’s a label for (Android 4.3+)
@android:labeledBytextIdentifier name of label element (Android 4.3+)
@android:layout_columnnumberColumn index of collection item (Android 5+)
@android:layout_columnSpannumberColumn span of collection item (Android 5+)
@android:layout_height(number)Height
@android:layout_rownumberRow index of collection item (Android 5+)
@android:layout_rowSpannumberRow span of collection item (Android 5+)
@android:layout_width(number)Width
@android:layout_x(number)Horizontal position on screen
@android:layout_y(number)Vertical position on screen
@android:longClickablebooleanReacts to long clicks
@android:maxnumberMaximum value for range (Android 5+)
@android:maxLengthnumberMaximum length of text input (Android 5+)
@android:minnumberMinimum value for range (Android 5+)
@android:passwordbooleanInput characters displayed as password dots
@android:progressnumberCurrent value of range (Android 5+)
@android:rowCountnumberRow count of collection (Android 5+)
@android:scrollablebooleanCan scroll
@android:selectedbooleanCurrently selected

Record interaction

If the floating record window doesn’t generate an event, e.g. when clicking a button, it’s likely due to the UI layout XML changing before its XPath expression can be created. A workaround is to instead use the search feature (magnifying glass icon) to pick the UI element shown on screen, then change the action, e.g. to Click.

On Android 11+ the record interactions and inspect layout windows are hidden on system Settings unless the Allow screen overlays on settings option is enable in system Developer settings.

ArgX, ArgY

Some targeted actions take additional input arguments:

  • Move windowargX/argY is screen coordinate number.
  • Next/Previous at movement granularityargX is granularity; 1 for character (default), 2 word, 4 line, 8 paragraph or 16 page. argY is a boolean whether to extend selection (Android 4.3+).
  • Next/Previous HTML elementargX is HTML tag name text, default is "INPUT".
  • Press and holdargX is duration in seconds number, default is 0.
  • Scroll to positionargX/argY is column/row number of item.
  • Set progressargX is progress number value.
  • Set selectionargX/argY is selection start/end, default is to clear selection.
  • Set textargX is text to set, default is to clear text.

Note! Interacting with Automate isn’t permitted, for security reasons. The Quick settings action may not work prior to Android 6. The Show input method picker action requires the “appear on top of other apps or parts of the screen” privilege on Android 9+.

Options

  • Proceed — when the fiber should proceed.
  • Record interaction — record new UI element interactions.
  • Recorded interactions — previously recorded interactions.

Input arguments

  • Action — action to perform, default Inspect.
  • Package — package name of the app displaying the UI element, default is any.
  • XPath expressionXPath expression for selecting UI element to interact with, used for targeted actions, default is the focused UI element.
  • ArgX — first generic argument for targeted action.
  • ArgY — second generic argument for targeted action.

Output variables

  • Inspected content — variable to assign the text content of the inspected UI element.

Layout

  • Inspect layout — save the UI layout of a screen as an XML file, then view it.
Note! This documentation is also accessible within the app from Help & feedback menu.