Fields

Many new features in field APIs and tooling. Highlights:

• Field finder evolved
• Field editing UI for inline/dynamic fields
• New events API for web
• Get selected text
• Masking a field

Fields

Field finder evolved

Old field finder

• Produced a single often very long path
• Often didn’t work (some drivers more than others)
• Required complex manual work in some cases to make the path work robustly
  • External tools sometimes required (e.g. Inspect.exe, Chrome dev tools, intuition)
• Unforgiving learning curve for new developers

Fields / Field finder

New field finder

• A set of working paths are automatically produced
• Evolutionary algorithm produces, combines and evaluates field variations
• Ranked list of working paths
• Explanations for the assigned scores (pros and cons)
• Much easier to use for new developers

Fields / Field finder

Evolutionary algorithms

Takes inspiration from Darwinian evolution.

Fields / Field finder

Mutations

A random sequence of the following mutations produce path variations for evaluation

• Deep match: A/B/C mutates to **/B/C, **/C, A/**/C
• Token swap: A/B/C mutates to A2/B/C, A/B2/C, A/B/C2, A/B2/C2…
• Positional: A/B/C mutates to **/some-label<left-of>C, **/some-label<above>C…
• Relational: A/B/C mutates to A/B/D<sibling-of>C. **/E<child-of>C…

The result is non-deterministic. Run it again!

Fields / Field finder

Fitness criteria

The machine is opinionated. We have embedded our own experience into the algorithm.

• Working path: Main criterion. Is the right UI element found.
• Speed: Prefer faster paths.
• Brevity: Prefer shorter paths.
• Modifiers based on labels: These are usually quite robust.
• Unique tokens: Prefer paths with many unique tokens.
• Dynamic tokens: Prefer paths with word-like tokens.
• Too generic: E.g. **/(type)button is less preferred.

And even more. Weights for each can be customized in settings.

Fields

UI for dynamic fields

Same UI as for stand-alone fields via new inline editor link

Fields

UI for dynamic fields

Fields

New events API for web

Advanced api for rare cases. Emits an event from a field. Common use case: Trigger form validation to enable submit button.

Fields.nameInput.input("121212-5CD9");

Fields.nameInput.emit(Events.KEY_PRESS, "Enter");

Fields.okButton.click();

The second line is translated to

element.dispatchEvent(new KeyboardEvent("keypress", { key: "Enter" }));

Fields / New events API for web

Event types

Predefined e.g. LEFT_MOUSE_DOWN, RIGHT_MOUSE_UP, KEY_DOWN, FOCUS, BLUR, CHANGE.

The emit function also supports further customization for the adventurous user:

// in flow
Fields.nameInput.emit("Event", "custom-event", {
  bubbles: false,
  cancelable: true,
  composed: true,
  customProp: "foobar",
});

// in the browser it is translated to
element.dispatchEvent(
  new Event("custom-event", {
    bubbles: false,
    cancelable: true,
    composed: true,
    customProp: "foobar",
  }),
);

Fields

Get selected text

The selectedText property has been added to the inspect result for text inputs and textarea elements.

var selectedText = Fields.note.inspect().selectedText;

Fields

Masking a field

A mask is an “obstruction” that can hide a UI element or part of the screen.

It is not a fool-proof security measure but can be useful to prevent accidental clicks or to hide information that should not be seen by the user.

This works on a Field instance by invoking the mask function. In the simplest case you can supply no arguments:

// Assuming you have a "OkButton" field defined
Fields.OkButton.mask();

Fields

Masking a field

The mask function can take an object as argument to control the appearance and behavior of the mask. E.g. to control the color of the mask. The following options are available:

• color The color of the mask.
• blur Blur instead of color.
• throttleMs How aggressively to throttle the mask.

The blur: true mode looks like this:

New modules

Network

• Pings
• DNS lookups
• Wake-on-LAN
• ARP
• IP config
• Trace route
• Netstat

New modules

Compress

Zip and unzip files.

Screens

Detect physical screens and their properties.

Image

Crop, color detection.

New modules

Word

Parsing and basic editing of Word documents.

OpenAI

Interaction with OpenAI LLM models.