Introduction

UI automation refers to testing of the user interface using automated systems instead of manual user testing.

This is achieved through automation instructions that interacts with the user interface and ensuring that the response from the client is what was expected.

The current mindset around automation is giving instructions on what to do.
Click on this button, wait for this screen, type this text.
There are a few issues with this when it comes to dynamic systems.

1. Changes on the UI, such as moving a button from one location to another affects the tests negatively.
2. Test definitions are long winded, difficult to maintain and takes a long time to define and fix.
3. Defining complex processes become very difficult and it’s hard to understand what the test is trying to achieve.

Though there are more issues one can define I want to focus on these two for now.

We need to simplify the tests definitions.
Simple test definitions are easier to define, understand and maintain.
UI Testing should be focused on a theme so that scenario testing makes sense and is focused on real world scenarios not just interacting with the UI.

Tests should be focused.
Instead of having one large test that is trying to do a lot of things, it is better to have more tests, smaller in size and more focused on a particular scenario.

Instead of step-by-step action-based tests we should change the way of thinking towards intent based tests.
The test should want to achieve something on the system. The person defining the test should only need to define the intent not the steps required to achieve that intent.

For example: “I want to create a new work order and populate it with these fields”
Normally we would define all the user interactions required to create a work order.
With intent driven testing we instead define what we want to do and the testing system has modular extensions that understand how to achieve that intent.

Extensions

The automation engine relies heavily on the process api architecture.
As part of the process api we can register modules that becomes callable from the JSON definition.
These extensions are focused on one part of the system and specializes in only that part of the system.
When we have more complex interactions one extension can call another extension to perform work required in an area it does not understand.

To understand this a bit better let us use an example.

Using a create screen, the user interacts with inputs of different kinds.
Date, time, duration inputs, combo boxes, each one of these has different structures that needs to be contended with. The definition for creating a record only defines the entity and the changes to make.

In summary the process of creating a record is:

1. Open the relevant create screen.
2. For each change defined in the test definition find the field on the UI,
3. For that field make sure the right tab sheet is selected and its group box is expanded.
4. Focus the input if it is not read-only,
5. Enter the data defined in the test.

The form does not know how tab sheets work and it does not know how group boxes work.
It is aware that there is such a thing as a tab sheet extension and a group box extension.
The form extension deals with all the form generic actions.
When specialized actions are required, like selecting the relevant tab sheet, it asks the tab sheet extension to do that.

This way there is a separation of concern where each extension is a specialist in a particular area and others rely on the relevant specialist to ensure successful operations.

Another important point to note is that all extensions get registered on the process api, like a “plugin”.
All modules on the process api is accessible using the JSON definition and programmatically.
This means that should I have a specific scenario where do I want a bit more action-based control as part of my test process I have the capability.

Extensions don’t only perform actions but also do validations.
Validation methods always start with “validate”.
For example “validate_selected”

Test bundles

Though we can run individual tests, all tests should belong to a test bundle.
Not all bundles need to be run daily and different bundles serve different purposes.
These are some of the bundles.

1. Components – testing individual components in different environments it is being used.
   The quick status change component is an example of this.
   It has several test files, each testing the same component on a different screen.
   Over time each component should have one or more tests in this bundle.
   The focus of the test is just testing the component.
2. Screens – most of the tests we have will fall in this bundle, some generated based on a pattern, others handcrafted. Screens like the asset tree will have many tests, each testing a particular feature.
   There are several scenarios where there are dependencies that must be satisfied before creating for example can take place. In those cases, process tests are better suited.
3. Processes – these tests don’t test a particular component or screen but jumps through many interactions chaining actions as a cohesive process. Each process should have a corresponding test in this bundle. The length of the process is not of importance here. As long as it is a user gurney required to achieve an operational goal, the test should be defined here.
4. System health – these tests do not make data changes to the system. They are used to check for system issues. We have some tests in this space to test for memory leaks. We are aiming to have tests in future that checks profiles on all the screens to ensure they load properly. These tests are aimed to verify that migrations succeeded and should not rely on any data. When migrating the testing database from one version to the next, the system health tests are the first that should be run to ensure migration was successful.
5. Configuration testing – there are a great many scenarios that need to be verified based on system options and ui settings. Configuration tests start by making system setting changes and then verify that the required changes were applied as expected in the affected areas.

System health tests are tests that we can run on the weekend or even once per sprint.
Processes don’t need to run each day but it is a good idea to run them at least once every week.

Screen and component tests should run every day because if they don’t work, the others will not either.

The test bundle folders are not flat structures.
For example, the components bundle folder has subfolders for each component.
The component folder can have subfolders but typically contain all the different test scenarios, each in it’s own JSON file.

Generate what you can

There are tests that you should hand craft because there is no formula to determine the sequence of events.
Examples of this is when you need to insert several records before you can create the record you are trying to test. Defining processes that span several screens or diverse interactions is another example.

For the most part, if you can define a pattern, as many tests as possible should be generated using a script.
This drastically cuts down on time it takes to cover large quantities of screens and scenarios in the system.

Sometimes generating screens only get you 80% of the way due to some variations to the pattern.
Having to only handcraft 20% of the intent is acceptable.

Sequences

The process api does allow you to define sequences in test definition file.
This is done by defining different processes in the file and giving them proper names.
You can then execute each process, one after the other in series.

As a rule, this should be avoided.
It is too easy to create complex test definitions when they should have been separate test files.

Testing theme

Testing data up to this point has seemed a bit random.
Some randomness in the data is unavoidable. For example we need to have unique values for codes to avoid primary keys. We also can’t hard code date values, instead we formulate dates. For example (today – 1 day) or (today + 1 day). This gives us a progression of time in the testing data.

Generating randomized values serves the purpose of preventing duplicate key violations.
Sometimes it is enough on a per screen bases and sometimes you need to have a process to ensure success.

This is not enough.
We need to have a data theme that makes up part of the base line and enables cohesive and sensical scenario testing around the theme topic.

The theme determines the asset types, components, assets and other data.
The purpose of the theme is to provide order in the data we are creating during testing.

There is a question around multiple themes and how we would manage multiple themes.
Do we try and cater for different industry types on the same tenant?
Do we have different tenants with different themes on, one for mining and one for manufacturing?
How does the theme affect testing of systems options?
When we have profile packs, these packs need to make sense in the automation data theme.
Does not help you have a mining theme but the profiles are based on manufacturing.

Keep in mind that yes, this is primarily an automation database, but the same data must be used for “real” world tests also. Doing mobile tests on 100k records for example. Working with vast asset trees also being a real-world example.

The question here is how we bring order to automation testing, including processes and scenario testing.
The only way to have order is to have a reliable baseline.

Part of the base line will be to create sensible profiles on all screens and ensure that all screens have some data on it.

This means that the data base line serves different purposes.
1. Have access to testable data while we can’t generate it.
2. To have sensible data to do proper scenario testing with.
3. Having a set or data you know will be there to ensure durability of tests.

It is plausible that at some point in the future we might want to reset the database to be “clean” again.
“Clean” in this term means the data base line that was manually crafted as this should be the only data you depend on.

Looking ahead, we want to be able to generate synthetic data inline with the theme.
For this we will tap into AI to help generate this data.
I have already done some tests using llama3 and the results were very promising but it’s one thing to run it locally and another to have it operate as part of the testing pipeline.