Test your UI by treating it as a state machine

I want to introduce a state machine testing library for .Net called Scrutiny. You can get it from NuGet here. It aims to allow random testing of state machines, with a focus on UI testing, by modeling a state machine as a directed graph and path finding through all possible states.

Testing your application has always been a necessary part of any successful software project. You can write tests for individual units within your code base, to testing a single interaction between systems with integration tests, or you can test your entire UI with UI tests. There are countless ways on how to split and organize these tests, and countless debates on which tests are "better" than others, but I don't want to get into any of that here. I want to talk about UI Tests.

Current UI testing solutions

My entire professional career has revolved around web apps, so this article will focus on exactly that. This means that the UI tests I'll be talking about are web UIs, and the tooling and ecosystem surrounding that. In the .Net world, the primary tool for UI testing has always been Selenium, and along with that, there exists several wrappers that aim to make using selenium easier (e.g. Canopy), or adding a DSL on top to allow anyone without programming experience to write UI tests e.g. SpecFlow.

There are other solutions that aren't Selenium. In the Node.js world, a common tool is Puppeteer, but this only supports Chrome. There also exists a relatively new project from the developers of Puppeteer called Playwright which aims to fix some of Puppeteers problems, the biggest addition being multi browser support. These libraries also exist in .Net (here and here).

But what all these libraries and wrappers share is a common problem. You have to write every single test individually. This is of course how all types of tests are written, but I have personally always found it tedious when it comes to UI tests. Of course the same mechanisms for code reuse exist in these UI testing libraries as any testing library, but the big difference is the runtime. Unit tests, and in most case integration tests, enjoy the ability of being incredibly quick to run. So fast in fact that many tools exist to run all tests on every code change. This doesn't work for UI tests as each test needs to click through your UI, and this is simply much slower than crunching some numbers in memory. This is especially a problem when it comes to parameterised testing, as the same workflow is tested multiple times with different inputs. For UI tests, this can seriously blow up the time taken to run all tests.

Another problem is every test follows the exact same path. A UI test is usually written to accomplish a single workflow as directly as possible. But your end users usually won't use your product like that. They'll be looking around, scrolling the page up and down. Maybe navigate away from the page and hope that it retains its state to continue later. There's an infinite number of possibilities for how an end user may interact with the UI. Often times, its in one of these combinations of interactions that certain bugs can crop up.

Introducing Scrutiny

What is it

I created Scrutiny to try and solve these problems. With Scrutiny, you define all your possible pages in your UI as states within a state machine. Transitions between these states are defined within each page object, as well as any actions that can be performed within this page e.g. open/close an accordion or popup. Once you are satisfied with your page definitions, transitions, and actions, you can start the scrutinization process. At this point, your pages will be modeled as a directed adjacency graph, and any defined actions will be run in your starting state.

Once the testing has started, Scrutiny will pick a random unvisited state within the graph to navigate to, using a path finding algorithm. As it steps through each state until it reaches the chosen state, it will continue to perform any actions in random order in the intermediary states. Scrutiny will then continue this process until all reachable states have been visited at least once, at which point it will try to navigate to a state chosen as the exit state.

After it has completed a test run, or if it encounters a problem and throws an assertion exception, Scrutiny will generate an HTML report. This report contains a visualization of the aforementioned adjacency graph with all defined states and their respective transitions. You can then step through the state transitions and see the path that the library took through your UI, including showing in which state the error occurred.

There are many configuration options to this process, the most notable being that you don't have to run through all possible states and actions, but only a certain subset. There is also the option of only generating a map of your UI without running through the actual steps if you're interested in a visualization of your UI.

When should you use it

A common mistake people make when learning about a new tool is to try and use it in every situation and foregoing any other tools they or their team might be familiar with. I want to make crystal clear that Scrutiny is not a replacement for any other testing method, simply another tool to utilize alongside others. A single Scrutiny test on a complicated web app may take a very long time to run, whereas other tests are much quicker. So it's up to you and your development team to figure out a proper time to run these tests. For example, only on pull requests, or a nightly job within your CI environment.

So now the question becomes, if Scrutiny should be used alongside other tests, why should I write them at all? My answer to that is, that a single Scrutiny test will already test a large portion of your UI, and therefore you only need to write very few specific UI tests that test a certain edge case. It's all about striking a balance between the two. On top of that, Scrutiny allows for the random element that an end user will bring to the table. This is similar to property based testing where the inputs aren't known before runtime, and can therefore help uncover other edge cases.

What's to come

As this library is written in F#, it runs on .Net. This means that it should be usable from C# as well. This may be the case, but since the main construct in Scrutiny as a computation expression, the C# usage is rather ugly. Therefore I want to write (or accept help) a nice C# wrapper API that will make writing tests in C# much nicer.

Additionally, F# has an interesting library called Fable, which transpiles F# to Javascript (with typescript bindings). It would also be interesting to use Fable to create an npm package that would allow usage of Scrutiny from Node.js

As for now, check it out on Github grab the library from NuGet and write some UI tests, or test any state machine. I'm very curious to see how people like it.