How AI is Already Changing Test Automation: 6 Examples

What’s the first thing that comes to mind when you hear the words “Artificial Intelligence?” Most folks automatically think of sentient machines, like HAL from the movie 2001: A Space Odyssey or Skynet from The Terminator. If you’re a tester and you manage to get this concept stuck in your head, it’s undoubtedly going to trip you up.

Remember: there are two kinds of AI that most folks are familiar with. The first one is the strong type of artificial intelligence–like a Turing machine you can have a conversation with and not be able to tell if you’re speaking with a human or a computer. The other is a more statistics-based, machine learning AI. If you use the latter definition of AI, you can see how machine learning is already transforming the way we do test automation. Today we’re going to be talking about the machine learning kind of AI in test automation. Following are six examples of the automation testing scenarios that are already leveraging AI. ( “This article is based on my article, “How AI is changing test automation, originally published on TechBeacon.com.”)

First, what is Machine Learning?

Before we look at some automation testing examples impacted by machine learning (ML), let’s try to define what machine learning actually is. At its core, ML is a pattern recognition technology. So naturally, it’s excellent at finding patterns in data. It can then use those patterns identified by your ML algorithms to predict future trends. Of course, to do this, you need a set of data that would be impossible for a person to go through and detect all of the familiar repetitions. That’s why ML is so powerful.

It can consume tons of complex information and find patterns that are predictive, then alert you to those differences. Angie Jones shares a hands-on demo of this in her AiSummitGuild session on Show and Tell: Explain Machine Learning to Me Like I’m Five.

Real-World Examples of Machine Learning Applications

To see machine learning in action, you need to look no further than the smartphone you’re probably carrying. Phones that have voice recognition software (like Cortana or Siri) apply AI to mimic the voice/human interactions you take part in. Another frequent use case you’ve probably experienced is the list of recommendations served up when you’re shopping on Amazon, which is based on your previous, machine-learning-generated purchases. By now you’re probably saying, “Okay! Enough already. Show me some automation examples!” Fair enough–read on.

Visual Validation Automation Testing

As we’ve already covered, ML is a pattern recognition technique. So what kind of patterns can it recognize? One that is becoming more and more popular is image-based testing using automated visual validation tools. Using ML-based visual validation tools like Applitools can find differences that human testers would most likely miss. “Visual testing is a quality assurance activity that is meant to verify that the UI appears correctly to users,” explained Adam Carmi, co-founder, and CTO of Applitools, a dev-tools vendor. Many people confuse that with traditional, functional testing tools, which were designed to help you test the functionality of your application through the UI.

With visual testing, “we want to make sure that the UI itself looks right to the user and that each UI element appears in the right color, shape, position, and size,” Carmi said. “We also want to ensure that it doesn’t hide or overlap any other UI elements.” In fact, he added, many of these types of tests are so difficult to automate that they end up being manual tests. This makes them a perfect fit for AI testing.

This has already changed the way I do automation testing. I can create a simple ML test that automatically detects all the visual bugs in my software to help validate the visual correctness of the application without me having to implicitly assert what I want it to check. Pretty cool!

Not Testing UI Interfaces

Another ML change that is impacting how we do automation is the absence of a user interface to automate. Much of the testing is back-end-related, not front-end-focused. In fact, in her TestTalks interview, “The Reality of Testing in an Artificial World,” Angie Jones, an automation engineer at Twitter, mentioned that much of her recent work has relied heavily on API test automation to help her ML testing efforts. Angie went on to explain that in her testing automation, she focused on the machine learning algorithms. “And so the programming that I had to do was a lot different as well. … I had to do a lot of analytics within my test scripts, and I had to do a lot of API calls.”

Becoming a Domain Model Expert

Besides not having a traditional user interface to test against, being able to train an ML algorithm requires that you come up with a Testing Model. This activity needs someone with domain knowledge; many automation engineers are getting involved with creating models to help with this development endeavor. With this change, there is a need for folks who not only know how to automate but can also do more headless-based automation–along with analyzing and understanding complex data structures, statistics, and algorithms.

Running More Automated Tests That Matter

How many times have you run your entire test suite due to a very small change in your application because you couldn’t trace that change? Not very strategic, is it? If you’re doing continuous integration and continues testing you’re probably already generating a wealth of data from your test runs. But who has time to go through it all to search for common patterns over time? Wouldn’t be great if you could answer the classic testing question, “If I’ve made a change in this piece of code, what’s the minimal number of tests I should be able to run in order to figure out whether or not this change is good or bad?” Lots of companies are leveraging existing AI tools that do just this. Using ML, they can tell you with precision what the minimal subset of tests is to test the piece of changed code. They can also analyze your current test coverage and flag areas that have little coverage, or point out areas in your application that are at risk.

In my opinion, the most popular AI automation area right now is using ML to automatically write tests for your application by spidering. For example, newer AI/ML tools like Mabel have a feature that allows you to simply point it to your web app, and it will automatically begin to crawl the application. As the tool is crawling it is also collecting data having to do with features like taking screenshots, downloading the HTML of every page, measuring load times and so forth. And it continues to run the same steps again and again.

So over time, it’s building up a data set and training your ML models for what the expected patterns of your application are. When it runs, it compares its current state to all the known patterns it has already learned. If there is a deviation (for instance, a page that usually doesn’t have JavaScript errors but now does), a visual difference or it’s running slower than average, it will flag it as a potential issue/insight. Some of these differences might be valid. For example, there was a valid new UI change. In that case, a human with domain knowledge of their application still needs to go in and validate whether or not the issue(s) flagged by the machine learning algorithms are really bugs.

Although this approach is still in its infancy, Oren Rubin, CEO, and founder at machine learning tool vendor Testim, says he believes that “the future holds a great opportunity to use this method to also automatically author tests or parts of a test. The value I see in that is not just about the reduction of time you spend on authoring the test; I think it’s going to help you a lot in understanding which parts of your application should be tested.”