Learn how these leaders think about coverage, what metrics they value, and how they shift testing to the left. Last month, I had the pleasure to moderate a live panel featuring engineering directors from two Applause customers: eBay’s Srikanth Rentachintala and Walmart’s Anthony Tang.
At eBay, Rentachintala is responsible for all aspects of quality engineering within the buyer experience group, including test automation, environments and strategy. He is also responsible for the digital accessibility of eBay’s properties.
As part of Walmart Global Tech, Tang leads the developer experience tools team. He is responsible for test automation tooling as well as accessibility and performance tooling for customer experience.
We discussed several strategic areas that all engineering leaders face, such as how to leverage test automation, use test data properly and insert accessibility testing.
In this blog post, learn how Rentachintala and Tang approach test automation at eBay and Walmart, respectively, particularly around test coverage, metrics and shifting testing to the left.
What is the right percentage of coverage?
There are two kinds of coverage, which often causes confusion: test coverage and code coverage.
Test coverage refers to the percentage of test cases that are automated. Code coverage is the percentage of code paths covered by automated testing.
When it comes to test coverage, some test cases simply can’t be automated. Test cases such as specific error paths, for example, require manual testing. With code coverage, aim to be as close to 100%, but it might not be possible to get all the way there if you’re just getting started. This conversation, however, focused on test coverage.
The percentage of test coverage ranges widely by organization, often from a low of 20% to upwards of 80%. Here at Applause, our experience has been that it’s challenging to extend beyond 80% and still get value. However, the test coverage percentage can be misleading. Teams should monitor the quality of the tests too.
“Test case automation coverage is not the easiest measure to come by,” Tang said. “To get the denominator correct, it involves some manual thinking around, ‘What are all the tests? What is the full population of test cases that you need to automate?’
“One interesting thing that we see is that a lot of people, when they see percent of test cases, they think percent of coverage. And so, a lot of people will report their coverage percentage, which isn’t the same, and I think that’s important to point out.”
The percentage of test coverage ranges widely by organization, often from a low of 20% to upwards of 80%
Rentachintala said eBay is around 85%-90% for ROI on back-end services, while it’s more like 75% for front-end applications.
“At one point, there is a point of diminishing returns,” Rentachintala said. “So that’s where you don’t want to invest more. For those use cases, maybe it’s not really helpful to have more automation there.”
At Walmart, there have also been differences in getting higher percentages of test coverage based on existing projects vs. new projects. Tang noted that his team wanted to improve the test coverage of an existing project, and topped out at around 70%. However, for a new project where the team could utilize engineering and QA best practices from the beginning, it is now at 90% test coverage.
“The last 10% is difficult,” Tang said. “The goal is, of course, 100%, but you reach a point of diminishing returns.”
What metrics are important for test automation?
There are dozens — if not hundreds — of metrics to judge your team’s success with test automation. Test coverage and code coverage are two of those many test automation metrics.
https://www.applause.com/blog/how-ebay-walmart-tackle-test-automation