Test Automation Tools for Desktop Apps. Pt.1: Boxed solutions

Software development practices change over time, and so does the test automation landscape. Today most applications are either web- or mobile-based. The test automation of such systems is well described. There are some best practices, information-loaded conferences, good tools, and so on. However, we faced a lack of information regarding desktop testing automation, in particular, for Windows 7+ apps.

Our task was to test such an application, given that manual QA-process would take more than a month. Here we share the pros and cons of discovered solutions, along with our experience and some best practices of the test automation process.

If you’ve ever faced QA-automation, we bet there were automated tests for a web page, web blog, or a web interface. Your team probably enjoys Appium for functional testing of a mobile app, or instrumental tests for Android (Espresso).

Still, in 2018 we keep developing desktop apps and maintaining legacy projects. Institutions such as banks, financial departments, laboratories, and HoReCa keep utilizing Windows Desktop. Businesses of all forms use them to automate, organize, and control their business processes.

The web will never fill every users’ need, as opposed to a user PC (like working with local files and devices, big data processing, ability to plug in specific devices, accessing services. Still, there are reasons to keep using desktop apps and there are several ways to use them: 

• User identification. Fingerprint scanners, passport scanners, etc.

• Corporate security reasons. The Internet can be restricted or limited at some manufactures or banks.

• Other reasons. For example, a company’s existing equipment fleet that already consists of Windows 7 PCs.

These are real customers’ needs, and no Web-based system may meet them.

Does testing take that long? The Do pilot managing pattern

Our customer uses a .NET desktop application for document control and business process automation, which has been developed and maintained for years.

This means dozens of thousands of code lines (LOCs), which calls for hundreds of tests and for about 1.5 months for primary manual testing:

• Two weeks for primary manual regression testing

• Two weeks for secondary testing

• Time to verify bugfixes or correct errors and possible misconfigurations

Errors are being corrected, while estimated time increases. Planning and control mistakes even more time-consuming. So we get about 6 weeks of two engineers’ full-time work needed for regression tests.

In case a new feature or a module is needed ASAP, six weeks of testing is a huge cost for a client.

Every customer prefers fast release iterations: from an inquiry through speedy testing to deployment. The key is keeping the quality level, or even increasing it, by reliable quantitative criteria. This is how we figured that our testing had to be automated.  

Let’s admit that not every task in the QA process is automatable. Something still has to be done manually. When our team needs to make sure the possibility of the automation, we use the Do Pilot approach for automated testing.

The Do pilot strategy is about the automation of some part of your task. Automate the part, and evaluate the result: spent time, scalability of the tests, the cost for maintenance. Just do the pilot with your automated testing solution, and then decide if it is worth expanding for the whole project.

It is always better to spend 1-2 weeks exploring the strengths of the product and making sure that it is worth investing time and effort in the automation.

Do Pilot in WaveAccess projects: requirements and possibilities

So we decided to develop a pilot version: to build a suite of tests for our client’s .NET Windows Desktop application. Our team took the stable app’s version, and chosen six basic testing scenarios with Smoke or Regress tag. This would guarantee that the application is functioning as expected in the technical task.

The resulting solution must be:

• Continuous Integration. We need to read reports from the CI after night testing cycle (or to get the notification that they failed).

• Transparency. It should be clear to the QA team: what is being checked on every step, while the project manager needs a clear report (for example: ‘add a third field to an X form).

• Cost. We search for a tool that won’t increase the project’s costs.

What we should decide first, is what automation tool would meet each of our expectations.

Solutions and frameworks for automated testing

As it was mentioned before, automated QA is dominated by Web- and mobile technologies. Let’s change the game, and find something suitable for Windows Desktop. 

Commercial tools. TestComplete

TestComplete by SmartBear is one of leader among testing automation studios. TestComplete has often been named a well-designed and highly functional commercial automated testing tool, it offers a trial-period and is often described positively by its users.

Let’s choose a ‘boxed’ solution for a single user:

Let’s customize our solution by excluding Web and Mobile.

Let’s reduce costs as much as possible. For access to real and virtual machines Desktop, Video Replays of Your Tests Unlimited Screenshots, Unlimited Live Testing and Enterprise-Grade Security it is required to add another $1700 for a single license.

Certificates that ensure that your team well knows TestComplete you can get for 150-700$ for each certificate. Let’s choose one basic TestComplete Certification.

The bottom-line price resulted in:

For TestComplete we’d pay 4000$ for a studio, and for one year of support for a single user (our team consists on two specialists). However, our aim was to offer our customer the solution which wouldn’t force them to pay either us nor for the license, while keeping the quality level.

Telerik

The next commercial tool is Telerik Test Studio. Here’s the pricing:

But every new useful feature is paid additionally:

This amount of 2499 + 899 +599 = $4000 includes no more than a single user and a year of support. After multiplying it by two (the project team consists of two specialists) we got the result of 8000$ per year. Other commercial frameworks are surprisingly the same in costs.  

The Waveaccess team decided to keep searching for a solution that will provide our customer with good quality level at a reasonable price. But the main reason was the need for transparency: our team wants to know what we offer our customer precisely, and how we are going to use this. That’s why we paid attention to open source solutions which we’ll review in the next part of the article.