A lot of companies are jumping on the “automation train” for good reason… Automated testing has several benefits including:
- Predictability and Repeat-ability*
- Freedom from time constraints*
- Lower cost*
- Increased Code Coverage*
- Coolness factor
I know the tester in you is asking me why I put the asterisk by (most of) the items above. Well, because, in some cases, you don’t get to reap these benefits after your huge investment in automation. I’m going to talk today about some of the pros and cons of automating testing ‘stuff’ and why you would choose to automate or not to automate a certain “thing”. The only thing without an asterisk is Coolness factor because it is always “cool” to automate 😉
Automation Pros and Cons
As I mentioned above, there are many really cool benefits of automating testing. I have made a career out of finding ways to automate what used to be mundane tasks for testers. So let me list some of the benefits I’ve seen from automation:
Tests can be run quickly. This is true. But, how long will it take you to write the tests? If your deadline to ship is in 1 month and it will take you 3 weeks to write automated tests, automation may not be your best option. While the tests can be run quickly, how close are you to having them ready to roll? If the answer isn’t close enough, you won’t save any time at all. The main point is… Consider the following questions when determining whether automation will help you test more quickly:
- How quickly can you get the tests written?
- How soon do you need to know the results of the test?
- How often will you be using these tests? Can you re-use them next time?
- What is the _real_ time savings? Is this a test a human being could do in less than 1 minute?
Predictability And Repeat-Ability
Automated tests always test the same things every time. Mostly true. Consider the nature of your application. If it changes quite a bit, your test results will be very unpredictable. If you can write data driven tests for a data-driven application, this is usually pretty predictable, but if you are coding to a UI that changes often, you will find the tests are very unpredictable. But, for consistent interfaces, automation does do the same thing every time. You also need to consider whether this meets your needs. One simple example I can think of is if you are a tester of a product that does voice recognition. If you have somehow recorded speech and automated it into your test process, it’s true you will get predictable results… However, you will get ONLY those results. If your goal is to test the recognition, you need a variety of voices, a variety of background noises, etc. Automation can’t fill that need for you. When thinking about predictability of automated tests, think about the following questions:
- Does your application lend itself to repeating the same exact test over and over? Should there be any variation in your test input or your test approach?
- Does your application change a lot such that automation is difficult?
- Are there timing issues that make a difference with how the results of the test will turn out?
- Have you covered testing the “unpredictable” portions of the product or feature as well?
Freedom From Time Constraints
Automated tests can be run overnight, 24-hours a day, 7 days a week. This is a huge benefit. The automated tester can spend 8 hours in a day working on automation and the productivity gains may be tremendous. The automation written in 1 day could be run an infinite number of times with multiple configurations. If done correctly, automation can free up the tester to do other tasks while the automation is running. As long as the tester has made the work truly automated. Some questions to think about here:
- Have we written our automated tests such that they can run independently and without human interaction?
- Do we fully trust our automation to run without us there? (i.e. did we test the test?)
- Do we have the dedicated infrastructure to maintain the automated set-up?
- Is there a way for a human to review the results?
Execution of automated tests isn’t always cheaper. There are costs involved including hardware and software on which to execute your automation, set-up costs, etc. I’ve seen cases where automation of our tests required a purchase of a huge server farm. Was it worth it? Does automation pay off? Think about the long term costs of maintaining the automation PLUS the infrastructure of the automation and compare that to the costs of testing the same things manually. Some things to think about:
- What is the time estimate (read cost estimate) to get the tests planned, written, coded, data creation, etc?
- What is the infrastructure cost? What software or hardware do you need?
- What is the maintenance cost of both items above?
- What is the cost savings if we decide to implement automated tests here?
Increased Code Coverage
People believe that if you have automation, you will automatically also get increased code coverage. This is not necessarily true. The automation must be implemented strategically to ensure you are maximizing the value. As a tester, you know you have limited time to get the testing done. If you spend a huge chunk of it automating a small part of the software, you will have lost out on testing the other areas. Be sure you have analyzed this carefully to determine if it does in fact give you any additional testing over the course of the project.
If all of this makes you change your mind about automation, just think… You could automate stuff in your spare time if you want because it’s just damn cool!