Manual Test Cases vs. Automated Test Cases — who wins? Round 2.
About passed Round 1.
About a year ago I wrote an article:
where I explained test cases automation reasons with formulas given from EPAM’s book “Software Testing. Basic course”. That book is available only in Russian, so I made an article in English to make an insight for those who might be interested in the topic. And it has happened that that article got a lot of attention, thousands of views and reads. And that’s why I thought about to continue talks on this “hot” topic.
About Round 2.
During this last year I’ve seen so many cases when people misunderstand manual-vs-auto tests concept — so I just couldn’t stay away. That’s why last week I decided to make that article again, to name it “Round 2” and to explain the same concepts from “Round 1”, yet not with boring and academic formulas, but with simple and clear graphs.
On weekends I’ve made about 20 graphs and posted them here below and I hope you’ll enjoy reading and watching and will take away a lot of it!
Let’s get this thing started!
Round 2. Intro.
Welcome the Battlefield!
If you read materials about manual-vs-auto test cases battles and comparisons and debates, then you probably often face such kind of graphs (for example, in articles like this):
And the main thing that the authors of such graphs try to explain — is that at some moment in time automated test cases become “more profitable” than manual test cases:
— and this is the reason for automation!
Sounds clear, but it isn’t. And you’ll find out why below. Because this picture is too simple for real life. In real life we have much more complicated situations. And that’s why I wanted we to look better on each line and each phase of each line to show what really happens here: do these lines intersect at all? And if yes — then when and at what conditions? And what can we learn from this?
Good and interesting questions, right?
Welcome the Manual test case!
Design vs. Execution
Sometimes we design manual test cases before tested software has got ready for our tests. Sometimes we design manual test cases during execution on a working software. It just doesn’t matter. The order of phases for test case design and for the first test case execution may change. That’s why I united them in 1 initial phase: “design + 1 execution”.
Another thing with manual test cases design and execution is that sometimes execution of manual test case is much harder than manual test case design. And sometimes visa versa: manual test manual case design can be much harder than manual test case execution.
These both statements I depicted this way:
And we can make an average line for both types of executions to make things simpler:
And this is how we get our initial line for manual test case, but now with more understanding about it’s variable nature:
Executions, executions and executions…
Take a look at executions phase! Who told that manual test cases executed this way: execution right after execution?
If you ever worked with manual test cases, then you know, that they never executed this way! Never! Instead, they are executed this way, with a “waiting” phase:
Manual test cases often have this “wait” phase, where they do not consume any resources at all: neither on test case design, nor on execution! And this waiting phase may last from several weeks to several years even! And this is how test cases form a Knowledge Base about our product.
If you think, that we can just omit that waiting phase — so, we can’t. We can’t, because it’s very essential. And you’ll se why on the graphs below.
Real picture for Manual test case
To simplify things, again we can build an average line for “wait + executions”. And we’ll get:
and end up with this — a real image for manual test case lifetime in terms of time and efforts spent on test case design and execution:
So, now we’ve done with Manual test case presentation and it’s time to switch to an automation test.
Welcome an Automatic test case!
Auto tests Executions
Authors of graphs for auto tests tell, that test cases design and debugging of the test for the first time may cost a lot in terms of time and efforts, but then, they just don’t consume anything:
Don’t consume anything, they just run. What a lie! Everyone who worked with auto test cases knows how much does it cost their execution: troubleshooting, investigations of instabilities, redesign, failures or false passes investigations, etc., etc. And how much does it cost their non-execution either: the longer you don’t execute your auto tests the more time and energy you’ll need to run them again when you need.
Let’s put that costs on a graph!
Again, the same thing as with manual test cases: auto tests execution almost never goes this way: execution right after execution:
Auto tests also have their “waiting” phases. They might last for hours, days or even weeks and months, but they exist! And it is essential!
Again, to simplify things, we can introduce an average line:
And here we go — a real image for auto test case lifetime in terms of time and efforts spent on test case design and execution:
Round 2. Fight!
Now I put 2 graphs together:
And where is a fight? Is there any intersections at all?
Yes they are. And they are a number. But find them we need to de-average our lines back to reveal their nature:
And from this picture now we can start playing with these curves to make them intersect and understand situations that might cause that intersections in real life.
And the fight begins!
Reason #1 for test case automation
Regression test cases — are #1 test cases for automation
Namba 1 reason for test cases automation emerges from a situation when you need to execute your test cases either often or you have a lot of them and impact from their non-execution will cost you a lot. And they are, mostly, regression test cases.
And this is the graph that we’ve started from! Remember it? It is! That graph wasn’t a lie, it was just about regression test cases automation! But the truth is that it is not the only case and manner when and how auto tests beat manual tests!
Let’s go further!
Reason #2 for test case automation
Test cases that are easy to be designed, executed and maintained being automated from scratch — are #2 for automation
Sometimes it is really easy, fast and simple to implement and run and support automatic test cases right from the beginning. You have a framework, you have an infrastructure, test case is pretty simple and is going to be run from time to time at least — why to make it manual at all? No reason! More reasons to make them automated right from the start!
Take a look:
Reason #3 for test case automation
Too difficult or impossible to execute test case manually — are #3 for test case automation
Sometimes test case is very easy to design, but it’s execution takes so much efforts for manual execution or even becomes impossible for manual execution (for example, you need to send some protocol packet at a given time frame, and so on) — so it’s better to implement such test cases automatic right from the beginning:
Reason #4 for test case automation
It’s not a technical reason. It’s a business reason. That’s why it’s difficult to demonstrate it on a graph. But I tried. Here it is:
If you are on a “Radio-On-Internet” project — is #4 reason for test case automation
If you don’t know what is a “Radio-On-Internet” project, take a look on this episode:
“Radio-On-Internet” project — is the project where you don’t care about ROI (Return On Investment), about efficiency of everything. This is a project where you just burn someone’s else money. For example, you try to impress investors and tell them stories that you are a super team and you have 100% automation testing and everything is super automated — give us your money!
Round 2. Results
There are situations when Manual test cases beat Auto test cases in terms of speed of getting results. For example, you need to check, that login form works. You’ll check it manually faster, than you’ll write an auto-test for that.
There are situations when you need to check something critical manually. For example, NOC monitoring team on a hourly basis performs manual verifications of life-critical functions for the product: signup, purchase and a basic operation — just to make sure, that everything really works. Because mistake will cost business thousands and thousands of dollars.
There are situations when you don’t need to verify functionality often. For example, you implement some functionality, then check it (1-st test run), then perform regression verifications in a few weeks or months (2-nd run) and then in a few months another regression (3-rd run) — then you probably don’t need automation here and manual test cases are fine.
There are situations when auto tests just very expensive to implement — for example, in IP-telephony where you have tens of devices of different models from different vendors. And in this case you just hire manual tests to do the job you need: to verify some functions on every device.
There are situations when functionality changes very often, just because you experiment. In this case auto tests are also questionable.
But also there are situations (and they’ve been described above) when auto test cases more preferable and they beat manual test cases:
- high priority regression tests
- test cases that are much easier to design, execute and maintain automated right from the beginning
- test cases that are very difficult to execute manually rather than automatically
- when you don’t care about resources and just burn the money of investors
All these results, in a compact view, I’ve put in this article:
Leaving an arena
I hope you found this article interesting and useful. If so, please don’t hesitate to give it some claps — so I could know, that you are interested to read such articles more and more people could know about it.
Thanks!
