Test Plan: As defined in IEEE 829-1998, also known as the
829 Standard for software test documentation, test plan is a management
planning document describing the scope, approach,
resources, and schedule of intended testing activities. Test plan identifies test
items, the features to be tested, the testing tasks, who will do each
task, and any risks requiring contingency planning.
The above is a textbook definition of a test
plan. As the definition says, it describes the scope, approaches of the test
that's going to be performed on the target product. This does not describe/
mention anything about one important thing that I learned from my experience,
people may not emphasize on this part is - the Test
Strategy.
So... What is Test Strategy then?-
another textbook description - The test strategy is defined set of
methods and objectives that direct test design and execution. The test
strategy describes the overall testing approach for the
testing of application under test including stages of testing, completion
criteria, and general testing techniques. The test strategy forms the basis for
test plans.
Strategy plays the key role for the planning of
"attacking on software". According to its It defines the testing
approach. And the approach would be, in my opinion, "Attacking the
Software" from every possible angle of a "User". It not
about only test methodologies, its about going beyond normal scenarios,
scenarios that user may create accidentally or intentionally.
Human psychology is an integral
part of testing. A Tester/ QA personnel should approach testing software
thinking of him-self as a user- non tech, mid tech, high tech user. And even
sometimes like crazy and/or baby. Let me mention few examples that can
relate the aspect of human psychology and different behavior from
user point of view:
#1 Lets say I was given to test an old style Cassette player
(if people still can remember what is a Cassette player...). My
general approach would be to test all the functions according to the
conventional test plan (derived from specs). So I would have test cases for
each function, Play, Rewind, Forward, Stop, Record etc. The product is designed
with buttons that user would be able to easily use it- could be a soft-touch,
could be physical button etc. The general expectation (which actually leads
toward limitation) of testing this product would be whether all the functions work properly and as expected or not. Usually, we would assume / design out testing
thinking of user will be using it with "hand". Would anyone/
any tester would think about user may actually want to use it using foot finger? – The
answer could be 1% people or even less. To me, it’s not about the percent, it’s
about whether I thought about it and designed my test cases for the user's like
this or not.
#2 Another example from real life- with the most recent IOS 7 release, one of the bugs that user found is, it allows to call from the lock screen. It takes seven consecutive enters/ click to make it happen. Now who would think about that magic number of tries (and why would even do that in the first place)? This could be a great place to engaged automated testing. This kind of scenario has to be covered up to certain try so that we will have less chance to have it found by the users.
#3 I was working on one Automated Taller Machine
(ATM) project of a bank around the year 2008. They were implementing ATM
interface with the Core Banking Software (CBS). ATM was communicating with
hardware and interfacing with CBS. I was engaged at the time of UAT sign
off. I tried few things that I think customer might do and I was able to create
one scenario. It was more of an implementation issue.
When user sends a command for withdraw money, for
example $100 ($20 x 5 bills), machine do all the checking with CBS and then count
money and give the money through dispense. When money comes out through
dispense, customer takes it or if it stays on dispense for 30 seconds, system
takes it back inside. When it goes it inside, customer can claim that money
later at bank and the use case ends there. The problem arises when I took one
note from dispense and let it go back inside with 4 notes. Now, I'm still
eligible to claim the money and bank has two things- one they had 4 notes in
the tray and a withdraw transaction of 5 notes. The camera at the ATM
counter did not able to give clear view. So, theoretically I'm entitled to
get my $100 (if I don't fall into some text written in smaller font in terms
and condition). This scenario created an issue and people were figuring out how
to resolve it. This can be one of the example how user can do so many things
that the typical test plan would not cover such issue.
#4 Another experience would like to share. This
time it was a simple login issue for a dial-up internet service. In the early days,
around 2004, I used to use one dial-up internet connection from a company. One
of my friends wanted to use my internet connection service for a short period
of time and I gave him my credentials. So, when he uses it, I can’t use it at
the same time. One day, we saw, we both were online! How did happened using one
connection? We logged in at the exact same time- concurrent user. 2 persons
using one in connection at one price! We didn't do not intentionally, just
happened!
I tried to mention some example in different stages of
software- starting from design to implementation phase of the
software. From these examples we can see that human psychology and
behavior is something that needed to be learned/ designed to handle
during testing. Now the question is how can we handle such human
behavior in our test life cycle? I would recommend adding section in the test
strategy that will handle the user behavior based on user’s demographics- age,
knowledge, gender, country etc. Interview user, get to know about their
expectation, observe users point of view for the software. Even though it will
vary user to user, at least we would know that we covered this kind of user
behavior in our testing.
Without having test cases like this, we can’t be just
relying on the word that we are 80% confident of our testing; testing is done
80% etc. I feel the need of this kind of testing be mentioned in the test plan.
We can argue on the priority or the weight-age of this kind of bug/
issues later but it at least gives us more coverage of testing and chances to
minimize the bug.
I believe, our job is to find issues, it's the
products stake holder's job whether to decide to fix these kinds of issues or
to release it with known issues as it involves resources, time and obviously
money!
Off track observation- I have a 2 years old son. I gave my old blackberry and android phone to him. The way he actually operates it, it’s like crazy combination/ sequence. Randomness is his specialty. I tried to follow him and sometimes I wonder, if I were the tester of that product, would I ever think about anything like this in this order?! That's why I mentioned; sometimes we should extend our thinking beyond normal user- go crazy!
That will be all for today. Currently, I'm in the
middle of testing and releasing my product and I should go back from my testing
break.....
I'll have another blog where I'll try to mention at
what stage testing can be started in a
Software Development Life Cycle.
No comments:
Post a Comment