Importance of Reporting a Good Bug

Throughout my testing career, some of the key things that I have learned. Today I want to share with one of the key/ primary thing- Importance of Reporting a Good Bug. I'll try to keep it short and clean. 

Less than 24 hours we were supposed to releasing a version of the software. We made some changes on few areas. Every of those issues have been verified fixed by the testing team. At the eleventh hour, all on a sudden we have found that one of the functions is not working. Something that used to working, now with this new version its not working. We were scratching our heads what has happened to it! We looked into the issues and then later found that it was due to change in one area which nobody supposed/ reported to take care of it. When we asked the concerned developer, he said, testing team reported to him about an issue. We looked at the bug tracking system, nothing recorded like that. Then we get to know that it was a it was a verbal conversation between developer and testing team member. This is just one incident. 

In past I have seen many times in my career that testing team members verbally communicates with development team members to discuss the issue found. Later at some point that issue had been fixed before it's recorded. So, when tester finds that it's already been fixed, they do not record it anymore on the bug tracking system. Later, for that change, when it creates problem and the blame game starts. It's good to have that discussion to have a better understanding of the bug, no harm at it. But when things recorded, it's easier to track back to the root issue. I hate to see something gets fixed without any record/ proper reporting of a Issue. Yes, I literally hate it. 

In order to get into the importance of Bug reporting, I would like to start with few basic things.

What is a Bug?

In simple words of mine, when any function does not work as it should work or does not meet as per the written requirement or user expectation, its a bug/ issue (whatever you want to call it so that it eliminates unnecessary controversy to start the conversation).

Bug Life Cycle

Here is pictorial description of bug life cycle. It's a generic. Could vary with different bug tracking tools. 

Figure Bug life cycle

Importance of Reporting a Good Bug

Reporting a good bug is pretty important for a mission critical software/ firmware. One needs to understand some key aspect of it. 

With a good write up of a bug/Issue:  

1. Would directly help the developer to understand the bug

2. Would help the developer duplicate the bug

3. Would help the tester to re-produce the bug for bug verification

4. Assigning the proper Severity helps the developer to under the complexity of the bug

5. Assigning proper Priority of the bug helps the developer to order the bug fixing schedule

6. Helps to identify the history of the changes for bug

7. It helps to identify when and what measure was taken under what circumstances  

I have some mixed experience of Bug/Issue fixing cycle. On some project I have seen time, resource and money allocation for bug/issue fixing, sometimes no explicit time allocation (do not blame Agile!). Depending on the nature of the project budget, resources and time it becomes a very helpful tool if everything recorded properly. 

There is another side of recording good bugs. Reputation. As Captain John Miller said to Private Ryan, "Earn the Right to Live", I would say, it earns the respect/reputation from the developers for that testing team member. 

How-to report a good bug

In short answer, whatever does the best communication among interested parties. We can discuss some conceptual guidelines, it may vary in real world.  

1. Select/ Mention the correct Product/ Project 

2. Select/ Mention the correct  version of the Product/ Project 

3. Select/ Mention the operating system, hardware

4. If possible select the component of the Project. Sometimes it can be broken down to different layers of the     software. For example for a web-based layered system, it can be User Interface, Business Logic, Database       logic/ interface, etc. For device base, it can be different physical hardware component. 

5. Select the correct/ most appropriate severity. (Should always check with BA/SA/PM/Test Manager)

6. Select the correct Priority level. (Should always check with BA/SA/PM/Test Manager)

7. Write down the description using the key issues. (This will help to categories / search on different bug tracking tools). Sometimes, depending on tools, it helps if the operation name is prefixed in the short description of the bug. 

8. In the details of the bug, it helps if the following are give:

Description of the symptoms - More detailed expansion of the summary. Some times minor details also come very handy.    

Steps to reproduce - The minimal set of steps necessary to trigger the bug. Include any special setup steps. 

Actual results - What was the actual behavior after performing the above steps? 

Expected results - What would have been the correct behavior, were the bug not present

 9. Finally, some additional Information - Links, snap-shots, debugging information, any other relevant attachments  

Once the bug has been recorded, it will generate an ID of that issue from the tracking system. Software team then should use this number for the correspondence. 

Identifying/ defining Priority and/or Severity of a Bug
Different people interpret this in different way - Priority & Severity. Most of them, what I have seen, does not give importance (or it may have confused them). This is one of the important area that can help the whole bug fixing process to work a little bit faster. When we can identify them correctly, it will eliminate the need of detail discussion in different meetings. So, I would like to propose the following how to use them effectively.  

When we report a bug (if BA/ business stake holders are not available ), we usually prioritize them according the logical common sense. This priority can/will be revised in a meeting with the stakeholders (BA/SA/BM/PM/SM/Testing manager). Starting from P1 (highest priority) to P5 (lowest). This should be a non-editable field for the developers. Because business will decide which will be fixed in which order. 

On the other hand, any issue that is tough, complex, will take more time, has huge change impact etc- those are defined under the Severity of the bug. Again, as a tester we will report them with our logical common sense, but developers will have the chance to change the level of the severity and can consult with their team leader/ BM/SM/PM. If the severity is critical then consulting Business Manager it may come down to a P5 issue depending on the scenarios. 

Category of Severity: 

     Here are some brief descriptions of Severity
    
1. Show Stopper – It is impossible to continue testing because of the severity of the defect
2. Critical - Testing can continue but the application cannot be released into production until this 
defect is fixed
3. Major - Testing can continue but this defect will result in a severe departure from the business requirements if released for production
4. Normal -Testing can continue and the defect will cause only minimal departure from the business requirements when in production
5. Minor– Testing can continue and the defect will not affect the release into production. The defect should be corrected but little or no changes to business requirements are envisaged
6. Trivial– Minor cosmetic issues like colors, fonts, and pitch size that do not affect testing or 
production release. If, however, these features are important business requirements then they will receive a higher severity level.

I know its very tough to follow such details description for in issue, it will help a lot to the developer. And actually, it will help you at the end. In some cases, we report a bug of a product/ project, then we move on to some other. When we come back, its tough to recall all the details, unless it well written in the bug description.

Better to avoid mentioning what you think, give based on how and what happened. Try to give description what how happened. Sometimes some information you may not feel relevant for you, but it could we the key clue that you will be giving to the developer.

Motivation for Software Testing

Motivation- a text book definition- is a psychological feature that arouses an organism to act towards a desired goal and elicits, controls, and sustains certain goal-directed behaviors. It can be considered a driving force; a psychological one that compels or reinforces an action toward a desired goal.

Software Tester/ Software Quality Assurance Engineer/ Firmware Test Engineer etc all these are not that easy as it sounds. Software Testing can be a boring, repetitive task. Many people may not find it interesting enough to continue their career in this field. Many people would think it as 2nd level job (in terms of opportunity and ~20%-30% less payment from others) in software industry. A job that usually comes in shorter months (3 months, 6 months+ etc). It’s very hard to motivate him-selves in job like this and produce quality work. Testers who are motivated are more likely to strive for higher quality outcomes, achieve objectives and work in a more efficient manner and vice versa.

Think about a simple situation- you are working with a device, for example medical device. Medical devices are critical. They are used on patients, not on ordinary people. When you engaged with such job, and you have tested the firmware/ software of that machine, when it comes to the critical time, you know that machine will work; you know you tested it well. It can even save a life and opposite can happen if you are not self-motivated enough, not being able to understand the importance of your job. 

Or think about another scenario- in August 2005, Malaysia Airlines Jetliner, Flight between Perth, Australia and Kualalampur, Malaysia zoomed 3,000 feet upwards. A defective software program had provided incorrect data about the aircraft’s speed and acceleration, confusing flight computers. For a software bug, think about how many people’s life was at risk!

I have chosen to be a software/ firmware test engineer. To me it’s a challenging one. Every time a code is released for testing from the developers, we get a chance to break it, tear it down into pieces. To me it’s interesting, fun and I enjoy doing testing. It does always make feel good to developer's reaction after finding something; developer never thought about it- that's my primary motivation.

Real life experience
I would like to share one experience of mine how I tried to motivate the QA team, i.e. the whole software development team 

Around 2003, I joined as a member of the QA team which had 4 members. There were 15 developers and 4 testing team members. We had a QA team leader who was an expert of databases. They had all the procedure and tools for the testing team in places. From screening codes at the very early stage of software to recording bugs in Bugzilla. What I had experienced during my tenure was, somehow I felt that testing team member were not motivated enough to proactively do something. I felt like, they treat themselves as weaker than the development team.

So, based on that feeling, even though I was just a member of the testing team, I felt like I had to do something in this regard. So, I initiated few internal meetings with our QA team about the existing procedure/ bugs that we had. After that meeting I found the weaker point. QA team was lagging of one main thing – “Quality”. I looked at few bugs and realized why developers used to treat them so lightly. The quality of the bug reporting was poor, not enough description, not enough logic to support that it was a valid bug. Then decided to shared my experience with them, how to write a good bug and standardize the procedure. I noticed that, developers started to like it. Because it helped them, it helped them to duplicate the issue which saved lot more time.

Observing the good response from the development team, I looked the issue to motivate the QA team to keep them reporting the quality bugs. I started another approach for the QA team and later with the development team. That was something that I started to use 10 years back from now, nothing unique about it. It’s the "Rewards" system. Here, I will try to explain how on each phase Reward system how it worked and not any specific score card system.

I started to publish the statistics of QA team in terms of reporting bugs on the notice board, like the top ten chart. That was intended to motivate the QA team and it did! Not to praise myself, but I was top on that chart :). It actually started to work, when developers acknowledged my bugs. My other QA team members actually started to post more and more bugs after that. It became a sort of competition and following weeks numbers were different then the first one.

Reporting more bugs it-self does not generate good reputation for the testing team. As they were reporting issues, if they did not report legitimate bug, it was getting negative response from developers. So, to balance that issue, I put some weight-age on the bugs. What I did was, for each good bug of P1, which was accepted by the developer/lead, gets 10 points. Then for P2 accepted bugs, 8 points, P3 -6 points, P4 -4 points, P5- 2 points. To ensure that QA team would not report everything for higher score, I had used the negative marking as well. For each bug that gets rejected by developer reduced the -5, -4, -3, -2, -1 points. At the end we had a sum of all the numbers. Based on the number it was ranked. The good thing about negative marking was, when a QA team person reported 10 point bugs, when it gets rejected by the developer, he lost points. So they were careful about reporting it. That's one part of the solution- motivated the QA team.

Then comes the other part. As of then I had more bugs on board, developers need to fix them. It was outside of my scope. So approached to our Project/ Engineering Manager- who manages the whole Software Development team. With his permission, then started to introduce another section on my chart for the development team regarding the bug found/fixes. Followed the same positive and negative point system for them as well. Ranking of developers in terms of most issued bugs. When I published that, now developers were more serious about getting those issues fixed. So it helped in both ways, to report bugs and to had them fixed. After a while when I published the score card again, I had seen major changes on the chart.

It worked! Simple old fashion Reward system worked. It was a whole team effort to make that system actively working. Reward technique for this project worked well. But it may not work well with other project/ infrastructure. We can introduce different techniques like this which will be suitable for the project/ infrastructure. Motivation differ from person to person. Money may be the prime driving factor, but something like this small reward system can also motivate some group of people.

Few common barriers of Software testing can be easily handled in a different way. Software testing is not a repetitive job anymore. We can minimize it by engaging the automated testing. As a testing team member/ manager, we should establish/ follow a process of systematic testing, setting the goals for the testing team members, making sure they are clear about their goals and boundaries. Managers can provide training for the test team members if necessary, can have few motivation sessions with test team members. When necessary, don't feel shy to play the leadership role. Develop a good relationship with development team. That can help you to understand any issue with more detailed information (DO NOT get Biased though- it's important).

Lastly I would like to say, we are test engineers. We test things. Developers have an option to fail and we will catch it. But, if we fail, there will be no 2nd level. It will directly go to the end users. And the cost is real high when it fails on Releases.

Be a passionate test engineer!!! 

Software Testing @ what stage of SDLC

A Software Development Life Cycle (SDLC) model represents all the activities and work products necessary to develop a software system. Life cycles models make the software development and dependencies visible and manageable.

Conventional waterfall has the phases in SDLC are:
1. Requirement & analysis
2. Design
3. Development (Coding)
4. Integration/Testing
5. Maintenance

From above, we can see that testing starts after the development of some functionality. Testing team is not involved in any of the first two steps. This is how I have seen in my career. Could be different in other companies, but I can guesstimate it will be more or less for the smaller to medium size companies. 

In waterfall/ modified waterfall, iteration method, design team designs the software based on the specs. Specs, are not always written with every scenarios, especially the odd ball scenarios are not present in the Specs. So what happen is, when we start testing those odd ball scenarios, it falls into a gray area. According to developer, it’s not in specs and as a tester (represents users of the real world) it’s a problem and it get stuck in some gray area and sometimes it involves one level upper position people to resolve this (unnecessary) dispute. 

We can minimize this type of scenario if we could involve the testing team members who will give input to the requirement/ design team from user's perspective. Many things can be discussed at the beginning so that it can be prevented to occur at the first place. 

I would like to share few of my experience here. (I believe testing team is the middle man between developer and user. So, testing will, in SDLC will represent user.)

#1. I worked on a core banking project. They were developing a portal interface. Design team designed a web interface that will take some input and will submit it to the database.  I was not part of the initial design team. I was invited to participate on their final design review. I looked at it. The first thing that struck my mind is what if I create an automated process and dump garbage data. Clearly that was not handled in the design. I'm talking about Challenge key. Even though it’s a very common thing now a days to handle, it wasn't like 10/15 years ago. Designed team then realize the issue and they added additional thing to protect it. 

#2. Another simple example. We had one product that was already in the production. It was decided to add a new feature to the existing system. So, the user, BA and developer was there. Testing team was not there. Developers implemented the feature based on the verbal interactions with the user. (Agile allows not to have comprehensive specs. That is more emphasize on human interaction less on papers. Hmm, sounds like a haven for the people who hates to document). Then it was given to the testing team. The first thing that testing team found was they broke the existing system. It was quickly reported and then fixed by developer. I think, this can be called an iteration. I was wondering in my mind, was this iteration was necessary? One the first thing testing team did check whether it broke any functionality or not. It could have been discussed during the brainstorming session. It was skipped (blaming the Agile!). It defiantly incurred some time and money. 

I can continue with few more examples like this, but I think the point is clear that we will have to have test plan for all phase of SDLC. This is what I felt at the very early stage of my career as tester. Apparently I was not the only one felt for it. So, the software testing community introduced some modification/ improvement of the traditional system. 

V-Model design is such an improvement of the existing waterfall method and can also be applied, to some extent on Iterative or Agile method. This model is based on association of a testing phase for each corresponding development stage. This means that for every single phase/iteration in the development cycle there is a directly associated testing phase. I have chosen this one as it visually represents, easy to understand the concept of engaging testing in each phases, not necessarily the sequential phases of waterfall.

V- Model design:

(Photo credit – NaveenKumar Namachivayam / Google search..)

  

Following are the short description of validation phases in V-Model:
(I liked the description of Tutorials Point site. So I’ll be quoting following few para from them. http://www.tutorialspoint.com/sdlc/sdlc_v_model.htm)

Following are the Validation phases in V-Model:

Acceptance Testing: Acceptance testing is associated with the business requirement analysis phase and involves testing the product in user environment. Acceptance tests uncover the compatibility issues with the other systems available in the user environment. It also discovers the non functional issues such as load and performance defects in the actual user environment. 

System Testing: System testing is directly associated with the System design phase. System tests check the entire system functionality and the communication of the system under development with external systems. Most of the software and hardware compatibility issues can be uncovered during system test execution.

Integration Testing: Integration testing is associated with the architectural design phase. Integration tests are performed to test the coexistence and communication of the internal modules within the system.

Unit Testing: Unit tests designed in the module design phase are executed on the code during this validation phase. Unit testing is the testing at code level and helps eliminate bugs at an early stage, though all defects cannot be uncovered by unit testing.

Ok, lets says those are the steps in waterfall method which has many limitations. Now keeping these phases in mind, lets take a look at what Agile method is trying to say in this regard. From Microsoft's Testing Methodologies topic (http://msdn.microsoft.com/en-us/library/ff649520.aspx)- 

"In Agile, People and interactions are emphasized, rather than processes and tools. Customers, developers, and testers constantly interact with each other. This interaction ensures that the tester is aware of the requirements for the features being developed during a particular iteration and can easily identify any discrepancy between the system and the requirements." 

Agile basically in favor of engaging testing in all phases (if I interpret it correctly). There is an advantage of Agile method- customer is a part of team. So, the gap between the developer and customer being reduced here. Customer does not always be there and does not necessarily understand all technical thing of the software. So, testing team still can be a bridge between developer and customer. Testing team also gets the advantage of knowing the user, user's requirement,  expectation and importantly, user's psychology. 

Another method in SDLC is Iterative method. It says, according to Microsoft, " the development model breaks the project into small parts. Each part is subjected to multiple iterations of the waterfall model. At the end of each iteration, a new module is completed or an existing one is improved on, the module is integrated into the structure, and the structure is then tested as a whole."- (http://msdn.microsoft.com/en-us/library/ff649520.aspx). Which in other words supports the above model, but in smaller iteration. 

Figure: Quality Assurance in the Iterative Model
Credit: snyders.us

Unified Process
Another popular methodology used by Rational Unified Process (RUP) and is performed in an iterative and incremental manner. The life cycle of the UP is presented in following Figure. 

UP divides the project into four phases which are shown above in Figure and discussed below[3]:

• Inception – By the end of this process a business case should have been made; feasibility of the project assessed; and the scope of the design should be set.

• Elaboration – In this phase a basic architecture should have been produced and a plan of construction agreed. Furthermore, a risk analysis takes place and those risks considered to be major should have been addressed.

• Construction – This process produces a beta-release system. A working system should be available and sufficient enough for preliminary testing under realistic conditions.

• Transition – The system is introduced to the stakeholders and intended users. It is crossed when the project team and the stakeholders agree that the objectives agreed in the  inception phase have been met and the user is satisfied


Some of the key features of the UP are as follows [2]:

It uses a component based architecture which creates a system that is easily extensible, promotes software reuse and intuitively understandable. The component commonly being used to coordinate object oriented programming projects.

• Uses visually modeling software such as UML – which represent its code as a diagrammatic notation to allow less technically competent individuals who may have a better understanding of the problem to have a greater input.

• Manage requirements using use-cases and scenarios have been found to be very effective at both capturing functional requirements and help in keeping sight of the anticipated behaviors of the system. 

• Design is iterative and incremental – this helps reduce project risk profile, allows greater customer feedback and help developers stay focused.

• Verifying software quality is very important in a software project. UP assists in planning quality control and assessment built into the entire process involving all member of the team

Waterfall, Iterative, Agile method or RUP- I personally don't think any of the model is good or bad. It depends on project environment , company structure, budget, nature of business etc. In short,  whichever model is more suitable for the company will be the good one.

Management may argue with this concept to engage Testing Team members at Requirement level and/or Design level which may add cost resource, time and cost, but that certainly will minimize the Testing Life Cycle cost by some early measurement / detection. 

References:
[1] Microsoft's Testing Methodologies topic (http://msdn.microsoft.com/en-us/library/ff649520.aspx

[2] B. Grady, C. Robert, J. Newkirk, Object Oriented Analysis and Design with
Applications, 2nd edition, Addison Wesley Longman, 1998

[3] P. Kruchten, “What is Rational Unified Process?”, The Rational Edge,
http://www.therationaledge.com/content/jan_01/f_rup_pk.html Accessed 2/2/2005

Test Plan and User (Human) Psychology & behavior

Test Plan: As defined in IEEE 829-1998, also known as the 829 Standard for software test documentation, test plan is a management planning document describing the scope, approach, resources, and schedule of intended testing activities. Test plan identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. 

The above is a textbook definition of a test plan. As the definition says, it describes the scope, approaches of the test that's going to be performed on the target product. This does not describe/ mention anything about one important thing that I learned from my experience, people may not emphasize on this part is - the Test Strategy.  

So... What is Test Strategy then?- another textbook description - The test strategy is defined set of methods and objectives that direct test design and execution. The test strategy describes the overall testing approach for the testing of application under test including stages of testing, completion criteria, and general testing techniques. The test strategy forms the basis for test plans. 

Strategy plays the key role for the planning of "attacking on software". According to its It defines the testing approach. And the approach would be, in my opinion, "Attacking the Software" from every possible angle of a "User".  It not about only test methodologies, its about going beyond normal scenarios, scenarios that user may create accidentally or intentionally. 

Human psychology is an integral part of testing.  A Tester/ QA personnel should approach testing software thinking of him-self as a user- non tech, mid tech, high tech user. And even sometimes like crazy and/or baby.  Let me mention few examples that can relate the aspect of human psychology and different behavior from user point of view:

#1 Lets say I was given to test an old style Cassette player (if people still can remember what is a Cassette player...). My general approach would be to test all the functions according to the conventional test plan (derived from specs). So I would have test cases for each function, Play, Rewind, Forward, Stop, Record etc. The product is designed with buttons that user would be able to easily use it- could be a soft-touch, could be physical button etc. The general expectation (which actually leads toward limitation) of testing this product would be whether all the functions work properly and as expected or not. Usually, we would assume / design out testing thinking of user will be using it with "hand".  Would anyone/ any tester would think about user may actually want to use it using foot finger? – The answer could be 1% people or even less. To me, it’s not about the percent, it’s about whether I thought about it and designed my test cases for the user's like this or not. 

#2 Another example from real life- with the most recent IOS 7 release, one of the bugs that user found is, it allows to call from the lock screen. It takes seven consecutive enters/ click to make it happen. Now who would think about that magic number of tries (and why would even do that in the first place)? This could be a great place to engaged automated testing. This kind of scenario has to be covered up to certain try so that we will have less chance to have it found by the users. 

#3 I was working on one Automated Taller Machine (ATM) project of a bank around the year 2008. They were implementing ATM interface with the Core Banking Software (CBS). ATM was communicating with hardware and interfacing with CBS. I was engaged at the time of UAT sign off. I tried few things that I think customer might do and I was able to create one scenario. It was more of an implementation issue. 

When user sends a command for withdraw money, for example $100 ($20 x 5 bills), machine do all the checking with CBS and then count money and give the money through dispense. When money comes out through dispense, customer takes it or if it stays on dispense for 30 seconds, system takes it back inside. When it goes it inside, customer can claim that money later at bank and the use case ends there. The problem arises when I took one note from dispense and let it go back inside with 4 notes. Now, I'm still eligible to claim the money and bank has two things- one they had 4 notes in the tray and a withdraw transaction of 5 notes. The camera at the ATM counter did not able to give clear view. So, theoretically I'm entitled to get my $100 (if I don't fall into some text written in smaller font in terms and condition). This scenario created an issue and people were figuring out how to resolve it. This can be one of the example how user can do so many things that the typical test plan would not cover such issue. 

  

#4 Another experience would like to share. This time it was a simple login issue for a dial-up internet service. In the early days, around 2004, I used to use one dial-up internet connection from a company. One of my friends wanted to use my internet connection service for a short period of time and I gave him my credentials. So, when he uses it, I can’t use it at the same time. One day, we saw, we both were online! How did happened using one connection? We logged in at the exact same time- concurrent user. 2 persons using one in connection at one price! We didn't do not intentionally, just happened! 

I tried to mention some example in different stages of software- starting from design to implementation phase of the software. From these examples we can see that human psychology and behavior is something that needed to be learned/ designed to handle during testing.   Now the question is how can we handle such human behavior in our test life cycle? I would recommend adding section in the test strategy that will handle the user behavior based on user’s demographics- age, knowledge, gender, country etc. Interview user, get to know about their expectation, observe users point of view for the software. Even though it will vary user to user, at least we would know that we covered this kind of user behavior in our testing.    

Without having test cases like this, we can’t be just relying on the word that we are 80% confident of our testing; testing is done 80% etc. I feel the need of this kind of testing be mentioned in the test plan. We can argue on the priority or the weight-age of this kind of bug/ issues later but it at least gives us more coverage of testing and chances to minimize the bug. 

I believe, our job is to find issues, it's the products stake holder's job whether to decide to fix these kinds of issues or to release it with known issues as it involves resources, time and obviously money!

Off track observation- I have a 2 years old son. I gave my old blackberry and android phone to him. The way he actually operates it, it’s like crazy combination/ sequence. Randomness is his specialty. I tried to follow him and sometimes I wonder, if I were the tester of that product, would I ever think about anything like this in this order?! That's why I mentioned; sometimes we should extend our thinking beyond normal user- go crazy! 

That will be all for today. Currently, I'm in the middle of testing and releasing my product and I should go back from my testing break.....  

I'll have another blog where I'll try to mention at what stage testing can be started in a Software Development Life Cycle.