Unannounced changes in code impacts testing

One of the common troublesome thing that I found in my career of software testing is how to handle the changes / change impacts. Incremental testing is a common practice now-a-days (I believe). When we have a build on our hand from developers, we do the testing on it, find issues and report it back to developer. Developer fixes the issue. Now, during this procedure, we expect that developer will not break anything that was working in previous build. Yeah right! We can only EXPECT (no offense to developer). As a human being it’s very much natural to break something while fixing it and that’s why we testers are their fall back that we will check all possibilities. Being human, we also inherit the same nature – may slip to find the issue. So what can be done in this regard? 

Answer would be simple- engage automated testing. Okay, I would accept that. Now can anyone tell me what are the areas/ or which test cases I would run? I would certainly take any answer, other than “Run all test cases” answer. Reason is simple; it may not be reasonable/ suitable/ applicable. 

Let’s try to understand it with a hypothetical example- simple deposit transaction. On a deposit transaction, we deposit the money to a particular bank account. For that we do some checking on the account- account number, name, any other info, any restrictions etc. After that, transaction gets posted to the account. So we tested the possible scenarios of a deposit account. Most of the thing was working fine, except it was not allowing editing the date of birth. So, on a next build, we expect the fix would come around that area. Straight and simple. We tested that particular area and found okay. Later, when we released the build found that it was taking awfully longer time to perform a deposit transaction. So what has happened here? Tester found the issue, developer fixed it, tester verified that particular fix. Tester does not know anything about the code. Does not have access to the code or does not even understand the code. At the end, the client is suffering from a bad build. 

Hmmm. Time to investigate. Backtrack the issue- tester tested that particular issue. Developer fixed that particular issue as well. When developer was in that class/method of fixing the issue, he accidentally commented out one line of code that was necessary / would increase the performance or some code that he felt unnecessary and removed the code (hypothetical scenario).  Tester was not aware of this particular incident.  If we would, we would have been running the performance test cases as well.  

Let’s see, now, we have automated test cases ready for regression testing. If we were to run the automated test cases for such incremental build, would I run the whole process, or just the portion of the deposit and related test cases.  Let’s say run most of the known related test cases. If we do that, at the end the software actually would give us test result as “Pass”. But would miss the point- transaction taking longer time.

The point is, we need to establish a process of clear communication among the testing team and development team about the change/touched areas. My personal opinion to make a matrix that help us to know about the change impacts. 

I would like to propose a matrix with few queries like: 

1. Functionality - What are the possible functionality that can have the impact for the change?
2. Performance -  Any chance to effect the performance of the existing operation?
3. Code Clean up - Does this build contains any cleanup by the developer? 
4. Optimization- Was this done to related to any fix or just a generic code optimization?
5. Bug fix - Was this build was specific to a isolated bug fix?
6. Enhancement - Was there any enhancement done with this build?
7. Common Utility function - Was there any change in the Utility function that was used  as common 
     function?

When developer make a change s/he should mark those parameters that is applicable for that change. May be putting some comments would give testers an idea to narrow down his possible testing areas. This will be the additional information for the tester. Based on the matrix tester then can decide what test cases to run or not. 

I was doing it verbally as I have the setup/ environment like that. Every time I get a new build, after performing all my regular testing, I consult with developer, ask him about the changes he made. I do it nicely so that he shares the information with me without getting offended. Then I check the code from the version control, tries to match up his statement. With my limited knowledge  I try to understand the changes in the code and then decide whether I should run anything more or not. I have been doing this for a long time and it saved my several times. 

This is not a new concept for the software development/testing world. Shared the experiences  that I have been facing for long time. Some companies (for example, Parasoft,  Microsoft Test Manager, Vector Software) already made software/solution that can keep track of this kind of issues. Even though, i'm not sure whether i'll get the answer's about the above mentioned parameters that I suggested for matrix. As long as we maintain a system/ structure that is the main goal. 

It's always better to work as team- development team, testing team. Goal is to Avoid miss communication and perform the task effectively.