100% tested and yet failed 100%

In response to my blog about the World Cup software testing, I was interviewed by a journalist from the Stentor in late December. Her final question was: How do you recognize a good software tester? I realized this was quite a difficult question to answer. The reason for this is that software is complex and does not resemble any other software. A good testing method for one app does not necessarily make sense for another. Best practices are therefore not recommended. Due to its complexity, there is (almost) never a cause-and-effect relationship. How do you prevent a product from failing when it is tested extensively?

The key lies in the context: for whom, at what time, through which device, under what circumstances and for what purpose do you make the product? These are questions that you need to know the answer to before you can effectively start testing the software. If you know the context, there are actually two questions you will need to answer as a tester:

1. Have I built the right product?

2. Did I build the product in the right way? 

 Question 1 aims to do research into whether your app or website is a solution to the problem. For example: KLM wants to increase the number of repeat customers by ten percent. To achieve this, the airline has built a module for existing passengers to easily re-book a flight to a previous destination. This could be useful for regular passengers such as businessmen, you might say. A software tester uses question 1 to examine whether this solution actually increases the number of repeat customers. Maybe things turn out to work differently in practice from what was expected or maybe a different approach is needed in order to increase the number of those customers by 10%”.

Question 2 deals with the technical side of the software. Let’s stick with the KLM example: does the module work in the way it should work? Does the module truly enable travellers to easily re-book a flight? To detect bugs, software testers find possible improvements. At this stage software testers can be compared to a flashlight: they shine a light on the errors. 

 It proves to be difficult to find a winning formula for both question 1 and question 2. After all, for KLM software there are many different aspects that are of importance compared to say a word game puzzle app. In addition, there are also many methods that can be used to detect errors and opportunities. For this reason, software testing is a creative craft. There is one rule of thumb: try out different methods and see what works. And remember, context eats process for breakfast. In other words, think carefully about what you want to test and why.

 

To become a good – or even better – software tester, it is important to have good reading skills and analytical skills and it is important not to jump to conclusions. Be wary of people who say, “That has been going well for years” or “we have always done it this way”. A critical perspective is important since perhaps there is a (much) better way. Ask the right questions and listen carefully to the answers. Have patience and, last but not least, make sure you have good communication skills so you can properly convey your findings to the customer and your team members. Software testing is not an individual operation; it is a team sport. Human skills and interests are therefore required.

That being said, there are of course a number of tricks that I use to answer question 1 and/or 2. And since I would like to share my knowledge with others, I will discuss five commonly used methods:

  1. Ask a real user what he thinks of the application
    It may sound simple, but this trick is often forgotten. Usually software is designed to enable end-users of that software to do something with it. Who else would be better to ask than the end-user whether the product is working properly? Fully understanding the pros and cons is sometimes harder than you think if you are not an intended user of the application.
  2. Play around with the reliability of an internet connection 
    Many apps and websites assume that an internet connection is consistently good. For example, an internet connection may be required to contact a database, to make a service call or for file storage. For many apps and websites, however, it can be very important that everything continues to work if the internet connection is lost for a moment. Therefore you should try the app under these circumstances, for example when you take the elevator or on a train ride. Does everything still work properly?
  3. Start all over 
    Software testing can be frustrating sometimes. You feel that you are onto something, but it just does not work. The application starts to slow down and you find you are getting stuck. Often in such a moment you have dived too deeply into the matter. A clever trick is to use this emotion and frustration as a trigger to take a few steps back and start again. This will allow you to get back to the app or website with a fresh look.
  4. Take a tour
    If you get stuck or just want to test something else, taking a tour is a nice thing to try. One example is the couch potato tour. Here the aim is to accomplish your task as a user while doing this in the laziest way possible. You can do this either by clicking nothing but the next icon and only filling in something if doing so is necessary to continue. See what happens. Under this and this link you can find more types of tours, including the money tour and the FedEx tour.
  5. Use silly names
    Unwanted data is often a cause for a crash or unwanted behaviour of an app or website. Try to test out what happens if you use Russian or Chinese characters for specific input fields. There are several tools to generate crazy names. I use Bugmagnet, a plug-in for Chrome. I highly recommend it!

As you can see, due to the complexity of the context, it is impossible to test an app 100%. However, by using these 5 tricks you get good insight into two questions: “Have I built the right product?” and “Did I build the product in the right way?” And that’s what you want to know!