Why "Test Risk"

Recently I have had lots of interviews for QA Analyst roles in the company. We called engineers from the level of beginner to senior test engineers for these roles. One of the most important questions in our interview is for understanding why we are testing. Actually this question is more about the philosophy of the testing, as we think the philosophy is generally not liked by people, but everyone who does something about a subject everyday from 9 to 6 should think about why he does this job. I wonder whether the testing job is just only to feed himself or he has some other passion about his profession. That's why this question is very important for me. One of the training given by Ståle Amland which is "Exploratory Testing – Risk-Based Agile Testing", he explains the philosophy of testing as epistemology of testing and he is saying that "all good testers should  practice Epistemology", see these slides in his training:

What is epistemology is according to Amland, "Epistemology is the study of how we know
what we know. The philosophy of science
belongs to Epistemology. 

All good testers practice Epistemology.and the basic skills of epistemology:
  • Ability to pose useful questions.
  • Ability to observe what’s going on.
  • Ability to describe what you perceive.
  • Ability to think critically about what you know.
  • Ability to recognize and manage bias.
  • Ability to form and test conjectures.
  • Ability to keep thinking despite already knowing.
  • Ability to analyze someone else’s thinking.

In this post I want to explain my thought about test and risk.

Why we are testing? or why companies need testing? or why someone pays us money for testing? Have you ever asked this question yourself?

If you have not tried to answer this question, it is time for it. The simplest answer to this question is “because there are risks”. Everything started in favour of reducing the risks. You never know how much risk you would take if you took a production to live in front of real users. Therefore we should always consider the Risk while we are testing anything. Risk Based Testing helps us to construct an approach for including risks into testing effort so risk based testing is not an method or technic, it is an approach that you use it while applying any testing technics.

What is Risk

Risk is general terms for the probability of failure, see the definition here. However, in terms of testing in software industry, Risk is the probability of the damage that can be occurred by a failure in live. Therefore it has two parameters: the first one is the probability of a defect, what steps produces the defect, how much percentage of the user might possibly see it; and the second parameters is the damage, when a user see it how the system responses for further action. So let's define the formulation of risk:

Risk = (Likelihood of Failure) X (Damage of Failure)

Let's look at an example for calculating risks. For e-commerce web application, We have the following cases:

Practicing the Risk Calculation

Let's assume that the following defects we have in the production, these are most probably the major defect so they should be solved asap but they are good examples for calculation:

Defect 1: User can not add an address when he goes to the step of selecting address
Defect 2: User can not use the bonus of X type credit card on check-out step

Analysing the Risk Factors for Defect 1:

Likelihood: We need statistical data that can helps us to predict user behaviours, let's so the data converting from analytical tools.  1/10 users adds new address during order; 1/20 newly registered user who doesn't have address so they have enter new address; 1/100 users add new address from his membership page, that functionality works good. Let's 1000 users use the system and 1/10 users order products. 

Affected User Description Effect to System
1000 * 1/10 100 users order products +
1000 * 1/100 10 users add new addresses from account page, these user will be happy and doesn't affect the probability no-effect
100 * 1/20 5 new users -
100 * 1/10 10 users want to add a new address -

  • 15/100 users, in ordering process, are possibly affected
  • 15/1000 users, in the system, are possibly affected
the likelihood is 15/1000 * 100 = 1.5%, which means 1.5% of the users might be affected by defect 1.

Damage: The cost of failure changes depends on the sector that the application used by. In general it could be money lost, reputation decrease, causing injuries or even killing people. For e-commerce we can think about money and reputation lost. This defect stops the purchasing process, so we can give the highest point of damage. If scale the damage from 1 to 10, this should be 10. However this classification should be agreed with all stack-holders, for more about this look at about test strategy in this post.  

Risk of Defect 1:  1.5 X 10 =>  15 

Analysing the Risk Factors for Defect 2:

Likelihood: We have 1000 users, 1/10 users go to order, 1/50 users use X-Type credit card, 1/10 of that user use bonuses

Affected User Description Effect to System
1000 * 1/10 100 users order products +
100 * 1/50 2 users use X-type Credit cards no-effect
2 * 1/10 0.2 users use bonuses -

  • 0.2/100 users, in ordering process, are possibly affected
  • 0.2/1000 users, in the system, are possibly affected
the likelihood is 0.2/1000 * 100 = 0.02%, which means 0.02% of the users might be affected by defect 2.

Damage: cost of failure should be same with defect 1 by assumption for the damage of defect 1. Damage should be 10.

Risk of Defect 2:  0.02 X 10 => 0.2

This is just an introductory post and the next post will be related to "Risk Based Testing".


Popular posts from this blog

Testing WEB Sevices and Automating SOAP Services

Performance Testing on CI: Locust is running on Jenkins

Change Default Timeout and Wait Time of Capybara

Performance Testing on CI: Integration of Locust and Jenkins

Create an Alias for Interactive Console Work: Selenium and Capybara