Skip to main content

Why "Test Risk"

Image
By


Recently I have had lots of interviews for QA Analyst roles in the company. We called engineers from the level of beginner to senior test engineers for these roles. One of the most important questions in our interview is for understanding why we are testing. Actually this question is more about the philosophy of the testing, as we think the philosophy is generally not liked by people, but everyone who does something about a subject everyday from 9 to 6 should think about why he does this job. I wonder whether the testing job is just only to feed himself or he has some other passion about his profession. That's why this question is very important for me. One of the training given by Ståle Amland which is "Exploratory Testing – Risk-Based Agile Testing", he explains the philosophy of testing as epistemology of testing and he is saying that "all good testers should  practice Epistemology", see these slides in his training:

What is epistemology is according to Amland, "Epistemology is the study of how we know
what we know. The philosophy of science
belongs to Epistemology. 

All good testers practice Epistemology." and the basic skills of epistemology:
  • Ability to pose useful questions.
  • Ability to observe what’s going on.
  • Ability to describe what you perceive.
  • Ability to think critically about what you know.
  • Ability to recognize and manage bias.
  • Ability to form and test conjectures.
  • Ability to keep thinking despite already knowing.
  • Ability to analyze someone else’s thinking.

In this post I want to explain my thought about test and risk.

Why we are testing? or why companies need testing? or why someone pays us money for testing? Have you ever asked this question yourself?

If you have not tried to answer this question, it is time for it. The simplest answer to this question is “because there are risks”. Everything started in favour of reducing the risks. You never know how much risk you would take if you took a production to live in front of real users. Therefore we should always consider the Risk while we are testing anything. Risk Based Testing helps us to construct an approach for including risks into testing effort so risk based testing is not an method or technic, it is an approach that you use it while applying any testing technics.

What is Risk

Risk is general terms for the probability of failure, see the definition here. However, in terms of testing in software industry, Risk is the probability of the damage that can be occurred by a failure in live. Therefore it has two parameters: the first one is the probability of a defect, what steps produces the defect, how much percentage of the user might possibly see it; and the second parameters is the damage, when a user see it how the system responses for further action. So let's define the formulation of risk:

Risk = (Likelihood of Failure) X (Damage of Failure)

Let's look at an example for calculating risks. For e-commerce web application, We have the following cases:

Practicing the Risk Calculation

Let's assume that the following defects we have in the production, these are most probably the major defect so they should be solved asap but they are good examples for calculation:

 Defect 1: User can not add an address when he goes to the step of selecting address
Defect 2: User can not use the bonus of X type credit card on check-out step

Analysing the Risk Factors for Defect 1:

Likelihood: We need statistical data that can helps us to predict user behaviours, let's so the data converting from analytical tools.  1/10 users adds new address during order; 1/20 newly registered user who doesn't have address so they have enter new address; 1/100 users add new address from his membership page, that functionality works good. Let's 1000 users use the system and 1/10 users order products. 


Affected User Description Effect to System
1000 * 1/10 100 users order products +
1000 * 1/100 10 users add new addresses from account page, these user will be happy and doesn't affect the probability no-effect
100 * 1/20 5 new users -
100 * 1/10 10 users want to add a new address -

  • 15/100 users, in ordering process, are possibly affected
  • 15/1000 users, in the system, are possibly affected
the likelihood is 15/1000 * 100 = 1.5%, which means 1.5% of the users might be affected by defect 1.
Damage: The cost of failure changes depends on the sector that the application used by. In general it could be money lost, reputation decrease, causing injuries or even killing people. For e-commerce we can think about money and reputation lost. This defect stops the purchasing process, so we can give the highest point of damage. If scale the damage from 1 to 10, this should be 10. However this classification should be agreed with all stack-holders, for more about this look at about test strategy in this post.  
Risk of Defect 1:  1.5 X 10 =>  15 

Analysing the Risk Factors for Defect 2:

Likelihood: We have 1000 users, 1/10 users go to order, 1/50 users use X-Type credit card, 1/10 of that user use bonuses

Affected User Description Effect to System
1000 * 1/10 100 users order products +
100 * 1/50 2 users use X-type Credit cards no-effect
2 * 1/10 0.2 users use bonuses -

  • 0.2/100 users, in ordering process, are possibly affected
  • 0.2/1000 users, in the system, are possibly affected
the likelihood is 0.2/1000 * 100 = 0.02%, which means 0.02% of the users might be affected by defect 2.

Damage: cost of failure should be same with defect 1 by assumption for the damage of defect 1. Damage should be 10.

Risk of Defect 2:  0.02 X 10 => 0.2

This is just an introductory post and the next post will be related to "Risk Based Testing".

Labels:Software testing

Popular posts for software testing and automation

Selenium Error "Element is not currently interactable and may not be manipulated"

By
Selenium webdriver can drive different browsers like as Firefox, Chrome or Internet Explorer. These browsers actually cover the majority of internet users, so testing these browsers possibly covers the 90% of the internet users. However, there is no guaranty that the same automation scripts can work without a failure on these three browsers. For this reason, automation code should be error-prone for the browsers you want to cover. The following error is caught when the test script run for Chrome and Internet Explorer, but surprisingly there is no error for the Firefox. Selenium gives an error like below: Traceback (most recent call last):   File "D:\workspace\sample_project\sample_run.py", line 10, in <module>     m.login()   File "D:\workspace\ sample_project \test_case_imps.py", line 335, in login     driver.find_element_by_id("id_username").clear()   File "C:\Python27\lib\site-packages\selenium-2.35.0-py2.7.egg\seleni...
Read more

Change Default Timeout and Wait Time of Capybara

By
One of the biggest challenge for automation is handling timeout problem. Most of the time, timeout is 60 seconds but it may sometimes not enough if you have badly designed asynchronous calls or the third party ajax calls. This makes handling timeout more complex. set large enough to tolerate network related problems. For Selenium based automation frameworks, like Capybara, default Webdriver timeout is set to Net::ReadTimeout (Net::ReadTimeout) Changing ReadTimeout If you have timeout problem for Capybara, it gives an error like above. This means that the page is not fully loaded in given timeout period. Even you can see that page is loaded correctly but webdriver wait until the Ajax calls finish. class BufferedIO #:nodoc: internal use only def initialize (io) @io = io @read_timeout = 60 @continue_timeout = nil @debug_output = nil @rbuf = '' end . . . . . def rbuf_fill beg...
Read more

Create an Alias for Interactive Console Work: Selenium and Capybara

By
If you are working on shell most of the time Aliases are very helpfull and time saving. For testing purposes you can use Alias for getting ready your test suites. In this post, I want to explain both running Selenium and Capybara on console and creating aliases for each.  This post is for Windows machines, if you are using Unix-like see   this post . Creating Scripts for Selenium and Capybara First of all, it is assumed that you have installed Selenium and Capybara correctly and they work on your machines. If you haven't installed, you can see my previous posts. I am using the Selenium with Python and the Capybara with Ruby. You can use several different language for Selenium but Capybara works only with Ruby.  Create scripts in a directory called scripts (in your home folder, like as  ~/scripts ) for your automation tool as following, save them as capybara.rb, sel.py :  Creating Aliases Depends on your favourite shell, you need to add the al...
Read more

Page-Object Pattern for Selenium Test Automation with Python

By
Page-object model is a pattern that you can apply it to develop efficient automation framework. With the page-model, it is possible to minimize maintenance cost. Basically page-object means that your every page is inherited from a base class which includes basic functionalities for every page. If you have some new functionalities that every page should have, you can simple add it to the base class. Base class is like the following: In this part we are creating pages which are inherited from base page. Every page has its own functionalities written as python functions. Some functions return to a new page, it means that these functions leave the current page and produce a new page. You should write as much as functions you need in the assertion part because this is the only part you can use the webdriver functions to interact with web pages . This part can be evaluate as providing data to assertion part.   The last part is related to...
Read more

Performance Testing on CI: Locust is running on Jenkins

By
For a successful Continuous Integration pipeline, there should be jobs for testing the performance of the application. It is necessary if the application is still performing well. Generally performance testing is thought as kinds of activities performed one step before going to live. In general approach it is true but don't forget to test your application's performance as soon as there is an testable software, such as an api end point, functions, and etc. For CI it is a good approach to testing performance after functional testing and just before the deployment of next stage. In this post, I want to share some info about Jenkins and Locust. In my previous post you can find some information about Locust and Jenkins. Jenkins operates the CI environment and Locust is a tool for performance testing. To run the Locust on Jenkins you need command line arguments which control the number of clients ,   hatch rate,  running locust without web interface and there s...
Read more