Skip to main content

Repeatable Test Case Design: Escaping Flaky Behavior

Image
By
Automation means repeating some steps to have some tasks done with test code. However when it comes to test automation fulfilling this meaning is not that easy because of unpredictable reasons. You may face that a case has run successfully for a period of time but it fails for without an unknown reason. Actually there is always reasons which can make test failure but it may not be possible to handle it when writing the test. This kind of test cases are called as flaky test cases by Google. Flaky tests are the last thing you may want in CI environment because test are writing for fast feedback not for checking accidentally failed test cases. In this post I want to explain what I have done to overcome, actually minimise the effect of flakiness.

Nature Of Flakiness

Most of the time flakiness cause by our own mistakes. This could be not understanding the automation frameworks deeply or could be the result of non-adequate analysing the case. The other reasons like third party tools and test data related reasons are also our own mistakes. In nature, automation can be as smart as automation engineer. If you do something that doesn't work for some situations but works most of the time, means that you are facing a real life problems and doing something well or just sparing your time. According to Pava Sudarshan “If you do not have a flaky functional tests build, you are not doing anything real”. This is true, a real life problem is not that easy to analyse, simulate, repeat and test. 

Level of Flakiness by Layer of Application

The reason behind the reasons of flakiness is non-deterministic behaviour of application. The reasons of non-deterministic behaviour is increased by the number of dependent layers you are testing if you do not isolated them. Each layers serve some data or business to other layers so that the application works. See the following information for web application




In the image you can see the web application layers. Briefly, presentation layer is presenting the application to end user. Business Layer has businesses to give presentation layer so that presentation layer completes the presentation to end user. Data layer accesses the data for business rules so that the business layer produces relative businesses depends on the users. 

 If you are writing test codes for data layer, you will need to handle functions to retrieve data from data source. In this level test are called unit test or unit integration test or simply there should be just only codes for testing units. As in the definition of unit test is testing each units independently, and if the unit are bound to extra object you need to mock that object so that it runs independently and gives same result. By isolating test in its unit, you can remove the reasons of flakiness. Best method of managing test object is to keeping it as simple as possible. However if you test the outer layer of application, there will be more reasons for flaky test cases.

 Let's consider the presentation layer, you will need to test a simple login page, valid user functionality. There is a pretty login page which consumes some css and javascript for better user experience. This page may be include third party ajax call for keeping metrics and reporting user failures and etc. Assume that you correctly enter login information to text boxes and pressing submit button successfully. There are still two layers to work for this requests to return responses. If there is a flaky reason in business layer or data layer you need to handle it on presentation layer. Therefore, any reasons for flakiness in the layers are cumulatively occur on the presentation layer. 

This is another good points for reminding test pyramids why there should be more test on low level of application.

What to Do for Escaping Flakiness

Everything starts with understanding the cases and the dependent parts then removing the dependent parts by mocking them and running cases in isolation, if it is possible. However if you write acceptance testing on UI level (presentation layer), isolation is not easy. For this time you need to understand the behaviour of dependent parts then handle the possible cases generated by dependent parts. It means your test code should also cover the possible unexpected behaviour of dependent parts. 

 As an example for this see the following cases: You are testing checkout of an e-commerce company with credit card. Most of the time bank returns success then the page returns to order success page. But there is a case that the bank may add the credit card customer to black list and doesn't let any transaction over it or bank test environment doesn't work. This is number one priority case so you must test this for every. Best solution is mocking the bank and returning true for success scenario and false for fail scenario. Even for the best solution it is not guaranty that your mock is giving relevant responses for every requests. For this case you can check the response of the bank if it is what you expect then continue with test code, but if it is not a valid response then you should isolate the test and mark it as a flaky instead of a failure. This method requires more comprehensive test cases and more programming skills. 

 Extra info, always check your knowledge about the automation frameworks, language and test codes. Think twice that test code is written professionally with the latest tools and techniques. Don't forget test automation is a software development activity.



Labels:Software testing

Popular posts for software testing and automation

Selenium Error "Element is not currently interactable and may not be manipulated"

By
Selenium webdriver can drive different browsers like as Firefox, Chrome or Internet Explorer. These browsers actually cover the majority of internet users, so testing these browsers possibly covers the 90% of the internet users. However, there is no guaranty that the same automation scripts can work without a failure on these three browsers. For this reason, automation code should be error-prone for the browsers you want to cover. The following error is caught when the test script run for Chrome and Internet Explorer, but surprisingly there is no error for the Firefox. Selenium gives an error like below: Traceback (most recent call last):   File "D:\workspace\sample_project\sample_run.py", line 10, in <module>     m.login()   File "D:\workspace\ sample_project \test_case_imps.py", line 335, in login     driver.find_element_by_id("id_username").clear()   File "C:\Python27\lib\site-packages\selenium-2.35.0-py2.7.egg\seleni...
Read more

Change Default Timeout and Wait Time of Capybara

By
One of the biggest challenge for automation is handling timeout problem. Most of the time, timeout is 60 seconds but it may sometimes not enough if you have badly designed asynchronous calls or the third party ajax calls. This makes handling timeout more complex. set large enough to tolerate network related problems. For Selenium based automation frameworks, like Capybara, default Webdriver timeout is set to Net::ReadTimeout (Net::ReadTimeout) Changing ReadTimeout If you have timeout problem for Capybara, it gives an error like above. This means that the page is not fully loaded in given timeout period. Even you can see that page is loaded correctly but webdriver wait until the Ajax calls finish. class BufferedIO #:nodoc: internal use only def initialize (io) @io = io @read_timeout = 60 @continue_timeout = nil @debug_output = nil @rbuf = '' end . . . . . def rbuf_fill beg...
Read more

Create an Alias for Interactive Console Work: Selenium and Capybara

By
If you are working on shell most of the time Aliases are very helpfull and time saving. For testing purposes you can use Alias for getting ready your test suites. In this post, I want to explain both running Selenium and Capybara on console and creating aliases for each.  This post is for Windows machines, if you are using Unix-like see   this post . Creating Scripts for Selenium and Capybara First of all, it is assumed that you have installed Selenium and Capybara correctly and they work on your machines. If you haven't installed, you can see my previous posts. I am using the Selenium with Python and the Capybara with Ruby. You can use several different language for Selenium but Capybara works only with Ruby.  Create scripts in a directory called scripts (in your home folder, like as  ~/scripts ) for your automation tool as following, save them as capybara.rb, sel.py :  Creating Aliases Depends on your favourite shell, you need to add the al...
Read more

Page-Object Pattern for Selenium Test Automation with Python

By
Page-object model is a pattern that you can apply it to develop efficient automation framework. With the page-model, it is possible to minimize maintenance cost. Basically page-object means that your every page is inherited from a base class which includes basic functionalities for every page. If you have some new functionalities that every page should have, you can simple add it to the base class. Base class is like the following: In this part we are creating pages which are inherited from base page. Every page has its own functionalities written as python functions. Some functions return to a new page, it means that these functions leave the current page and produce a new page. You should write as much as functions you need in the assertion part because this is the only part you can use the webdriver functions to interact with web pages . This part can be evaluate as providing data to assertion part.   The last part is related to...
Read more

Performance Testing on CI: Locust is running on Jenkins

By
For a successful Continuous Integration pipeline, there should be jobs for testing the performance of the application. It is necessary if the application is still performing well. Generally performance testing is thought as kinds of activities performed one step before going to live. In general approach it is true but don't forget to test your application's performance as soon as there is an testable software, such as an api end point, functions, and etc. For CI it is a good approach to testing performance after functional testing and just before the deployment of next stage. In this post, I want to share some info about Jenkins and Locust. In my previous post you can find some information about Locust and Jenkins. Jenkins operates the CI environment and Locust is a tool for performance testing. To run the Locust on Jenkins you need command line arguments which control the number of clients ,   hatch rate,  running locust without web interface and there s...
Read more