Repeatable Test Case Design: Escaping Flaky Behavior

Automation means repeating some steps to have some tasks done with test code. However when it comes to test automation fulfilling this meaning is not that easy because of unpredictable reasons. You may face that a case has run successfully for a period of time but it fails for without an unknown reason. Actually there is always reasons which can make test failure but it may not be possible to handle it when writing the test. This kind of test cases are called as flaky test cases by Google. Flaky tests are the last thing you may want in CI environment because test are writing for fast feedback not for checking accidentally failed test cases. In this post I want to explain what I have done to overcome, actually minimise the effect of flakiness.

Nature Of Flakiness

Most of the time flakiness cause by our own mistakes. This could be not understanding the automation frameworks deeply or could be the result of non-adequate analysing the case. The other reasons like third party tools and test data related reasons are also our own mistakes. In nature, automation can be as smart as automation engineer. If you do something that doesn't work for some situations but works most of the time, means that you are facing a real life problems and doing something well or just sparing your time. According to Pava Sudarshan If you do not have a flaky functional tests build, you are not doing anything real”. This is true, a real life problem is not that easy to analysesimulaterepeat and test

Level of Flakiness by Layer of Application

The reason behind the reasons of flakiness is non-deterministic behaviour of application. The reasons of non-deterministic behaviour is increased by the number of dependent layers you are testing if you do not isolated them. Each layers serve some data or business to other layers so that the application works. See the following information for web application

In the image you can see the web application layers. Briefly, presentation layer is presenting the application to end user. Business Layer has businesses to give presentation layer so that presentation layer completes the presentation to end user. Data layer accesses the data for business rules so that the business layer produces relative businesses depends on the users. 

If you are writing test codes for data layer, you will need to handle functions to retrieve data from data source. In this level test are called unit test or unit integration test or simply there should be just only codes for testing units. As in the definition of unit test is testing each units independently, and if the unit are bound to extra object you need to mock that object so that it runs independently and gives same result. By isolating test in its unit, you can remove the reasons of flakiness. Best method of managing test object is to keeping it as simple as possible. However if you test the outer layer of application, there will be more reasons for flaky test cases.

Let's consider the presentation layer, you will need to test a simple login page, valid user functionality. There is a pretty login page which consumes some css and javascript for better user experience. This page may be include third party ajax call for keeping metrics and reporting user failures and etc. Assume that you correctly enter login information to text boxes and pressing submit button successfully. There are still two layers to work for this requests to return responses. If there is a flaky reason in business layer or data layer you need to handle it on presentation layer. Therefore, any reasons for flakiness in the layers are cumulatively occur on the presentation layer

This is another good points for reminding test pyramids why there should be more test on low level of application.

What to Do for Escaping Flakiness

Everything starts with understanding the cases and the dependent parts then removing the dependent parts by mocking them and running cases in isolation, if it is possible. However if you write acceptance testing on UI level (presentation layer), isolation is not easy. For this time you need to understand the behaviour of dependent parts then handle the possible cases generated by dependent parts. It means your test code should also cover the possible unexpected behaviour of dependent parts. 

As an example for this see the following cases: You are testing checkout of an e-commerce company with credit card. Most of the time bank returns success then the page returns to order success page. But there is a case that the bank may add the credit card customer to black list and doesn't let any transaction over it or bank test environment doesn't work. This is number one priority case so you must test this for every. Best solution is mocking the bank and returning true for success scenario and false for fail scenario. Even for the best solution it is not guaranty that your mock is giving relevant responses for every requests. For this case you can check the response of the bank if it is what you expect then continue with test code, but if it is not a valid response then you should isolate the test and mark it as a flaky instead of a failure. This method requires more comprehensive test cases and more programming skills. 

Extra info, always check your knowledge about the automation frameworks, language and test codes. Think twice that test code is written professionally with the latest tools and techniques. Don't forget test automation is a software development activity.


Popular posts from this blog

Testing WEB Sevices and Automating SOAP Services

Performance Testing on CI: Locust is running on Jenkins

Change Default Timeout and Wait Time of Capybara

Performance Testing on CI: Integration of Locust and Jenkins

Create an Alias for Interactive Console Work: Selenium and Capybara