Skip to main content

Centralized Monitoring System for Automated Tests with Elasticsearch and Kibana

Image
By
In this post, I want to give you some practical information about a monitoring system for automated tests. I will touch the following topics:
  • Why we need a monitoring system for automated tests
  • What are the potential benefits of the monitoring system
  • A simple approach using the following technologies:
    • Elasticsearch
    • Kibana
    • Firebase test lab, gcloud and gsutil
    • XCTest and Espresso

What is the Centralization of Log Systems

In a development pipeline, we have a variety of tools and technologies with different languages and scripts. CI/CD pipeline is a good place the see the process of the development, but to get the detail of an item in the pipeline, we need to get into that item and tackle with logs and data. Some times this investigation requires some capabilities to understand it. One of the easiest ways to get a common understanding from the logs in the pipeline is using one easy tool for the whole process. When we collect all the logs and data into one place then we can start debugging activity from here. Therefore the centralized log system means that all the logs and data related to the development pipeline should be stored in one place with a timestamp and/or tags for data separation.

What are Automated Tests

Any types at any testing level which are scripted and run over tools can be called as automated tests. Therefore, it is not aiming to one level just as a unit test or an integration level testing and not also functional or non-functional tests, any tests that we can run on CI/CD pipeline could be considered in the fashion. Basically, the following test matrix shows the test types considering the test levels. As the test pyramid suggests that the more test in the low-level, the more reliable and fast automated tests we would get so staring from the unit level most of the tests could be automated and bonded to the monitoring system.
The test types considering the test levels

What Means the Monitoring System for Automated Tests

As we can see from the testing types table, we have lots of testing types and each testing types have a great variety of tools with different language support. Each test type produces different kinds of output, such as XML, JSON, HTML, and so on, in a different environment so reaching these outputs and understanding them is a pretty hard and time-consuming task and it also requires credential management for each environment. 

The monitoring system for the automated test can collect the test results from the different environments as soon as the tests completed or during a test run. Then the data can be used to draw graphs, create tables, and some simulations. In this way, everyone can see the result of all the automated tests written for the product in realtime.

Why We Need a Monitoring System For Automated Tests

As defined in the previous section, collecting the logs and data into one common place makes it easier to work with these data. These data can be used for monitoring, debugging, reporting, alerting, and other purposes so important thing is collecting data and making them available for the whole team. 

The benefits of the monitoring system for automated tests can be summarized into the following topics:

1. Team Awareness

If everything in a system is transparent, then the system itself shows what it is. There will no extra word to explain it. Every day the monitoring system shows the result of the tests run against the product and it highlights how much of them passed and failed. If there are two kinds of tests on the monitor and one has a red part, then there is no more word to explain that the red part test has a problem and it needs to be solved. This can create team awareness of the product to take any action about failure. 

2. Contribution

Anyone can write a test for any test type if he is willing to write. The monitoring system can be itself a self-explained system that you can easily see the separation between test types and products. If the number of unit tests written for the Android app is half of the number of the unit tests written for the iOS app then the team should take action to remove this gap.  

3. Quality-Centric Development

Software development is a continuous effort so every commits adds something new on the product. If there is something wrong with one of the quality metrics then the related action should be taken to improve the related metric. 

4. Increased Number of the Automated Test

The stability in the results is good as far as the numbers of the tests go up. If the number of the test stay stable for every commit, we will get lower code coverage. Each new requires new automated tests and fixing defects also requires updating correctness of tests or adding some extra tests for edge cases. Monitoring these metrics, courage us to write more tests and make them passed.

5. Complete-Working Process - CI

In practice, sometimes, the CI process doesn't have tests, or these steps are somehow skipped. The main responsibility of the team is to develop a high-quality product. The meaning of the high-quality product depends on the requirement but there is only one way to understand the quality of the product is to check requirements, or other meaning testing it against to the requirements. The good news is that; if we can monitor the tests, we should have already implemented those tests to the CI. 

6. Easier Debugging

The best part of the centralized logging system, having a starting point for debugging errors, failures, exceptions. Logging not only results but also logging the deepest info of the failures and the exceptions make debugging easier. 

What is Elasticsearch

Elasticsearch is the heart of the Elastic (ELK) Stack. Elasticsearch is an open-source search engine. It is built on Apache Lucene. It provides a Restful API. The first version of Elasticsearch was released in 2010, then it became the de facto search engine and it is commonly used for full-text search, business analytics, log analytics, security intelligence, and some of the operational intelligence. 

Elastic Stack

Elasticsearch is a full-text and distributed NoSQL database so it stores documents with indexing them instead of tables and schemes. We can send data in JSON format to the Elasticsearch database by Restful API or Logstash, another tool for logs operations. 

What is Kibana

Kibana is a product from ELK Stack. It is an open-source data virtualization tool that is designed to use Elasticsearch as a data source. Once you integrated Elasticsearch and Kibana and fed the Elasticsearch database with data then you use the Kibana.  Kibana has a user-friendly web user interface to create indexes, search, and dashboards. On the dashboard, you can add flexible chars, graphs, heat maps, histograms.

Setting Up Elasticsearch and Kibana in Docker

Containers make everything easier, we can run Elasticsearch and Kibana easily with a docker-compose.yaml. Basically, I am using the docker images provided by Elastic company officially with the version of 6.6.0 for both Elasticsearch and Kibana. Then it runs the Elasticsearch on port 9200 and the Kibana on port 5601. Therefore after the docker-compose up command, you can navigate to http://localhost:9200 for Elasticsearch and http://localhost:5601 for Kibana.

Export Data from Firebase to Elasticsearch

I mentioned above, to feed the Elasticsearch database Logstash is used for most generic architecture. However, in this post, we want to integrate mobile application testing in the cloud. We use Firebase as a cloud mobile testing provider.  Firebase provides test labs to run mobile application testing. For mobile application testing we have implemented tests with native tools so for Android testing, we will use Espresso and for iOS testing, we will use XCTest.

To be able to export data from Firebase to Elasticsearch we need to follow these steps:
  1. Run mobile tests with auto-generated UUID by using GCLOUD cli 
  2. Calculate the related test result files with this UUID
  3. Grasp the .XML files from Google Storage by using GSUTIL
  4. Export data on those .XML files to Elasticseach by using Restful API. For each run to be distinctive, create new label called timeString. This variable will be used to separate the test runs and make them ordered.
You run the script with android or ios options:

# run test and export data
bash run_tests.sh android
bash run_tests.sh ios

# only export data
python3.6 export_xml_data_elastic.py android
python3.6 export_xml_data_elastic.py ios

and this is the Python script to export the data

Creating Interactive Dashboard on Kibana 

Search on Kibana with Filters

Go to the Discover menu and select one item in the list then click the + button of the related filter item. Once you click the + button next to the name item it filters the search item with value name has and then I want to add the testProject to filter both ios project with unit tests.

Kibana searching with filters

Create Visualizations

Click Visualize then click + button on the middle of the page, it will pop-up types of visualization menu then you can select one of your favorite charts. Then it opens the saved search item, you can select the search you saved in the previous step.

Kibana setting the metrics

I choose the bar chart, later I will add one column to show the number of success, failure, and error.  As you can see from the picture above you need to define the Metrics and the picture below shows how to split the chart for each run by setting the Buckets. 

Kibana setting Buckets

Creating an Interactive Dashbord

Finally, we can create a Kibana Dashboard that can display every chart related to the automated test results. Click on the Dashboard menu and click the add button on the top-right then select the visualization that you created in the previous step. That's all! Now we can check the dashboard.

Kibana dashboard with the result of automated tests


If you want to see the repository and contribute it, you can fork it from Github :)



Labels:Software testing

Popular posts for software testing and automation

Selenium Error "Element is not currently interactable and may not be manipulated"

By
Selenium webdriver can drive different browsers like as Firefox, Chrome or Internet Explorer. These browsers actually cover the majority of internet users, so testing these browsers possibly covers the 90% of the internet users. However, there is no guaranty that the same automation scripts can work without a failure on these three browsers. For this reason, automation code should be error-prone for the browsers you want to cover. The following error is caught when the test script run for Chrome and Internet Explorer, but surprisingly there is no error for the Firefox. Selenium gives an error like below: Traceback (most recent call last):   File "D:\workspace\sample_project\sample_run.py", line 10, in <module>     m.login()   File "D:\workspace\ sample_project \test_case_imps.py", line 335, in login     driver.find_element_by_id("id_username").clear()   File "C:\Python27\lib\site-packages\selenium-2.35.0-py2.7.egg\selenium\webdriver\r
Read more

Change Default Timeout and Wait Time of Capybara

By
One of the biggest challenge for automation is handling timeout problem. Most of the time, timeout is 60 seconds but it may sometimes not enough if you have badly designed asynchronous calls or the third party ajax calls. This makes handling timeout more complex. set large enough to tolerate network related problems. For Selenium based automation frameworks, like Capybara, default Webdriver timeout is set to Net::ReadTimeout (Net::ReadTimeout) Changing ReadTimeout If you have timeout problem for Capybara, it gives an error like above. This means that the page is not fully loaded in given timeout period. Even you can see that page is loaded correctly but webdriver wait until the Ajax calls finish. class BufferedIO #:nodoc: internal use only def initialize (io) @io = io @read_timeout = 60 @continue_timeout = nil @debug_output = nil @rbuf = '' end . . . . . def rbuf_fill beg
Read more

Create an Alias for Interactive Console Work: Selenium and Capybara

By
If you are working on shell most of the time Aliases are very helpfull and time saving. For testing purposes you can use Alias for getting ready your test suites. In this post, I want to explain both running Selenium and Capybara on console and creating aliases for each.  This post is for Windows machines, if you are using Unix-like see   this post . Creating Scripts for Selenium and Capybara First of all, it is assumed that you have installed Selenium and Capybara correctly and they work on your machines. If you haven't installed, you can see my previous posts. I am using the Selenium with Python and the Capybara with Ruby. You can use several different language for Selenium but Capybara works only with Ruby.  Create scripts in a directory called scripts (in your home folder, like as  ~/scripts ) for your automation tool as following, save them as capybara.rb, sel.py :  Creating Aliases Depends on your favourite shell, you need to add the alias to .bashrc bash
Read more

Page-Object Pattern for Selenium Test Automation with Python

By
Page-object model is a pattern that you can apply it to develop efficient automation framework. With the page-model, it is possible to minimize maintenance cost. Basically page-object means that your every page is inherited from a base class which includes basic functionalities for every page. If you have some new functionalities that every page should have, you can simple add it to the base class. Base class is like the following: In this part we are creating pages which are inherited from base page. Every page has its own functionalities written as python functions. Some functions return to a new page, it means that these functions leave the current page and produce a new page. You should write as much as functions you need in the assertion part because this is the only part you can use the webdriver functions to interact with web pages . This part can be evaluate as providing data to assertion part.   The last part is related to asserting your test cases against to the
Read more

Performance Testing on CI: Locust is running on Jenkins

By
For a successful Continuous Integration pipeline, there should be jobs for testing the performance of the application. It is necessary if the application is still performing well. Generally performance testing is thought as kinds of activities performed one step before going to live. In general approach it is true but don't forget to test your application's performance as soon as there is an testable software, such as an api end point, functions, and etc. For CI it is a good approach to testing performance after functional testing and just before the deployment of next stage. In this post, I want to share some info about Jenkins and Locust. In my previous post you can find some information about Locust and Jenkins. Jenkins operates the CI environment and Locust is a tool for performance testing. To run the Locust on Jenkins you need command line arguments which control the number of clients ,   hatch rate,  running locust without web interface and there should be so
Read more