I had an opportunity to give a speech about how we can automate performance testing in CI/CD pipeline in a conference held in Istanbul for the first time "Test automation and Digital QA Submit Istanbul, #TAS19". My subject was "Automated Performance Testing", the first part of the speech is to explain what is performance tests and the second part is to explain performance tools, and the last section is to explain what is automated performance testing and how to implement it to CI/CD.
In this post, I want to explain my thoughts about the basics of performance testing tools. Next posts will focus on automated performance testing and using K6 as a performance testing tool.
How to Evaluate Performance Testing Tools
Before diving into the performance testing tools and technologies we have in the industry, we should define the criteria for evaluation. Each tool has its own advantages or disadvantages and similarity between ancestors or others. When we start to use the tools and try to learn it, most of the time the learning curve is having a critical factor. It is fine since the learning is and using it efficiently is a huge impact for tools but after this period the performance of the tool during running it becomes more critical and having support from open-source community or consultancy companies is another very important factor. Let's list the factors for evaluating performance testing tools:
Criteria for Performance Testing Tools Evaluation:
- Learning curve
- Using a tool, in the beginning, can be difficult so a good tool should have well-defined documents and a better one ready project to run
- Tools can have many features so running simple scripts doesn't mean that the tool is efficiently using for complex test scenarios. A good tool also support complex cases to handle real-world scenarios
- Price
- None of us wants to spend many if there is a free version of the tools. If the case is the money then you should go with the open-source tools but don't forget to search that the tool has a good community so that you get some support from there.
- Some tools are created by the community so you can use without paying money, however if you want to get some enterprise-level support, some companies are giving this supports for the tools
- Supports
- Be sure that tool has a supportive community if it is open-source
- Good company support for commercial tools
- Maturity
- Tools may have defects, so having a buggy tool integration in your environment may cause maintenance costs. Before investing in a tools check the known issues about the tools.
- Languages supported for performance testing scripts
- One can not have a profession in many languages so the supported language should include your favorite one
- In general, performance scripts are easy in the beginning but if you want to cover complex scenarios with scalable test environment then the language may be a barrier to handle these situations
- Performance of the tool itself
- Testing tools require resources to create virtual users and requests. This resource may increase exponentially if you want to create lots of users for your stress tests so be sure that how the performance of the tools itself
- Generally, the performance is evaluating by AB - Apache Benchmark tool. The response time and resource which is consuming by the tools are comparing with the result of the AB. To get the idea about the performance of some well-known performance testing tools, read this blog.
- Integration to other necessary tools
- How you can integrate it with databases, reporting tools, CI/CD and so on
- Embed features
- Separated load creation
- Load distribution
- Writing scenarios
- Grouping the scenarios
- Setting thresholds for automated performance tests
- CLI supports
- Supporting only GUI or CLI
- Working headlessly
- Dockerized or can be dockerized easily
Performance Testing Tools
We have a variety of tools in the industry, they change from open source to commercial and/or commercial support; newly started to 20+ years old performance testing tools. Some of the well-known performance tools, that I have experienced and I can suggest you, are as follows
Open-source
- Jmeter - Apache project, v1.0 1998
- Gatling - Enterprise support by Frontline
- K6 - Enterprise support by LoadImpact
- Locust
- and many others
Commercial
- LoadRunner by Microfocus (previously Mercury - HP)
- Blazemeter - support multiple opensource tools
- LoadUI Pro by Smartbear
- and many others