QA in Production
For a while, I have been thinking the responsibilities of QA Engineers by broadening them to the production environment. I this post I want to write the reasoning behind the needs of QA Engineers in production.
Integration Testing: Tools and TechnologyEveryday we are facing new integration points with third party tools or technologies for win-win strategy. Most of the time these technologies are in the term of trial period or just an cheap alternative to the present one. These technologies or tools simple don't have adequate documentations so it makes the development and testing of integration more risky. This means that development, test, uat or preprod test environment are not enough for catching all possible bugs, as a result production environment is the final destination for checking logs and monitoring the integration results.
Validation of Non-Functional RequirementsThere are also unclear points behind the non-functional requirements like performance and usability. The real performance of the applications can be seen under real user environments. This doesn't mean that performance testing should be done in production. This means that with the performance of the application, the real users behave as expected or they just leaving the application just before the message which was integrated with last build appears. Another meaning of this, just need to check the performance requirement is statistically valid in term of customer satisfaction. The usability part, even we had done usability testing with real user before the new feature request in usability lab, it is not guaranty that the result will be repeated by the real user in their environments. This time we need to check the usability requirement with the real user.
The Nature of Development ProcessAlso development process can impose QA Engineer to monitor production environment. If you adopt CI or CD the process more like to monitor on in production because production is tent to update regularly which means you have more risky situation. Translation from traditional QA to agile adopted QA requires more collaboration with other teams; furthermore translation from manual deployment to CD requires better monitoring system for development process. Checking the monitoring system is a task of QAs because it gives the overall quality of process, which test result can be as good as development process. About the subject ThoughtWorks has taken it to radar and name it as a technology with "QA in Production" and to recommend it as "Trial" which means that enterprise should use it for the project which risk can be handed. It give the following explanation:
Traditionally, QA roles have focused on assessing the quality of a software product in a pre-production environment. With the rise of Continuous Delivery, the QA role is shifting to include analyzing software product quality in production. This involves monitoring of the production systems, coming up with alert conditions to detect urgent errors, determining ongoing quality issues and figuring out what measurements you can use in the production environment to make this work. While there is a danger that some organizations will go too far and neglect pre-production QA, our experience shows that is a valuable tool for organizations that have already progressed to a reasonable degree of Continuous Delivery.