Browser Compatibility Testing

1. What is Compatibility Testing?

According to the ISTQB, compatibility is an activity which independent computer systems work successfully in the same environment without affecting the other systems in the environment. In terms of testing, compatibility testing is a testing activity to ensure that after any updates in the environments the application is still work as expected. From this point of view, compatibility testing is aiming to explore the environment of the users, which should be supported by the statistical data and to find any un-expected behaviors of the application before the customers/users have such experiences. Compatibility matrix should be prepared to ensure that all the environment can be tested by desired frequency, a sample compatibility matrix can be found below: 

Browser Compatibility Matrix

Aim of the compatibility testing:
  • Determining any functional or visual missing, errors or bugs of the application in the users’ environments with testing 
  • By a range of most known web browser and most favorite hardware platforms, testing your web site for compatibility 
  • Determine the effects of the operating system patch and/or update
  • By different add-ins, operating systems and application, testing the servers 
  • By a number of different operating systems with installed many other applications which the target users may have, testing the application
Normally, compatibility testing should be run after system acceptance test and user acceptance test are completed successfully.

2. Determination of Sample E-Commerce Site User Environments

Constitute a risk to the test cases to test and provide better service to customers on behalf of the media, the statistics of current customer can be investigated by two dimensions, operating systems and used browsers. Information obtained from the Google Analytics provides the necessary requirement about the users' browsers and operating systems. In this compatibility analysis a specific time slot information is obtained. Therefore, this information will also show the user trend and the compatibility test environment should be updated by the user trend. 

Let's look at the type of operating system and browser used by the site's customers, 27 operating system and more than 40 different operating system versions, 57 different browsers and more than 100 browser versions. To cover all the possible test environment, there must be more than a thousands test environments. Even in this case we will not perform testing all possibilities. According to the ISQTB,  100% test is impossible! Risk analysis can be helpful in this situation. An applicable testing aim is to cover the 95% of customers environments and taking risk of 5% is manageable test objectives.

You can find some of the configurations below; operating system and browser information.

Operating System vs Visit

Generally customers use Windows operating system so by testing all the version of the Windows can cover the testing objectives (95% confidence range is considered). However, for the image of the company testing the Linux, Mac and Iphone is important. In addition, growing Android market should be take into the account.

Browsers and Operating Systems

Also if we look at the Browser Table, the first three browsers cover the of 98% of the current user's browser. But the Safari web browser which generally used by Macintosh operating system may be reasonable. A large part of the browser of left in the list (more than 50) are composed of browser of mobile devices. If any problems encounter, it should be solved reported for the mobile browsers. Mobile users can be motivated to use Opera Mini to reduce the browser originated problem.
Browser vs Visit

The table above shows that Windows operating system with Internet Explorer, Chrome and Firefox cover the majority users environment by 97% of the sum of all users. But the reason above, testing the Linux, iPhone, Mac is benefitial for the image of the company.

By the help of this information, compatibility test environments shown as a matrix below.

Compatibility Test Environments

3. Determination of Compatibility Test Environments

Compatibility testing can be done on test scenarios which is matured. Therefore, the test scenarios should be develop and rating the risk of each test case. Since seen in the table above, approximately 15 different test environment should be created this can be made by Virtual Box virtual. Each test environment, compatibility tests should be performed. This is a time consuming event for manual test execution and update of the test environment so this should be automated, at least a part of the test cases. Some interesting tools in order to reduce the workload on these tests can help for automation. For compatibility testing, the following requirements come up:
  1. A powerful computer: A test machine - Ubuntu 64 bit installed
  2. We want to install operating system images for media
  3. Test automation tools
  4. iPhone, iPad, and Mac simulator
  5. iPhone, iPhone, Mac computer 

4. Compatibility Test Levels

By considering the project scope and the requirements of customers, there should be three different test levels: Partial Coverage, Full Coverage and Sanity Testing should be done under the name of compatibility tests. In general, these tests covered by the test environments, the amount (%) covered by the test case and the priority level ( Priority: Critical, High, Medium, Low ) were evaluated. Levels in this test should be done as follows. 

4.1. Partial Coverage 

In the 95% of the test environment for at least run the main test cases. ( in the test case Priority = critical )
Partial coverage

4.2. Full Coverage

In the 90% of the test environment all the master test case should be executed
Full Coverage

4.3. Sanity Test

Sanity test is to test the basic functions of the application which help to make decision for continue further testing or stop the testing. Testing the main test cases in an adequate test environment.

4.4. Unsupported Region (not support)

Test environment or test cases which are not included in full coverage testing cannot be tested so these test cases never be tested.


  1. Replies
    1. many thing have changed since 2011. We are now just talking about mobile and mobile:). Your service is very critical.


Post a Comment

Popular posts from this blog

Testing WEB Sevices and Automating SOAP Services

Performance Testing on CI: Locust is running on Jenkins

Change Default Timeout and Wait Time of Capybara

Performance Testing on CI: Integration of Locust and Jenkins

Create an Alias for Interactive Console Work: Selenium and Capybara