Skip to main content

Appium vs Calabash

Image
By

If you want to test your mobile application and looking for an open source testing tools then most probably you will face Calabash and Appium. These are the top two most popular tools for mobile automation. At first glance, they seem to be learnt easily but in detail you would have to deal with many challenges and must spend little bit longer time than you expected. Since these tools are open-sourced and still not mature level of quality there would be some problems when you want to use them for your special needs.  In this post I want to write down my experience about Calabash and Appium. This post may help you to choose which one is suitable for your needs. I used following versions:

  • Calabash-Android: 0.7.3
  • Calabash-iOS: 0.19.1
  • Appium: 1.5.2

Appium Support Many Languages But Calabash is RUBY!

In the background Appium uses Selenium which was created 14 year ago by Thougthwork, check thoughts about Selenium for mobile testing by Thougthwork. Since Selenium supports Java, C#, Ruby, Python, JavaScript, you can select one of your favourite language and just focus on what Appium does is. 

In terms of Calabash, it only supports Ruby but one of the speech I learnt that they want to support Java too. 

Appium Doesn't Need Building but Calabash Needs Building iOS App

Appium can interact with the applications (both Android and iOS) directly if you set-up them correctly. However Calabash can interact with .apk for Android application only but iOS application needs to have new target called -cal for injecting Calabash-iOS server into the application. This doesn't mean to change the application source code but building it for test server. For more explanation see the following title for the differences of Calabash-iOS architecture. 

Appium Uses Selenium Server But Calabash Uses Calabash Server

Calabash has developed his own server to handle client/server interaction.  The protocol used for communication is called as JSON over HTTP which sends the commands via Calabash server as JSON by using standard HTTP protocol. During the interaction, server should be up and running. 

For this Calabash-Android creates a /test_servers directory inside the project when you running the application for the first time. The server is also an application which is installed and executed in device or simulator. If the signature of the application is changed, the server is also updating.   
Calabash-Android Architecture

For Calabash-iOs project, things are little bit different than Calabash-Android, it is composed of two part: calabash.framework written in Ruby and a Calabash-iOS-Server written in Objective-C. We need a new target which called -cal in XCode project to link the calabash.framework to application. This doesn't mean to change the source code of the application but there should be a new build. The server is a part of the application, so when we run the application the server in the device/simulator listens the commands from calabash.framewok for interaction.
Calabash-iOS Architecture

For Appium, it is using Selenium server, standard Webdriver Rest API, so you must run it before starting automation code.

Appium Has Inspector But Calabash Has Console

If you have install Calabash-console you can use it by typing `calabash-ios console` for iOS and `calabash-android console` for Android testing. These consoles give you opportunity to work interactively with the application you want to test. During my experiences, it gives you very nice features which you need to code your tests such as querying all the elements in the screen, touching, flashing for locating the correct element, entering text for text fields. The console has limited features of Calabash but good enough for interactive working. 


Inspector of Appium is also created for interactive working with application but it is more suitable for getting the xpath of an element you want interact with. If you want to use Appium you will use lots of xpath created by this inspector. Many time using xpath for selenium is not recommended because of performance problem, and confusing path expression. The path always tends to change.   

 In general inspector works well but if your application has small piece of object it is really hard to find it. If I want to find the "addToFavoriteButton.png" image to add product to favourite it just let me to click on product container. If I go throught the inspector panes and find the correct containers and child container and finally the favourite image element, it doesn't give me the id of the element. Finally tapping the favourite button doesn't work. See the screenshot for the example.

Doesn't Show ID of Element

Couldn't find small elements 
Inspector couldn't tap the favourite button 


 For me, console is the best part of the Calabash when we compare with Appium. In Calabash console we can query all element with this query("*"), then we can find favourite button and touch it easily. See the console output.

 Appium has also recording option which is very nice for especially beginner of the tool.



Appium Can be BDD but Calabash Has Built-in BDD

If you have experience with Capybara in BDD style, you can apply exactly same style for Calabash in Ruby. With the Ruby environment you can add cucumber gem for supporting BDD easily. What is more Calabash also has predefined BDD steps that you can create test steps without writing single line of code. But iOS and Android project may have some differences for the steps. The example below doesn't need any code. This is great! You can see the full list for Calabash-ios, just click here.

Then I touch "login"
Then I fill in "username" with "mesut.gunes"
Then I fill in "passord" with "password123"
Then I touch the "login" button 
Then I wait until I don't see "login"
Then I should see "Mesut Güneş"

Appium is a Single Project But Calabash Has Two

When you install Appium you can use it for iOS and Android but Calabash-iOS and Calabash-Android are separated project that means there are some common functions but there are also platform specific functions. Therefore you can ask "why they didn't do it same", most probably this is the worse part of the Calabash. 

Appium can Drive Browser but Calabash Supports WebView

Thanks for the comment of Gustavo Rivera. Appium supports to create android browser as an instance of Chrome. This can help you to automate web pages in a mobile device, or emilator/simulator. 

However, for Calabash, you can see if you query("*") and search for webview it prints the webview. If you wanna interact an element inside the webview you can use css to reach it. See the below, it is for paypal webview.


What is on Github

Since these tools are opensource, it is very important to contribute from other developers and testers so this is also important factor for comparison. Github information shows that Appium has more contributer and commits but Calabash has also good number contributers as of 26 of July, 2016.
CommitsBranchesReleasesContributorsOpen IssuesClosed IssesVersion
Appium5,966421091517203,8471.5.2
Calabash-iOS2,6181010555495750.19.1
Calabash-Andoid1,6055010264274140.7.3

Labels:Software testing

Popular posts for software testing and automation

Selenium Error "Element is not currently interactable and may not be manipulated"

By
Selenium webdriver can drive different browsers like as Firefox, Chrome or Internet Explorer. These browsers actually cover the majority of internet users, so testing these browsers possibly covers the 90% of the internet users. However, there is no guaranty that the same automation scripts can work without a failure on these three browsers. For this reason, automation code should be error-prone for the browsers you want to cover. The following error is caught when the test script run for Chrome and Internet Explorer, but surprisingly there is no error for the Firefox. Selenium gives an error like below: Traceback (most recent call last):   File "D:\workspace\sample_project\sample_run.py", line 10, in <module>     m.login()   File "D:\workspace\ sample_project \test_case_imps.py", line 335, in login     driver.find_element_by_id("id_username").clear()   File "C:\Python27\lib\site-packages\selenium-2.35.0-py2.7.egg\seleni...
Read more

Change Default Timeout and Wait Time of Capybara

By
One of the biggest challenge for automation is handling timeout problem. Most of the time, timeout is 60 seconds but it may sometimes not enough if you have badly designed asynchronous calls or the third party ajax calls. This makes handling timeout more complex. set large enough to tolerate network related problems. For Selenium based automation frameworks, like Capybara, default Webdriver timeout is set to Net::ReadTimeout (Net::ReadTimeout) Changing ReadTimeout If you have timeout problem for Capybara, it gives an error like above. This means that the page is not fully loaded in given timeout period. Even you can see that page is loaded correctly but webdriver wait until the Ajax calls finish. class BufferedIO #:nodoc: internal use only def initialize (io) @io = io @read_timeout = 60 @continue_timeout = nil @debug_output = nil @rbuf = '' end . . . . . def rbuf_fill beg...
Read more

Create an Alias for Interactive Console Work: Selenium and Capybara

By
If you are working on shell most of the time Aliases are very helpfull and time saving. For testing purposes you can use Alias for getting ready your test suites. In this post, I want to explain both running Selenium and Capybara on console and creating aliases for each.  This post is for Windows machines, if you are using Unix-like see   this post . Creating Scripts for Selenium and Capybara First of all, it is assumed that you have installed Selenium and Capybara correctly and they work on your machines. If you haven't installed, you can see my previous posts. I am using the Selenium with Python and the Capybara with Ruby. You can use several different language for Selenium but Capybara works only with Ruby.  Create scripts in a directory called scripts (in your home folder, like as  ~/scripts ) for your automation tool as following, save them as capybara.rb, sel.py :  Creating Aliases Depends on your favourite shell, you need to add the al...
Read more

Page-Object Pattern for Selenium Test Automation with Python

By
Page-object model is a pattern that you can apply it to develop efficient automation framework. With the page-model, it is possible to minimize maintenance cost. Basically page-object means that your every page is inherited from a base class which includes basic functionalities for every page. If you have some new functionalities that every page should have, you can simple add it to the base class. Base class is like the following: In this part we are creating pages which are inherited from base page. Every page has its own functionalities written as python functions. Some functions return to a new page, it means that these functions leave the current page and produce a new page. You should write as much as functions you need in the assertion part because this is the only part you can use the webdriver functions to interact with web pages . This part can be evaluate as providing data to assertion part.   The last part is related to...
Read more

Performance Testing on CI: Locust is running on Jenkins

By
For a successful Continuous Integration pipeline, there should be jobs for testing the performance of the application. It is necessary if the application is still performing well. Generally performance testing is thought as kinds of activities performed one step before going to live. In general approach it is true but don't forget to test your application's performance as soon as there is an testable software, such as an api end point, functions, and etc. For CI it is a good approach to testing performance after functional testing and just before the deployment of next stage. In this post, I want to share some info about Jenkins and Locust. In my previous post you can find some information about Locust and Jenkins. Jenkins operates the CI environment and Locust is a tool for performance testing. To run the Locust on Jenkins you need command line arguments which control the number of clients ,   hatch rate,  running locust without web interface and there s...
Read more