Appium vs Calabash


If you want to test your mobile application and looking for an open source testing tools then most probably you will face Calabash and Appium. These are the top two most popular tools for mobile automation. At first glance, they seem to be learnt easily but in detail you would have to deal with many challenges and must spend little bit longer time than you expected. Since these tools are open-sourced and still not mature level of quality there would be some problems when you want to use them for your special needs.  In this post I want to write down my experience about Calabash and Appium. This post may help you to choose which one is suitable for your needs. I used following versions:

  • Calabash-Android: 0.7.3
  • Calabash-iOS: 0.19.1
  • Appium: 1.5.2

Appium Support Many Languages But Calabash is RUBY!

In the background Appium uses Selenium which was created 14 year ago by Thougthwork, check thoughts about Selenium for mobile testing by Thougthwork. Since Selenium supports Java, C#, Ruby, Python, JavaScript, you can select one of your favourite language and just focus on what Appium does is. 

In terms of Calabash, it only supports Ruby but one of the speech I learnt that they want to support Java too. 

Appium Doesn't Need Building but Calabash Needs Building iOS App

Appium can interact with the applications (both Android and iOS) directly if you set-up them correctly. However Calabash can interact with .apk for Android application only but iOS application needs to have new target called -cal for injecting Calabash-iOS server into the application. This doesn't mean to change the application source code but building it for test server. For more explanation see the following title for the differences of Calabash-iOS architecture. 

Appium Uses Selenium Server But Calabash Uses Calabash Server

Calabash has developed his own server to handle client/server interaction.  The protocol used for communication is called as JSON over HTTP which sends the commands via Calabash server as JSON by using standard HTTP protocol. During the interaction, server should be up and running. 

For this Calabash-Android creates a /test_servers directory inside the project when you running the application for the first time. The server is also an application which is installed and executed in device or simulator. If the signature of the application is changed, the server is also updating.   
Calabash-Android Architecture

For Calabash-iOs project, things are little bit different than Calabash-Android, it is composed of two part: calabash.framework written in Ruby and a Calabash-iOS-Server written in Objective-C. We need a new target which called -cal in XCode project to link the calabash.framework to application. This doesn't mean to change the source code of the application but there should be a new build. The server is a part of the application, so when we run the application the server in the device/simulator listens the commands from calabash.framewok for interaction.
Calabash-iOS Architecture

For Appium, it is using Selenium server, standard Webdriver Rest API, so you must run it before starting automation code.

Appium Has Inspector But Calabash Has Console

If you have install Calabash-console you can use it by typing `calabash-ios console` for iOS and `calabash-android console` for Android testing. These consoles give you opportunity to work interactively with the application you want to test. During my experiences, it gives you very nice features which you need to code your tests such as querying all the elements in the screen, touching, flashing for locating the correct element, entering text for text fields. The console has limited features of Calabash but good enough for interactive working. 


Inspector of Appium is also created for interactive working with application but it is more suitable for getting the xpath of an element you want interact with. If you want to use Appium you will use lots of xpath created by this inspector. Many time using xpath for selenium is not recommended because of performance problem, and confusing path expression. The path always tends to change.   

In general inspector works well but if your application has small piece of object it is really hard to find it. If I want to find the "addToFavoriteButton.png" image to add product to favourite it just let me to click on product container. If I go throught the inspector panes and find the correct containers and child container and finally the favourite image element, it doesn't give me the id of the element. Finally tapping the favourite button doesn't work. See the screenshot for the example.

Doesn't Show ID of Element

Couldn't find small elements 
Inspector couldn't tap the favourite button 


For me, console is the best part of the Calabash when we compare with Appium. In Calabash console we can query all element with this query("*"), then we can find favourite button and touch it easily. See the console output.

Appium has also recording option which is very nice for especially beginner of the tool.



Appium Can be BDD but Calabash Has Built-in BDD

If you have experience with Capybara in BDD style, you can apply exactly same style for Calabash in Ruby. With the Ruby environment you can add cucumber gem for supporting BDD easily. What is more Calabash also has predefined BDD steps that you can create test steps without writing single line of code. But iOS and Android project may have some differences for the steps. The example below doesn't need any code. This is great! You can see the full list for Calabash-ios, just click here.

Then I touch "login"
Then I fill in "username" with "mesut.gunes"
Then I fill in "passord" with "password123"
Then I touch the "login" button 
Then I wait until I don't see "login"
Then I should see "Mesut Güneş"

Appium is a Single Project But Calabash Has Two

When you install Appium you can use it for iOS and Android but Calabash-iOS and Calabash-Android are separated project that means there are some common functions but there are also platform specific functions. Therefore you can ask "why they didn't do it same", most probably this is the worse part of the Calabash. 

Appium can Drive Browser but Calabash Supports WebView

Thanks for the comment of Gustavo Rivera. Appium supports to create android browser as an instance of Chrome. This can help you to automate web pages in a mobile device, or emilator/simulator. 

However, for Calabash, you can see if you query("*") and search for webview it prints the webview. If you wanna interact an element inside the webview you can use css to reach it. See the below, it is for paypal webview.


What is on Github

Since these tools are opensource, it is very important to contribute from other developers and testers so this is also important factor for comparison. Github information shows that Appium has more contributer and commits but Calabash has also good number contributers as of 26 of July, 2016.


CommitsBranchesReleasesContributorsOpen IssuesClosed IssesVersion
Appium5,966421091517203,8471.5.2
Calabash-iOS2,6181010555495750.19.1
Calabash-Andoid1,6055010264274140.7.3

Comments

Popular posts from this blog

Selenium Error "Element is not currently interactable and may not be manipulated"

Change Default Timeout and Wait Time of Capybara

Performance Testing on CI: Locust is running on Jenkins

Selenium Chromedriver Error Message: u'unknown error: unable to discover open pages

Getting the text of elements in Espresso