TEST THE RISK!

If you are testing your application against emulator / simulator, you will still have risks that the expected features may possible not work on several real devices. To handle the real device testing you should have many devices to run your tests. In the current market, we have lots of possibilities for device and operating system versions combinations. Best approach to find most used devices, you should use some statistical data. Most of the cases Pareto analysis can help for selecting high coverage devices. However if you want to find more bug before release, you can focus on the newest device with latest OS version and the oldest devices with the oldest OS version. Any case you should have at least 5-10 android devices and 3-5 iOS devices for a good level of coverage in the beginning. Therefore managing the devices is another problem if you want to have your own local test suites. As a good alternative you can use the cloud services. In this post I want to share some information a

If you have Microsoft SQL server in your test environment and want to populate test data in Capybara, Calabash scripts you will most probably have difficult time to installing appropriate Ruby gems. One of most popular ruby gem for Mssql connection is tiny_tds but I was not able to install it for both Windows and Unix like environment just because of dependencies. I got the following errors: Errno::EACCES: Permission denied @ rb_sysopen - /Users/mesutgunes/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/tiny_tds-1.1.0/.codeclimate.yml An error occurred while installing tiny_tds (1.1.0), and Bundler cannot continue. Make sure that `gem install tiny_tds -v '1.1.0'` succeeds before bundling. Another popular ruby gem is sequel . It does not give an error for installation but when it comes to usage, it gives the following error for not found adapter. Main problem with the Mssql in unix is finding compatible adapter. See the log: irb#1(main):002:0> db = Sequel.odbc(&qu

If you want to test your mobile application and looking for an open source testing tools then most probably you will face Calabash and Appium . These are the top two most popular tools for mobile automation. At first glance, they seem to be learnt easily but in detail you would have to deal with many challenges and must spend little bit longer time than you expected. Since these tools are open-sourced and still not mature level of quality there would be some problems when you want to use them for your special needs.  In this post I want to write down my experience about Calabash and Appium. This post may help you to choose which one is suitable for your needs. I used following versions: Calabash-Android: 0.7.3 Calabash-iOS: 0.19.1 Appium: 1.5.2 Appium Support Many Languages But Calabash is RUBY! In the background Appium uses Selenium which was created 14 year ago by Thougthwork, check thoughts about Selenium  for mobile testing by Thougthwork. Since Selenium supports Ja

I think the most challenging part of test automation is nowadays having Continuous Integration and one step more is having Continuous Deployment. These two terms have been a buzzwords around the software community, and everyone is talking about them but there are very few companies that can apply these. The reasons behind this can be listed as: Mis-understanding of agile and scrum Lack of "real" test engineers and Lack of Dev-ops culture  Lack of stable test environments  Lack of vision in the software industry, which saving the day is more critical than saving the future Out-of-budget, not investing on good people but investing in tools On the other hand, there are lots of open-source tools that you can just build and configure your own CI, and maybe next step goes to continuous deployment. In this post, I want to explain how you can set up CI on a Windows server. I will use the following tools/technology and language: Ruby, Gem (many Gems), Capybara, Cucumbe

One of the biggest challenge in test automation is updating existing codes. Because of the update in the development, it is very prone to have some changes for the attributes of ui elements and less likely to have some change in behaviour in actions. The second option is most of the time called as bug, otherwise if the business is changed, it could be called as a new feature. For any reasons that cause the automation code gives failures you need to update something. Sometimes I need to refresh my mind :) For easy maintenance, one of the good approach keep your codes modular. It means that each module should responsible for one job and a update problem should be solved by applying just one change in your system. Using Capybara plus Cucumber gives you this features but if you are using them professionally. For this framework, business rules are kept on Cucumber so you can update business rules from your .feature files but related update should reflect to the code. In thi

Appium is an open source tool for mobile test automation, you can automate functional test cases for Android and iOS application. As in my some previous posts, I explained Calabash and I just used Ruby for writing test cases because Calabash only supports Ruby and as I know they have plan to support Java. However Appium doesn't have this language constriction. Since it uses Webdriver, it supports the languages which are supported by Webdriver. It means that Java, Objective-C, PHP, Python, C#, Clojure, Perl, JavaScript with Node.js and plus Ruby are supported. You just install the language client you want to test and then write your test cases. Check for the client for your favourite languages from Appium page . Install Appium: npm install -g appium Install Appium Client: npm install wd Install Python Appium Client: pip install Appium-Python-Client Ensure that you have Python installed, recommend to have 2.7 or above: python --version Ensure that you have

If you tend to use tools effectively, testing is easy and enjoyable. For mobile testing there are lots of challenges and to be honest if you don't push yourself to improve your ability as every developer does you can not test the mobile application good enough. Instead of testing mobile application manually, we can automate the stories whenever the stories are prepared by business analyst with writing cucumber files and then we can start writing the code of automation and finally we can run it when the stories are developed by developer for agile development environment. This is iteration based approach, means you can repeat this steps for each iteration. At the end of the milestone you can run your whole automation code and see the result. For the remaining time you can still test application by using Exploratory Testing techniques to find-out any bugs.  The important point is to make your job effective and efficient. For this you should automate or scripted some repetiti

When you set-up a new android devices or update your .apk file, you may possible get INSTRUMENTATION_FAILED exception: android.util.AndroidException: INSTRUMENTATION_FAILED: com.package.name/sh.calaba.instrumentationbackend.CalabashInstrumentationTestRunner at com.android.commands.am.Am.runInstrument(Am.java:1093) at com.android.commands.am.Am.onRun(Am.java:371) at com.android.internal.os.BaseCommand.run(BaseCommand.java:47) at com.android.commands.am.Am.main(Am.java:100) at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method) at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:251) To fix this issue you can follow the instruction below: Create new folder Copy your .apk file into the newly created folder go to your newly created folder and run this command to create calabash-android project calabash-android gen run calabash-android via your new devices calabash-android run latest.apk features ADB_DEVICE_ARG=192.168.56.101:5555 resign .apk

BDD, which is stand for behaviour driven development, is created by Dan North . BDD is developed on the top of TDD, which is stand for test driven development to remove the gap of unit testing and acceptance testing. In TDD, every unit is started to be written right after defining the expectations from the units, by the unit testing. However by BDD, regarding the agile story, every unit is specified and tested in terms of desired behaviour. This 'desired behaviour' is what exactly business requires in agile stories.  BDD can be applied by the help of some tools to interpret the business 'desired behaviour' to test code. There are some tools that help this aim such as Behat  for PHP, JBhave  for Java world, and the most famous one Cucumber  for the Ruby world. In this post I want to give some points on how to effectively we can use Cucumber for a Calabash project to automate some agile stories for sample android application. Cucumber takes some keywords lik