Image CAPTCHA
Enter the characters shown in the image.

You are here

A Structured Evaluation for Selecting a Right Automated Testing Tool

One of the challenges in applying software test automation successfully in your projects is to select a most appropriate automated testing tool or framework.  It is a challenging decision as you have many tools to choose from and a number of requirements to satisfy, and automated testing tools may have hidden problems you don’t see or overlook at the time of making decision. Making the right tool choice is crucial to avoiding problems related to the tool that haunt your project execution.

In selecting a tool, you have a set of selection constraints, which can be classified into hard and soft constraints. Hard constraints are fixed and must be satisfied in any case. These constraints include platforms, executing environments, programming languages, and frameworks used to develop the target application under test (AUT).  Soft constraints can be compromised or partially met. Such constraints include schedule, cost, maintainability, usability, level of programming and technical skills required. As these constraints are adjustable, you find it challenging in balancing among them in order to select the best tool given that most of the tools do not satisfy all of these constraints perfectly.

 

 

Tool Selection Criteria

To objectively evaluate a set of tools, it is recommended that you compare them based on the same set of criteria. You can define a set of criteria using your tool selection constraints and common characteristics of automation testing tools. Below are common characteristics that you may consider when weighting tools:

Platform: on which platforms does the tool being evaluated run? Platforms can be Windows, MacOS, Unix, Linux, Web, Mobile, etc. Some tools such as Selenium and Maveryx support multiple platforms, while others like Ranorex run mainly on Windows.

AUT programming languages: many tools support only a certain programming language used to write the AUT. For example, Abbot and Maveryx are used for testing Java applications. Many other tools like TestComplete and Ranorex can work with any languages used in the AUT provided that it can run on supported platforms.

Scripting languages: which languages does the tool support for writing test scripts? Many automation testing tools provide flexible scripting options, allowing testing teams to write test scripts in their most favorable languages.

Support: it is important to know whether the tool is actively maintained, upgraded, and supported by the original vendor and communities of users and developers. You most likely need help from the vendor or communities during your project.

If the tool has not been released recently or the documentation is not up to date, it is possible that the vendor and the user community are no longer supporting the tool, or at least the interest in the tool has waned.

Usability: this criterion refers to how easy to learn and use the tool. It also indicates whether the tool is stable, robust and efficient. Most of software testing tools is easy to learn and use their basic features at first, but they require much more time to master advanced features, especially programming and maintaining scripts efficiently.

Script maintainability: one of the most challenging problems in test automation is to maintain test scripts to reflect changes in the requirements of AUTs. This is a time-consuming activity in projects where requirements change frequently, which is the nature of agile projects.  Some tools on the market such as Maveryx  and QTP have capabilities to automatically, albeit not fully, update scripts when software is changed. At the present, however, most of the script maintenance is done manually.

Required programming skills: one selling point for many automated testing tools is to support non-programmers, allowing testers who have limited programming skills to do automation test effectively. Tools like Selenium, Katalon Studio, Ranorex and TestComplete provide the record and playback features with scripts automatically generated. Still, some programming knowledge is needed to enhance and maintain test scripts, especially when you have complex test cases which have sophisticated scenarios and require many loops and checkpoints.

Automated testing approaches: various approaches are introduced in existing automated testing tools on the market. Available tools follow one or many of the following:

  • Linear: procedural scripts executing step-by-step from the start to the end of a test case. The linear scripts are typically generated by the tool’s record feature.
  • Record/playback: tools have capabilities to record tests’ actions, generate test scripts, and playback test scripts automatically.
  • Structured: unlike linear, this approach allows scripts to include control structures such as “if-else” and loop (“for”, “while”).
  • Data-driven: the test execution flow is driven by data stored externally, in a database, spreadsheets or files.
  • Keyword-driven: the test execution flow is driven by keywords typically stored in tables mapping keywords and input data needed for the execution. Objects captured from the record and playback capabilities are used as keywords in many tools such as Katalon Studio, Selenium, Ranorex, and TestComplete.
  • Model-based: test procedures or scripts are automatically generated using requirements and behaviors model.
  • Hybrid: supporting two or more of the above. Most of existing tools provide this hybrid approach.

Cost: of course, cost is an important factor in deciding which tool to use. You have to take into account both licensing and support costs. Some tools require no or nominal cost to acquire, but they incur a large sum of money spent to call for external support. Additional cost comes from training and spending extra effort for understanding and solving problems related to the tool.

Quantifying criteria’s value

It is sometimes difficult to determine a winner when facing a number of choices with many similar descriptive or qualitative characteristics. Some criteria are more precise to quantify using a numerical scale than using descriptions. For example, the level of programming and technical skills required can be easily quantify using a rating scale between 0 (none or no programming or technical skills required) to 10 (highest level or advanced programming and technical skills required).

Many of the criteria above can be quantified using the same range, such as, from 0 (none) to 10 (highest). For criteria with a definitive value “No” or “Yes”, using 0 for “No” and 10 for “Yes”.

Putting things together

The table below provides a brief summary of evaluation of 7 popular testing tools using the criteria discussed above. Much of the rating for Support, Usability, Script maintainability, Required programming skills is based on my subjective judgment. You may have a different assessment, resulting in other rating values. Due to space limitation, the information on the tools shown in the table is not complete, and the costs of commercial tools can vary dependent on types of license and maintenance needs.

 

Criteria Maveryx Selenium Cucumber Katalon Studio TestComplete Ranorex UFT (QTP)
Open source Yes Yes Yes No No No No
Platform Windows, Linux and Mac Cross-platform Cross-platform Windows, Mac, Android, iOS Windows (mainly), Android, iOS Windows (mainly), Android, iOS Windows (mainly)
AUT programming languages Java Web-based languages Ruby, Java, .NET, Flex or web applications Many Many Many Many
Scripting languages Java Many (Java, C#, Perl, Python, etc.) Ruby, Java, C# Java/Groovy VBScript,  Jscript, DelphiScript,     C++Script,    C#Script C#, VB.NET VBScript
Support Rate 7 (very active) 9 7 7 9 8 9
Usability 8 (easy to use) 7 6 8 9 8 9
Script maintainability 6 (smart object detection) 3 (tool provides little support) 1 (almost no  support by tool) 6 (object repository, Script editor, object re- identification) 7 6 7 (smart object detection and correction)
Required programming skills 7 (high level) 6 (support with record/ playback) 5 4 (record and playback, intuitive testing steps) 7 (strong record/playback capabilities) 6 8
Automated testing approaches Data-driven, keyword-driven Record/ playback, keyword-driven, data-driven Structured, keyword-driven, data-driven Data- and keyword- driven Record/ playback, keyword-driven, data-driven Record/ playback, keyword-driven, data-driven Record/ playback, keyword-driven, data-driven
Cost Free (community edition), from  €940/license (professional edition) Free Free Free (community edition) From $999/license From €609/ license, one year maintenance From $8000/ license
               

 

 

There are other criteria you can include in your evaluation in order to meet your particular tool requirements. They may include performance, stress testing, reporting, and other certain capabilities. With these specifics, you can narrow down the most suitable tool that satisfies your needs.

 

yes

About The Author: 

Dr.Vu Van Nguyen

Dr. Vu Nguyen is a Lecturer at the University of Science, Vietnam National University – Ho Chi Minh city. At the University of Science, he teaches several classes in software engineering and does research on a few software engineering topics including software estimation, software maintenance, software processes, and education in software engineering. Previously, he has worked for IBM US and CSC Vietnam on many software development projects on different capabilities including software developer, software architect, and project manager.

Media: 
GUI Testing Tools

.

Desktop testing

List mode
Table mode
Icons mode

Displaying 1 - 93 of 93

Title Testing Objectives Since Price Quotes Ping
TOSCA Testsuite DataBase testing, Data Warehouse Testing, Desktop testing, Mobile Testing, Pair-Wise Analysis, SAP Testing, Server testing, SOA testing, Test Cases Statistics, Web testing 2,008 Call Manufacturer for details Active
eggPlant Functional Application Life Cycle Tracking, Cross Platform Testing, Desktop testing, Image Based Testing, Embedded Software, Image Comparison, Java testing, Mobile Testing, Network testing, Performance Analysis, Web testing, Cross-Browser testing 2,008 Active
Ranorex Integrated Test Automation Tools Mobile Testing, Web testing, Cross-Browser testing, Java testing, Desktop testing 2,007 November 2015 : Ranorex Runtime Shared License € 690, November 2015 : Ranorex Premium Node-Locked License: € 1990, November 2015 : Ranorex Premium Floating License: € 3490 Active
Katalon Studio Desktop testing, Mobile Testing, UI testing, Web testing 2,016 Free Use Active
Automation Anywhere Enterprise DataBase testing, Desktop testing, EMail Testing, Web testing 2,008 2015: Standard - 1495 USD / desktop user, 2015: Premier - 2995 USD / desktop user Active
IQ Bot Desktop testing, Web testing 2,012 Active
Badboy Desktop testing, Web testing 2,012 Individual license - USD $45, 10 license pack - USD $300 Active
SilkTest Desktop testing, Mobile Testing, SAP Testing, Test Design, Web testing, Cross-Browser testing 1,994 Call Manufacturer for more details Active
SeleniumHQ Desktop testing, Object Based Testing, Web testing, Cross-Browser testing 2,006 Free use Active
qTest Platform Desktop testing, Web testing 2,014 September 2014 : qTest Enterprise - $49 per user/month billed annually - on demand, September 2014 : Pro w/eXplorer - $38 per user/month billed annually - on demand, September 2014 : Pro - $29 per user/month billed annually - on demand , September 2014 : qTest eXplorer - $29 per user/month billed annually - on demand Active
Robot Framework Desktop testing, Java testing, Web testing 2,008 Free to use Active
WinAutomation Web testing, Test Design, Desktop testing 2,005 WinAutomation 6.0 Professional Edition US$ 485, WinAutomation 6.0 Standard Edition US$ 249 Active
JSystem Desktop testing, SOA testing, Web testing 2,011 Free Use Active
Unified Functional Testing Web testing, Back-End Service, Desktop testing, Image Based Testing, Object Based Testing, Image Comparison 2,000 Active
Abbot Desktop testing, Web testing 2,002 Free use 2015
M-eux Test Desktop testing, Object Based Testing, Mobile Testing 2,008 July 2015: M-eux Test Standard for One Mobile Platform (1 Year): € 2600, July 2015: Public Cloud: Per Hour: € 27 / month Active
WatiN Web testing, Desktop testing 2,005 Free Use 2011
Rapise API testing, Desktop testing, Object Based Testing, Mobile Testing, SOA testing, Web Services, Web testing, Cross-Browser testing 2,009 2016: Single license per $1000 Active
Axe Platform Code Coverage testing, Desktop testing 2,008 Call Manufacturer for details Active
AutoIt Desktop testing, Object Based Testing 2,010 2015
FitNesse DataBase testing, Desktop testing, Web testing 2,009 Free Use Active
QF-Test Desktop testing, Image Comparison, Java testing, Load Capabilities, Performance Analysis, SAP Testing, Test Cases Statistics, Test Design, Web testing, Cross-Browser testing 2,001 Developer license (Test creation and execution): One GUI-technology included. 1,995 €, GUI-technology extensions: Purchase now or later on demand. 500 €, Maintenance (obligatory for initial license purchase): All Upgrades + support for 1 year with one GUI-technology. 480 €, Runtime license (Test execution): Substantial volume discounts for load testing. 995 € Active
CodedUI Desktop testing, Object Based Testing, Web testing, Cross-Browser testing 2,010 Active
TestComplete Desktop testing, Web testing, Cross-Browser testing 1,999 Nov 2015 : Node-locked License - $1,000, Nov 2015 : Floating User License - $2,500, Nov 2015 : Test Execute - $499 Active
LoadRunner Desktop testing, Image Based Testing, Java testing, Load Capabilities, SAP Testing, SOA testing, Web testing 2,000 Free for Unlimited Time - Available with 50 Virtual Users (VUsers), Community edition : Free, Virtual User days : $0.56 per virtual user day Active
Squish Desktop testing, Embedded Software, Java testing, Mobile Testing, Web testing, Cross-Browser testing 2,003 April 2015 - Group license : Up-front license fee 7440 €; includes first 12 months of support & updates subscription., April 2015 - Named User License : Up-front license fee 6900 €; includes first 12 months of support & updates subscription. Active
QA Wizard Pro Desktop testing, Object Based Testing, Load Capabilities, Stress Testing, Web testing 2,007 2015 October: QA Wizard Pro, $25/mo (Billed annually), 2015 October: OCR Add-on, $10/mo (Billed annually), 2015 October: 50 Load Test VUs, $20/mo (Billed annually) Active
FEST Desktop testing, Web testing 2,008 Free Use 2011
TestingWhiz DataBase testing, Desktop testing, Object Based Testing, Mobile Testing, Web testing, Cross-Browser testing 2,011 September 2014 - Free use on sourceforge Active
AppLoader Desktop testing, Image Based Testing, EMail Testing, Image Comparison, Java testing, Load Capabilities, Stress Testing, Performance Analysis, SAP Testing, Web testing 2,000 Please Call Manufacturer for more details... Active
TestMagic Desktop testing, Mobile Testing, Server testing, SOA testing, Web testing 2,009 Active
TestDrive Desktop testing, Image Comparison, Java testing, SAP Testing, Web testing 2,005 Call Manufacturer for details Active
LDTP Desktop testing, Web testing 2,004 Free Use Active
Robotium Desktop testing, Mobile Testing 2,010 Free Use Active
Rational Functional Tester [RFT] Desktop testing, Object Based Testing, Web testing 2,001 July 2014: User license = 6820 USD Active
OpKey Desktop testing, Mobile Testing, Server testing, SOA testing, Web testing, Cross-Browser testing 2,006 2016: Cloud-free - free, 2016: Cloud-Lite - $50/month Active
Maveryx Desktop testing, Java testing, Test Cases Statistics 2,010 Community Edition: Unlimited free trial, Maveryx Basic Subscription: 940 €per year, Maveryx Business Subscription: 1640 € per year, Contact Manufacturer for more details Active
Qualify Application Life Cycle Tracking, Desktop testing, SAP Testing, Test Cases Statistics, Web testing, Cross-Browser testing 2,009 Active
TestStack.White Desktop testing 2,007 Free to use Active
ZAPTest Mobile Testing, Web testing, Cross Platform Testing, Desktop testing 2,001 Active
Guitar Desktop testing 2,008 Free Use 2016
TestShell Test Automation Framework Desktop testing, Object Based Testing, Web testing 2,004 Active
T-PLAN ROBOT Mobile Testing, Web testing, Desktop testing 1,989 Active
AutoHotkey Desktop testing 2,004 Free to use Active
Test Studio Desktop testing, Java testing, Mobile Testing, Performance Analysis, Test Cases Statistics, Test Design, Web testing, Cross-Browser testing 2,002 Test Studio Web & Desktop $2,499 perpetual license, include: Automated testing for Web and Desktop Visual Studio plugin, Test Studio Load $999 perpetual license, include: Load testing web UI Load testing web services, Test Studio Ultimate $3,499 perpetual license include: Test Studio Web & Desktop, Mobile, Load, Performance + API testing Active
RIATest Desktop testing, Image Based Testing, Web testing 2,013 Oct 2015 : Professional : $799, Oct 2015 : Standard : $499, Oct 2015 : Runner : $249 Active
TestMaker Desktop testing, Performance Analysis, SOA testing, Web testing 2,000 Active
iMacros API testing, DataBase testing, Desktop testing, Image Comparison, Server testing, SOA testing, Web testing, Cross-Browser testing 2,001 2015: Enterprise Edition - 995 USD per license, 2015: Standart Edition - 495 USD per license, 2015: WebBrowser Component for .NET - 2,995.00 USD per license Active
JAutomate Desktop testing, Image Based Testing, Java testing, Web testing 2,011 2015: Pro License - $999 per perpetual license, 2015: Ent License - $9999 per year 2016
BlazeMeter Desktop testing, Stress Testing, Performance Analysis, Web testing, Cross-Browser testing 2,011 October 2014 :Basic - $249/month/1000 users, October 2014 :Pro - $649/month/5000 users, October 2014 :Pro Plus - $1349/month/10,000users Active
SeeTestAutomation Desktop testing, Image Based Testing, Mobile Testing, Test Embedded systems 2,008 July 2015: $3500/ year for All versions of SINGLE mobile OS Active
Sikuli Desktop testing, Image Based Testing, Mobile Testing 2,010 Free use, Call Manufacturer for more details Active
Robotil Desktop testing 2,012 Free use Active
Silk TestPartner Desktop testing 2,010 Please contact the Manufacturer for more details 2016
GNU Xnee Desktop testing 2,002 Free use 2014
SWTBot Desktop testing 2,001 Free use Active
StoryTestIQ Web testing, Desktop testing 2,009 Free Use 2011
WinTask Desktop testing, Web testing 1,998 2015 Sept: WinTask (for Windows 32 bit) - $249.00 Software only, 2015 Sept: WinTask x64 (for Windows 64 bit) - $399.00 2 years upgrades* 2015
AppPerfect Web Test Desktop testing, Mobile Testing, Web testing, Cross-Browser testing 2,003 Dec 2014: Desktop License = $299, Dec 2014: Floating License = $399 Active
Testuff Web testing, Test Cases Statistics, Application Life Cycle Tracking, Desktop testing 2,007 2015: Monthly subscription $27 per user, 2015: Yearly subscription $270 per user, 2015: Site License $1,840/month Unlimited Active
Selenium IDE Desktop testing, Web testing 2,006 Free use Active
Frank Mobile Testing, Desktop testing 2,011 Free use Active
Twebst Desktop testing, Web testing 2,008 2014
StoryText Desktop testing 2,009 Free Use 2015
TestOptimal Web testing, Cross-Browser testing, Desktop testing 2,009 Call Manufacturer for details Active
CloudQA API testing, Desktop testing, Web testing, Cross-Browser testing 2,016 Active
App Test Desktop testing, Java testing, Mobile Testing 2,003 2016
MTAS API testing, Desktop testing, Mobile Testing, Network testing, Web testing, Cross-Browser testing 2,011 Active
Mago Desktop testing 2,009 Free use 2010
eZscript Web testing, Data Flow Analysis, Banking Process validation, Desktop testing 2,010 Active
UISpec Desktop testing, Mobile Testing 2,010 Free Use Active
Autokey Desktop testing 2,010 Free Use 2012
CUITe Web testing, Desktop testing 2,011 Free Use Active
MobileCloud Platform Desktop testing, Mobile Testing, Web testing 2,006 Active
VisualCron Desktop testing 2,007 May 2015: $299 per server license without maintenance Active
AppVerify Security Testing, SAP Testing, Web testing, Java testing, Desktop testing, Image Based Testing 2,014 Active
Expect Protocol Testing, Server testing, SOA testing, Test Embedded systems, Desktop testing 1,987 Free use 2009
SmarteScript Desktop testing, Web testing, Cross-Browser testing 2,007 Active
Validata Software Assets Suite Message testing, Application Life Cycle Tracking, Banking Process validation, Desktop testing 2,004 Active
HtmlUnit Web testing, Java testing, Desktop testing 2,002 Free Use Active
RCP Testing Tool Desktop testing, Object Based Testing 2,016 Free Use Active
UISpec4J Web testing, Java testing, Desktop testing 2,004 Free use Active
Commercial Tools
Trust Desktop testing, Mobile Testing 2,011 Active
Batik Code Analysis, Code Review, Code Coverage testing, Desktop testing, Image Based Testing, Object Based Testing 2,008 Free use Active
selite DataBase testing, Desktop testing, Web testing 2,014 Free use Active
UA Studio Desktop testing, Object Based Testing, Web testing, Cross-Browser testing 2,006 Waiting for Input Active
TextTest Desktop testing 2,008 Free use Active
TestDrive-UAT Web testing, Desktop testing 2,000 Active
UFT Pro LeanFT Web testing, Cross-Browser testing, Desktop testing, Object Based Testing 2,014 Not Published Active
GXtest Desktop testing, Web testing 1,988 Active
Reflex Unit Desktop testing, Web testing 2,008 Free Use Active
ClicAutomate Web testing, Cross-Browser testing, Desktop testing, Object Based Testing 2,015 Not Published Active

MANUFACTURERS Wallboard

Testing tool manufacturers world-wide list
10Levels ABID CONSULTING AccelQ Accord Software ActiMind AdaCore
AdaLog AgileLoad AgileWay Agitar Algorismi ALL4TEC
Andreas Kleffel Android Apache Apica Apollo Systems AppAssist.mobi
Applitools AppPerfect Appsee ApTest Assertible Assure
Atlassian AutoIt Consulti .. Automation Anyw .. Automation Cons .. Axosoft Aztaz Software
Backtrace I/O Badboy BlazeMeter Borvid BrowserStack BSQUARE
BStriker Intern .. CA Technologies Canonical Canoo Engineeri .. Catch Software CelestialTeapot
Chris Mallett Cleanscape ClicTest CloudQA Codeborne CodeCentrix
CodePlex projec .. Codoid Cogitek Compuware Configure IT Conflair
ConSol Core Services Coronys Ltd Countersoft CresTech Softwa .. CrossBrowserTes ..
Crosscheck Netw .. Crowdsourced Te .. Cucumber Ltd Cyara Cygnet Infotech DareBoost
Databene Datamatics Glob .. DevExpress DTM soft Dynatrace LLC EasyQA
Eclipse EkaTechserv Elvior Emmanuel Jorge Empirix EPAM Systems
Equafy Esterel Technol .. eXept Software .. Experitest Finaris Froglogic
FrontEndART Ltd GeneXus GitHub project gnoso Google Code Pro .. GrammaTech
Gurock Software HelpSystems HENIX Hewlett Packard .. Hexawise High-Tech Bridg ..
Hiptest Hitex IBM Rational imbus Shanghai Impetus Inflectra
informUp InTENSO - IT Ex .. Ipswitch Jamo Solutions Janova JAR Technologie ..
JBoss Developer jClarity Jellly.io JetBrains Jively jQuery foundati ..
JS Foundation Jspresso Kanoah KMS Technology Kualitee LDRA Limited
Litmus LoadFocus Loadster Perfor .. MarathonITE Marketcircle Marketcircle
Maveryx Meliora Ltd Micro Focus Sof .. Microsoft Mobile Labs Mobile1st
Mockaroo, LLC Monkop Mozila MSys Technologi .. Navicat NeoTys
Neowise Softwar .. NetCart NORIZZK.COM Novosync Mobili .. NRG Global NTT Resonant
OC Systems Odin Technology OpCord Oracle Orcanos Original Softwa ..
OW2 PANAYA Parasoft PassMark Patterson Consu .. Perfecto Mobile
Pivotal, Inc. Plutora Postman (API To .. PractiTest PrimaTest Process One
Programming Res .. Psoda PureLoad PushToTest Python Q-Assurance
QA Systems QACube QASymphony QAWorks QMetry Quali
Qualitia Softwa .. Quality First S .. Quotium RadView Softwar .. Ranorex RedLine13
Reflective Solu .. ReQtest RevDeBug Robotium Tech Rogue Wave Soft .. Rommana Softwar ..
RTTS ruby-doc.org Runscope Sandklef GNU La .. Sauce Labs Seapine Softwar ..
SeleniumHQ Sencha Sensiple Siemens PLM Sof .. SmartBear Softw .. SmarteSoft
SOASTA SoftLogica Softomotive Softsmith Solution-Soft SonarSource
Sourceforge Spirent Technol .. SQS Software Qu .. Square Stimulus Techno .. Swifting AB
Synopsys T-komp T-Plan TechExcel TechTalk Telerik By Prog ..
Tellurium Test Collab Test Goat Test Recon TestCaseLab testCloud.de Gm ..
TestCraft Techn .. Testenium Testim.io TestingBot TestLodge Testmunk
Testomato TestOptimal TestPlant TestPro Testuff The Core Bankin ..
The MathWorks The Open Group thePHP.cc Thoughtbot Thoughtworks Tigris.org
Time Simulator Top-Q Trace Technolog .. TrendIC TRICENTIS Tritusa Pty Ltd
TWD Solutions P .. TypeMock Tyto Software Ubertesters UniTESK Universal Test ..
Usetrace Ltd Utrecht Univers .. Validata Group Vanamco AG Vector Software Veracode
Verifaya Corpor .. Verit VersionOne Viewlike.us Vornex Inc. Watir.com
WcfStorm Soluti .. We Are Mammoth Web Performance .. Wintask Wireshark Found .. Worksoft
Xceptance XK72 Xpand IT XQual ZAPTEST Zeenyx Software ..
Zephyr Zeta Software zutubi pty

Theme by Danetsoft and Danang Probo Sayekti