Enter the characters shown in the image.

You are here

The Testing Swing

Over the past few years, my team has moved from waterfall process to agile and now DevOps.

For organizations deploying new software in production several times a day, one of the most challenging transformation issue is the need to revolutionize testing. In an era of continuous deployment and updates, there’s no time to have QA teams identify a problem and kick it back to the developers. So times are gone when you were testing you app after the development and before going in production.

During this journey (read Our long journey towards DevOps) I’ve come up with a very simple framework that has changed my vision of testing (my main mission being product manager).

From the testing perspective of a feature, we try to answer to 4 different questions related to its maturity and its stage in the product lifecycle. Typically maturity is about the feature being an experiment or a validated functionality, whereas stage in the product lifecycle relates to the feature being actively developed or in production and maintenance stage.

This is basically the testing matrix:

Testing Matrix


What should we build?

At this stage, we get prepared for the coming sprint and we analyse the new features to be developed through the user stories. Each story describes the intent of the feature. For example: “as a user, I want to be able to import existing test cases from excel sheet into Hiptest”. Ok fine but what should we build exactly? This is where I’ve found Behavior Driven Development – BDD – a very powerful approach over the past few years.

It enables to align the team on the definition of what we should build. All the stake holders (tester, Devs, Product Owner, marketing) collaborate on:

  • Product acceptance criteria – the examples,
  • Business acceptance criteria – the assumptions we want to validate.

That’s the shift left. We define tests before building the features.

Guess what? We use Hiptest for BDD :). Eat your own dog food as they say.

Then, our dev team develops the feature AND the unit tests. We perform some exploratory testing and deploy the feature in production.


Is it worth?

Now we have the feature (or prototype) in the hands of real users. This phase is about measuring wether or not the business assumptions are met. Whenever the answer is no, then just throw away the code together WITH the tests and iterate again…

A good way to test the assumption is by measuring the change in behavior. For example: “does the import feature enables to increase the activation rate by 10%?” Measuring change in behavior is most effective way I’ve found to evaluate the impact of a feature.

Monitoring live customer behavior can also give us a lot of insight about the usability of our app. A button that is almost never clicked may be at the wrong place, a pop-up window closed after 5 seconds may need a more entertaining content, or shorter phrases.

So the biggest part during this first 2 steps is about business discovery (with BDD) and business validation (observe & measure). Testing is not about the product but about business AND the product that supports it.


How to automate?

Experiment was successful so let’s make this feature a first citizen class of our product. Now it is time to invest more in testing to ensure the service is delivered at scale: automate the BDD tests, do some performance testing and other-ity testing (security)

More importantly, tests should be integrated in the continuous integration process. They will give a rapid feedback on unanticipated side effects of changes. Speed is one of the most important attributes of tests to consider when evaluating their value.


Is it reliable and does it scale?

The critical thing is that while a server may go down, the application can’t. Scale and reliability are the big things, and testing has to be in real time.

That’s the shift right

We typically monitor two characteristics: performance and usability. We use tools like Appsignal to monitor our app and gain insight about its performance continuously. It helps to see how a feature behaves right after its deployment to production. Tuning performance of a feature for days before deployment may be a total waste if it did not meet customer expectations first. It is better to launch a feature fast, test its behavior in production and then adjust it once all assumptions made during development are validated. It’s the “Experiment, test and learn” motto.

As importantly, we also have a real-time user feedback (with live chat) to raise issues that might not be detected by the previous tools. Sometimes, wrong behaviors might not be due to errors in the code, but simply to a bad UX or will only appear when a certain mass of data is reached. For example at the beginning of Hiptest, we did not specify any kind of sort for displaying the folders and we did not yet have enough scenarios to detect the issue but users saw the problem when they started having hundreds of scenarios. There was no errors raised by the application, just annoyed users until we forced an alphabetical sort of scenarios.


Testing Swing

The biggest part of testing was previously performed after development activities and before going in production. No matter how hard you resist, this time is over.

Nowadays with the rise of DevOps and the acceleration of deployment cycles, testing has moved upstream and downstream. This is the testing swing!




About The Author: 

Laurent Py
has extensive experience in software testing as product owner at Smartesting (2004-2014) and now as Product manager at Hiptest Inc. He has been speaker at several conferences: IBM Innovate (Orlando, 2013), JFTL (Paris, 2014), Step-in summit (Bangalore, 2014), Starteast (Orlando, 2014).



Test Management

List mode
Table mode
Icons mode

Displaying 1 - 107 of 107

Title Testing Objectives Since Price Quotes Ping
TOSCA Testsuite DataBase testing, Data Warehouse Testing, Desktop testing, Mobile Testing, Pair-Wise Analysis, SAP Testing, Server testing, SOA testing, Test Cases Statistics, Web testing 2,008 Call Manufacturer for details Active
Katalon Studio Desktop testing, Mobile Testing, UI testing, Web testing 2,016 Free Use Active
DevTest Test Cases Statistics 1,998 2015: Free trial - 10 user licenses included, 2015: DevTest Live - Limited to 1 day/ No installation required, Call Manufacturer for details Active
Tarantula Test Cases Statistics 2,008 Free use 2014
Testlink 2,003 Free use Active
TestRail Test Cases Statistics 2,003 March 2015: TestRail Server - $239 per user, March 2015: TestRail Hosted - $25/month per user Active
Xray 2,013 Active
Bamboo 2,007 March 2015: On Demand : 1 agent 10 Jobs $10/mo ; 1 agent Unlimited jobs $50/mo , March 2015: On Demand : 10 agents Unlimited jobs $250/mo ; 25 agents Unlimited Jobs $500/mo , March 2015: Download: 10 Jobs on local agents $10 Starter / 1 agent Unlimited Jobs $800 , March 2015: Download: 10 agents Unlimited Jobs $4,000 / 25 agents Unlimited Jobs $8,000 Active
SpiraTest Test Cases Statistics 2,006 Active
Zephyr for JIRA Test Cases Statistics 2,007 Dec 2014 : starting at $10/10 users Active
Quality Center Enterprise (QC) Application Life Cycle Tracking, Test Cases Statistics 2,000 Call Manufacturer for more details Active
Hiptest 2,014 Active
Testopia Test Cases Statistics 2,007 Free use Active
CA Agile Central Application Life Cycle Tracking, Test Cases Statistics 2,001 Jan 2015: Community edition - Free for 10 users, 5 projects, Jan 2015: Enterprise edition - Free Trial, contact manufacturer for quote, Jan 2015: Unlimited edition - Free Trial, contact manufacturer for quote Active
RTH Application Life Cycle Tracking 2,006 Free use 2009
QMetry Test Management Test Cases Statistics, Application Life Cycle Tracking 2,008 Active
TestMagic Desktop testing, Mobile Testing, Server testing, SOA testing, Web testing 2,009 Active
Enterprise Tester Test Cases Statistics 2,009 Active
Squash TM (Test Management) 2,011 Free Use Active
Lean Testing 2,015 Free use Active
Klaros Testmanagement Test Cases Statistics 2,010 December 2015 : Base Price € 1,280 / 3 users, December 2015 : Support € 640 / year Active
Oracle Test Manager Test Cases Statistics 2,011 April 2015 : Named User Plus for perpetual term / user : US$2,000.00, April 2015 : Software Update License and Support (First Year) US$440.00 Active
Zephyr Enterprise Test Cases Statistics 2,007 2013-08-19: $80 user/month Active
OpKey Desktop testing, Mobile Testing, Server testing, SOA testing, Web testing, Cross-Browser testing 2,006 2016: Cloud-free - free, 2016: Cloud-Lite - $50/month Active
AscentialTest Application Life Cycle Tracking, Test Cases Statistics 2,006 Call Manufacturer for details Active
QPack 2,005 Active
Test Analytics Test Cases Statistics 2,011 Free to use 2012
XStudio Test Cases Statistics 2,007 May 2015: XStudio Community - Free, May 2015: XStudio Professional - $179 /user/year, May 2015: XStudio Business - $269 /user/year Active
QAComplete Application Life Cycle Tracking, Test Cases Statistics 2,003 Saas - Named License $399, Saas - Concurrent License $549, On Premises - Named License $899, On Premises - Concurrent License $1,149 Active
Silk Central 2,009 Active
SalomeTMF Test Cases Statistics 2,005 Free use 2011
TestBench DataBase testing, Data Migration Testing, Data Warehouse Testing, Database Upgrade Testing 2,004 Active
Qualify Application Life Cycle Tracking, Desktop testing, SAP Testing, Test Cases Statistics, Web testing, Cross-Browser testing 2,009 Active
Validata ATS Application Life Cycle Tracking, Banking Process validation, Message testing 2,004 Active
Gemini Test Management 2,003 Active
ApTest Manager 1,994 $595 each (perpetual license; one-time fee) Active
Test Studio Desktop testing, Java testing, Mobile Testing, Performance Analysis, Test Cases Statistics, Test Design, Web testing, Cross-Browser testing 2,002 Test Studio Web & Desktop $2,499 perpetual license, include: Automated testing for Web and Desktop Visual Studio plugin, Test Studio Load $999 perpetual license, include: Load testing web UI Load testing web services, Test Studio Ultimate $3,499 perpetual license include: Test Studio Web & Desktop, Mobile, Load, Performance + API testing Active
MozTrap Test Cases Statistics 2,005 Free to use Active
PractiTest Application Life Cycle Tracking, Test Cases Statistics 2,008 Active
Polarion QA 2,004 Call Manufacturer for details Active
Testitools Test Cases Statistics 2,002 Free Use 2003
Rational Quality Manager Test Cases Statistics, Application Life Cycle Tracking 2,008 May 2014 :IBM Rational Quality Manager Contributor Authorized User Single Install Initial Fixed Term License + Subscription & Support 12 Months: $775, IBM Rational Quality Manager Contributor Floating User Single Install Initial Fixed Term License + Subscription & Support 12 Months: $2320 Active
QABook Desktop Test Cases Statistics 2,014 Active
RapidRep Test Suite DataBase testing, Data Migration Testing, Data Warehouse Testing, Protocol Testing 2,012 Aug 2015 : Test Suite 699 €/license, Aug 2015 : Test Suite 1.328 €/2 licenses, Aug 2015 : Test Suite 2.874 €/5licenses Active
Pulse 2,006 August 2014 : Evaluation period : 30 days : free, August 2014 : Standard : $950/agent, August 2014 : Professional : $2500/ 5 agents, August 2014 : Enterprise : $5500/ 15 agents Active
Helix ALM 2,012 April 2015: Issue Management - $20/user/month, April 2015: Test Case Management - $30/user/month, April 2015: Requirements Management - $50/user/month, April 2015: TestTrack Complete ALM - $99/user/month Active
TestFLO Test Management for JIRA Application Life Cycle Tracking 2,016 Not Published Active
TestDrive-Assist 2,007 Active
Testuff Web testing, Test Cases Statistics, Application Life Cycle Tracking, Desktop testing 2,007 2015: Monthly subscription $27 per user, 2015: Yearly subscription $270 per user, 2015: Site License $1,840/month Unlimited Active
Stryka Application Life Cycle Tracking 2,016 Not Published Active
Adaptavist Test Management Test Cases Statistics 2,015 Active
VersionOne Lifecycle Application Life Cycle Tracking, Test Cases Statistics 2,002 August 2014 : 10 user pack : free, August 2014 : 20 user pack : $175/month, August 2014 : Enterprise : $29/user/month, August 2014 : Ultimate : $39/user/month Active
TestLodge 2,011 2015 : Basic - $14/month for 5 Test plans 100 Test cases, 2015 : Plus - $29 month for 15 Test plans 500 Test cases, 2015 : Premium - $59 month for 50 Test plans 3,000 Test cases, 2015 : Max - $119 month for 500 Test plans 10,000 Test cases Active
R.T.M.R Test Cases Statistics 2,011 Free use 2013
TestWave 2,011 June 2015 : Monthly Subscription Per User : Testwave = $15 , June 2015 : Monthly Subscription Per User : Testwave Pro = $30, June 2015 : Monthly Subscription Per User : Testwave Enterprise = $45 Active
Imbus TestBench 2,005 Active
Agile Test 2,010 Free use 2011
Test Collab 2,016 Not Published Active
SW Test Automation Framework Test Cases Statistics 1,998 Free Use Active
Ubertesters Platform Mobile Testing 2,012 Dec 2014 : Indie - $19 / 4 months, Dec 2014 : Startup - $59 / 6 months, Dec 2014 : Pro - $129 / 12 months Active
ReQtest Application Life Cycle Tracking 2,009 2016: 1-15 users - 56$/user , 2016: 100 users and more - 10$/user Active
CloudQA API testing, Desktop testing, Web testing, Cross-Browser testing 2,016 Active
Zeta Test 2,012 Oct 2015 : Zeta Test Management 3.6.2 - USD 599 per user, Oct 2015 : 10 users, USD 539 per user, Oct 2015 : 25 users, USD 509 per user Active
TestLog Test Cases Statistics 2,009 A TestLog Site License costs US$4750.00 Active
Wave Test Manager Web testing 2,005 June 2015 : Wave Test Manager : $89 per license 2012
SQS-TEST Professional Test Cases Statistics 2,010 Call Manufacturer for details Active
TETware Test Cases Statistics 2,009 Free use 2012
DevSuite Test Cases Statistics, Application Life Cycle Tracking 1,998 2015: DevSuite Free trial - 10 user licenses included, 2015: Try DevSuite Live - No installation required/ Limited to 1 day, Call Manufacturer for details Active
Q-Assurance Web testing, Cross-Browser testing, Back-End Service 2,013 November 2015 : $25 USD per user, per month Active
SCADE Suite API testing 1,999 Active
Mago Desktop testing 2,009 Free use 2010
Test Goat 2,015 Free Use Active
QAMonitor Test Cases Statistics 2,011 Active
Endeavour Agile ALM Cross-Browser testing, Test Cases Statistics 2,011 Free use 2013
Project Harmonizer Application Life Cycle Tracking, Test Cases Statistics 2,011 Please call manufacturer for more details Active
eggPlant Manager 2,010 Not Published Active
Quality Spy Test Cases Statistics 2,013 Free use Active
X-TRAX Test Cases Statistics 2,010 Please Contact Manufacturer for more details... 2016
CERTIFY IMPACT SAP Testing 2,011 Active
Testsigma 2,017 Not Published Active
TEST MANAGER - ALTM Test Cases Statistics 2,013 Call Manufacturer for details Active
TestUp Test Cases Statistics, Application Life Cycle Tracking 2,011 July 2014 : Standalone Pricing: 10 users : $400, July 2014 : Hosting Pricing: $5 per user per month Active
AppPerfect Test Manager Test Cases Statistics, Application Life Cycle Tracking 2,003 Active
Application Testing Suite SOA testing, Web testing 2,008 Call Manufacturer for details Active
Speed Test Test Cases Statistics 2,006 Free use 2011
Visual Studio Test Professional 2,012 Active
Assertible API testing, Web Monitoring, Web testing 2,016 Not Published Active
Parasoft Development Testing Platform Security Testing, Data Flow Analysis, Code Analysis, Static Code Analysis, Code Review 1,996 Call Manufacturer for details Active
Psoda Test Cases Statistics 2,006 Active
TESTMANAGER™ Banking Process validation 2,014 Active
Tellurium API testing 2,010 August 2014 : Individual License (1 user/1project) : $15/month, August 2014 : Professional License (5 users/10 projects) : $75/month, August 2014 : Enterprise License (20 users/20 projects) : $250/month Active
TestLab Application Life Cycle Tracking, Code Analysis, Code Coverage testing, Security Testing 2,012 Active
Rommana Application Life Cycle Tracking 2,012 Nov 2015 : Rommana-RM - $895 per user, Nov 2015 : Rommana-TM - $595 per user, Nov 2015 : Rommana-IM - $295 per user, Nov 2015 : Rommana-ALM - $2000 per user Active
Team Foundation Server Server testing, Web testing 2,010 Active
ALM Complete Application Life Cycle Tracking 2,003 August 2014 : SaaS-Standard $399, August 2014 : SaaS-Universal $549, August 2014 : Enterprise-Standard $899, August 2014 : Enterprise-Universal $1149 Active
Kualitee Application Life Cycle Tracking 2,012 October 2014 - Basic- $200Per Month /25 users, October 2014 - Professional - $400Per Month /unlimited users Active
expecco ALM 2,009 Active
accelQ 2,016 Not Published Active
TCLab 2,016 Basic - $14 / mo. 500 test cases limit, no other limits., Plus - $29 / mo. 1000 test cases limit, no other limits., Premium - - $59 / mo. 3000 test cases limit, no other limits. Active
Panaya Test Center 2,006 Not Published Active
EasyQA 2,016 Active
QACoverage 2,016 Not Published Active
T-PLAN PROFESSIONAL Code Analysis, Code Coverage testing 1,989 Call Manufacturer for details Active
AppAssist Mobile Testing, Performance Analysis 2,014 Active
NORIZZK Test Cases Statistics 2,014 2015: SINGLE - 1 Test manager, 2015: TEAM - 10 Test managers, 2015: CORPORATE - 30 Test managers Active
RapidRep QualityMaster 2,012 Active
TestRecon 2,016 Not Published Active



Total Manufacturers in our archive: 280
Testing tool manufacturers world-wide list
10Levels ABID CONSULTING AccelQ Accord Software ActiMind AdaCore
AdaLog AgileLoad AgileWay Agitar Algorismi ALL4TEC
Andreas Kleffel Android Apache Apica Apollo Systems
Applitools AppPerfect Appsee ApTest Assertible Assure
Atlassian AutoIt Consulti .. Automation Anyw .. Automation Cons .. Axosoft Aztaz Software
Backtrace I/O Badboy BlazeMeter Borvid BrowserStack BSQUARE
BStriker Intern .. CA Technologies Canonical Canoo Engineeri .. Catch Software CelestialTeapot
Chris Mallett Cleanscape ClicTest CloudQA Codeborne CodeCentrix
CodePlex projec .. Codoid Cogitek Compuware Configure IT Conflair
ConSol Core Services Coronys Ltd Countersoft CresTech Softwa .. CrossBrowserTes ..
Crosscheck Netw .. Crowdsourced Te .. Cucumber Ltd Cyara Cygnet Infotech DareBoost
Databene Datamatics Glob .. DevExpress DTM soft Dynatrace LLC EasyQA
Eclipse EkaTechserv Elvior Emmanuel Jorge Empirix EPAM Systems
Equafy Esterel Technol .. eXept Software .. Experitest Finaris Froglogic
FrontEndART Ltd GeneXus GitHub project gnoso Google Code Pro .. GrammaTech
Gurock Software HelpSystems HENIX Hewlett Packard .. Hexawise High-Tech Bridg ..
Hiptest Hitex IBM Rational imbus Shanghai Impetus Inflectra
informUp InTENSO - IT Ex .. Ipswitch Jamo Solutions Janova JAR Technologie ..
JBoss Developer jClarity JetBrains Jively jQuery foundati ..
JS Foundation Jspresso Kanoah KMS Technology Kualitee LDRA Limited
Litmus LoadFocus Loadster Perfor .. MarathonITE Marketcircle Marketcircle
Maveryx Meliora Ltd Micro Focus Sof .. Microsoft Mobile Labs Mobile1st
Mockaroo, LLC Monkop Mozila MSys Technologi .. Navicat NeoTys
Neowise Softwar .. NetCart NORIZZK.COM Novosync Mobili .. NRG Global NTT Resonant
OC Systems Odin Technology OpCord Oracle Orcanos Original Softwa ..
OW2 PANAYA Parasoft PassMark Patterson Consu .. Perfecto Mobile
Pivotal, Inc. Plutora Postman (API To .. PractiTest PrimaTest Process One
Programming Res .. Psoda PureLoad PushToTest Python Q-Assurance
QA Systems QACube QASymphony QAWorks QMetry Quali
Qualitia Softwa .. Quality First S .. Quotium RadView Softwar .. Ranorex RedLine13
Reflective Solu .. ReQtest RevDeBug Robotium Tech Rogue Wave Soft .. Rommana Softwar ..
RTTS Runscope Sandklef GNU La .. Sauce Labs Seapine Softwar ..
SeleniumHQ Sencha Sensiple Siemens PLM Sof .. SmartBear Softw .. SmarteSoft
SOASTA SoftLogica Softomotive Softsmith Solution-Soft SonarSource
Sourceforge Spirent Technol .. SQS Software Qu .. Square Stimulus Techno .. Swifting AB
Synopsys T-komp T-Plan TechExcel TechTalk Telerik By Prog ..
Tellurium Test Collab Test Goat Test Recon TestCaseLab Gm ..
TestCraft Techn .. Testenium TestingBot TestLodge Testmunk
Testomato TestOptimal TestPlant TestPro Testsigma Techn .. Testuff
The Core Bankin .. The MathWorks The Open Group Thoughtbot Thoughtworks Time Simulator Top-Q Trace Technolog .. TrendIC TRICENTIS
Tritusa Pty Ltd TWD Solutions P .. TypeMock Tyto Software Ubertesters UniTESK
Universal Test .. Usetrace Ltd Utrecht Univers .. Validata Group Vanamco AG Vector Software
Veracode Verifaya Corpor .. Verit VersionOne Vornex Inc. WcfStorm Soluti .. We Are Mammoth Web Performance .. Wintask Wireshark Found ..
Worksoft Xceptance XK72 Xpand IT XQual ZAPTEST
Zeenyx Software .. Zephyr Zeta Software zutubi pty

Theme by Danetsoft and Danang Probo Sayekti