Enter the characters shown in the image.

You are here

Top 10 Risks to Using a Test Suite Approach

Top 10 Risks to Using a Test Suite Approach

Here are the top 10 risks to keep in mind when taking a test suite approach to QA.

Test suites have long aided software development efforts, helping quality assurance professionals to effectively gauge the functionality and performance of an application. However, when team members rely too heavily on these tools, they may run into significant challenges that could threaten to derail production. Here are the top 10 risks to keep in mind when taking a test suite approach to QA:

  • Consume too much time -
    As a compilation of test cases, test suites can pack a lot of resources into a single package. However, creating those test cases can eat up a lot of time for QA members and prevent them from taking on other tasks.
  • Oversight may be limited -
    Accurately monitoring testing efforts is a crucial part of running an effective software development operation. Ineffective processes need to be identified and weeded out in order to enhance the overall quality of an organization's products. "Real World Haskell" authors Bryan O'Sullivan, Don Stewart and John Goerzen noted that without the aid of extensions such as Haskell Program Coverage, QA management 
    may miss out on this insight, essentially leaving them in the dark regarding which components have been tested.
  • Creating ineffective subsets - It's actually possible for a test suite to be too large for its own good. According to the University of Wisconsin-Milwaukee's Christine Cheng, there are a limited number of resources available to carry out test cases and testers often have difficulty finding a subset that will accommodate everything while still addressing the widest scope possible.
  • Reduction challenges linger - QA professionals have attempted to alleviate various test suite issues by finding ways to reduce the size of a given tool in certain circumstances. However, as software development experts Scott Mcmaster and Atif Memon explained, disruptive forces in the industry, like multi-language implementations and component reuse, have created obstacles to these efforts.
  • Build up resistance - QA expert Kelvin Kam explained that a major concern with regression test suites is the capacity for the software to become resistant over time to their effectiveness. This occurs when team members become complacent, relying on the same test suites and running them repeatedly without making any adjustments. Eventually, these tools will be unable to bring any new defects to light.
  • Updating automated tools - According to the School of Science and Computer Engineering at the University of Houston-Clear Lake, automated test suites can become ineffective if they are not diligently updated. QA professionals should establish a central point for them to make changes to various aspects of an automated test suite, particularly if they may change in the near future.
  • Stabilizing test suites - The University of Houston-Clear Lake also explained that a significant challenge that QA teams may run into when taking a test suite approach is maintaining stability of the tool. This is especially difficult when dealing with complex test suites. If not addressed, errors could arise when various test scripts are carried out simultaneously.
  • Accounting for licensed data - When attempting to build a test suite for Unstructured Information Management Architecture components, Philip Ogren and Steven Bethard, from the University of Colorado-Denver and Stanford University, respectively, ran into a few issues, including how to work around licensed data. In their instance, Ogren and Bethard needed their tool to read PennTreeBank content, but did not have the rights to do so. As such, they had to eliminate that portion of their testing.
  • Escalating costs - Automated test suites can be extremely beneficial when used in moderation, enabling QA management to quickly scan through code and identify certain defects. However, software engineering expert Cem Kaner argued that creating the test cases needed tocarry out these tasks can be a time-consuming and expensive process. If automated test suites are relied on too much, QA teams may see their budgets spiral out of control.
    -Some people recommend that testers automate 100 percent of their test cases. I strongly disagree with this,-Kaner said. -I create and run many black box tests only once. To automate these one-shot tests, I would have to spend substantially more time and money per test. In the same period of time, I wouldn't be able to run as many tests. Why should I seek lower coverage at a higher cost per test?-

  • Lack of complexity - Another concern that Kaner raised is the tendency for automated test suites to be overly simple and incapable of effectively scanning for a wide variety of defects. These tools may prove themselves to be more useful early on in the design process when the in-development application is far more basic. As production continues, these assets may begin to lose their value.

By no means should QA management and their teams avoid test suites - including automated iterations - altogether. These resources can be extremely useful for software testers and help them dig up a variety of issues. However, no single asset should be given too much focus, as they will each display some strengths and weaknesses. Quality development practices should include a comprehensive arsenal of testing tools to aid the efforts of QA professionals.




About The Author: 
Test Suite Approach


Testing tool manufacturers world-wide list
10Levels ABID CONSULTING AccelQ Accord Software ActiMind AdaCore
AdaLog AgileLoad AgileWay Agitar Algorismi ALL4TEC
Andreas Kleffel Android Apache Apica Apollo Systems
Applitools AppPerfect Appsee ApTest Assertible Assure
Atlassian AutoIt Consulti .. Automation Anyw .. Automation Cons .. Axosoft Aztaz Software
Backtrace I/O Badboy BlazeMeter Borvid BrowserStack BSQUARE
BStriker Intern .. CA Technologies Canonical Canoo Engineeri .. Catch Software CelestialTeapot
Chris Mallett Cleanscape ClicTest CloudQA Codeborne CodeCentrix
CodePlex projec .. Codoid Cogitek Compuware Configure IT Conflair
ConSol Core Services Coronys Ltd Countersoft CresTech Softwa .. CrossBrowserTes ..
Crosscheck Netw .. Crowdsourced Te .. Cucumber Ltd Cyara Cygnet Infotech DareBoost
Databene Datamatics Glob .. DevExpress DTM soft Dynatrace LLC EasyQA
Eclipse EkaTechserv Elvior Emmanuel Jorge Empirix EPAM Systems
Equafy Esterel Technol .. eXept Software .. Experitest Finaris Froglogic
FrontEndART Ltd GeneXus GitHub project gnoso Google Code Pro .. GrammaTech
Gurock Software HelpSystems HENIX Hewlett Packard .. Hexawise High-Tech Bridg ..
Hiptest Hitex IBM Rational imbus Shanghai Impetus Inflectra
informUp InTENSO - IT Ex .. Ipswitch Jamo Solutions Janova JAR Technologie ..
JBoss Developer jClarity JetBrains Jively jQuery foundati ..
JS Foundation Jspresso Kanoah KMS Technology Kualitee LDRA Limited
Litmus LoadFocus Loadster Perfor .. MarathonITE Marketcircle Marketcircle
Maveryx Meliora Ltd Micro Focus Sof .. Microsoft Mobile Labs Mobile1st
Mockaroo, LLC Monkop Mozila MSys Technologi .. Navicat NeoTys
Neowise Softwar .. NetCart NORIZZK.COM Novosync Mobili .. NRG Global NTT Resonant
OC Systems Odin Technology OpCord Oracle Orcanos Original Softwa ..
OW2 PANAYA Parasoft PassMark Patterson Consu .. Perfecto Mobile
Pivotal, Inc. Plutora Postman (API To .. PractiTest PrimaTest Process One
Programming Res .. Psoda PureLoad PushToTest Python Q-Assurance
QA Systems QACube QASymphony QAWorks QMetry Quali
Qualitia Softwa .. Quality First S .. Quotium RadView Softwar .. Ranorex RedLine13
Reflective Solu .. ReQtest RevDeBug Robotium Tech Rogue Wave Soft .. Rommana Softwar ..
RTTS Runscope Sandklef GNU La .. Sauce Labs Seapine Softwar ..
SeleniumHQ Sencha Sensiple Siemens PLM Sof .. SmartBear Softw .. SmarteSoft
SOASTA SoftLogica Softomotive Softsmith Solution-Soft SonarSource
Sourceforge Spirent Technol .. SQS Software Qu .. Square Stimulus Techno .. Swifting AB
Synopsys T-komp T-Plan TechExcel TechTalk Telerik By Prog ..
Tellurium Test Collab Test Goat Test Recon TestCaseLab Gm ..
TestCraft Techn .. Testenium TestingBot TestLodge Testmunk
Testomato TestOptimal TestPlant TestPro Testsigma Techn .. Testuff
The Core Bankin .. The MathWorks The Open Group Thoughtbot Thoughtworks Time Simulator Top-Q Trace Technolog .. TrendIC TRICENTIS
Tritusa Pty Ltd TWD Solutions P .. TypeMock Tyto Software Ubertesters UniTESK
Universal Test .. Usetrace Ltd Utrecht Univers .. Validata Group Vanamco AG Vector Software
Veracode Verifaya Corpor .. Verit VersionOne Vornex Inc. WcfStorm Soluti .. We Are Mammoth Web Performance .. Wintask Wireshark Found ..
Worksoft Xceptance XK72 Xpand IT XQual ZAPTEST
Zeenyx Software .. Zephyr Zeta Software zutubi pty

Theme by Danetsoft and Danang Probo Sayekti