Image CAPTCHA
Enter the characters shown in the image.

You are here

Performance Testing - Types and Tools

When QA plan performance tests, in most cases they mean type of Load Testing, but this is not the only option, here is few samples of testing types that can be generated in order to get the performance analyzing, that our application needs.
Every testing type can be also describe by the graph based time of the load activated on the SUT.
 
 
Spike Tests
Testing the software with sudden overloads, the main goal is validate that the software can handle situations where there is a sudden jump in the user activity.
 
Reliability Tests
Testing the system with different load for a long period of time, the main goal is to see that the system can handle sustained use for a long period of time.

Reliability Testing is a field of software testing that relates to testing a software's ability to function, given environmental conditions, for a particular amount of time. Software reliability testing helps discover many problems in the software design and functionality.

Software reliability is the probability that software will work properly in a specified environment and for a given amount of time. Using the following formula, the probability of failure is calculated by testing a sample of all available input states.

Probability = Number of failing cases / Total number of cases under consideration

The set of all possible input states is called the input space. To find reliability of software, we need to find output space from given input space and software.

For reliability testing, data is gathered from various stages of development, such as the design and operating stages. The tests are limited due to restrictions such as cost and time restrictions. Statistical samples are obtained from the software products to test for the reliability of the software. Once sufficient data or information is gathered, statistical studies are done. Time constraints are handled by applying fixed dates or deadlines for the tests to be performed. After this phase, design of the software is stopped and the actual implementation phase starts. As there are restrictions on costs and time, the data is gathered carefully so that each data has some purpose and gets its expected precision. To achieve the satisfactory results from reliability testing one must take care of some reliability characteristics. For example Mean Time to Failure (MTTF) is measured in terms of three factors:

  1. operating time,
  2. number of on off cycles,
  3. and calendar time.

If the restrictions are on operation time or if the focus is on first point for improvement, then one can apply compressed time accelerations to reduce the testing time. If the focus is on calendar time (i.e. if there are predefined deadlines), then intensified stress testing is used

Software reliability is measured in terms of mean time between failures(MTBF).

MTBF consists of mean time to failure (MTTF) and mean time to repair(MTTR). MTTF is the difference of time between two consecutive failures and MTTR is the time required to fix the failure. Reliability for good software is a number between 0 and 1. Reliability increases when errors or bugs from the program are removed.

For example, if MTBF = 1000 hours for average software, then the software should work for 1000 hours for continuous operations.

Types of Reliability testing

Software reliability testing includes feature testing, load testing, and regression testing.

Feature test

Feature testing checks the features provided by the software and is conducted in the following steps:

  1. Each operation in the software is executed once.
  2. Interaction between the two operations is reduced and
  3. Each operation is checked for its proper execution.

The feature test is followed by the load test.

Load test

This test is conducted to check the performance of the software under maximum work load. Any software performs better up to some amount of workload, after which the response time of the software starts degrading. For example, a web site can be tested to see how many simultaneous users it can support without performance degradation. This testing mainly helps for Databases and Application servers. Load testing also requires software performance testing, which checks how well some software performs under workload.

Regression test

Regression testing is used to check if any new bugs have been introduced through previous bug fixes. Regression testing is conducted after every change or update in the software features. This testing is periodic, depending on the length and features of the software.

 

Soak Testing
Also known at Endurance Testing, Testing a system with a typical production load, over a continuous availability period, to validate system behavior under production use.
Soak tests are used primarily to check the reaction of a subject under test under a possible simulated environment for a given duration and for a given threshold. Observations made during the soak test are used to improve the characteristics of the subject under further tests.
 
Load Testing
Testing the software with increasing set of data (Near to the software limitation), the main goal is to see that the software can handle such loads without having any side effects (unexpected crushes, memory leaks, bottlenecks Etc.).
Load testing is the process of putting demand on a software system or computing device and measuring its response. Load testing is performed to determine a system's behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation. When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test.
There is little agreement on what the specific goals of load testing are.The term is often used synonymously with concurrency testing, software performance testing, reliability testing, and volume testing. 
Load testing is usually a type of non-functional testing although it can be used as a functional test to validate suitability for use.
 
Concurrency Testing
Concurrent testing is a software testing activity that determines the stability of a system or application under test during normal activity. Concurrent testing is the exercise of running continuous testing with functional testing concurrently in order to discover defects that would not otherwise be detected without the additional activity. Concurrent tests commonly put a greater emphasis on robustness, performance, and system integration with production-like activity, which should determine correct behavior of the system under normal circumstances.
 
 
Stress Tests
Testing the software under load that exceed the system limitation, the main goal is to cause system failures ,  analyze the crash, and determine the way that the software recover in case of such failure.

Stress testing tries to break the system under test by overwhelming its resources or by taking resources away from it (in which case it is sometimes called negative testing). The main purpose of this process is to make sure that the system fails and recovers gracefully—a quality known as recoverability.

Concurrent testing implies a controlled environment staying at a constant level of activity. Stress testing focuses on more random events, chaos and unpredictability.

Stress testing (sometimes called torture testing) is a form of deliberately intense or thorough testing used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. Reasons can include:

  • to determine breaking points or safe usage limits
  • to confirm mathematical model is accurate enough in predicting breaking points or safe usage limits
  • to confirm intended specifications are being met
  • to determine modes of failure (how exactly a system fails)
  • to test stable operation of a part or system outside standard usage
 
Scalability Testing
Testing the software performance while increasing the volume of users/environment and determine how the software will adjust to the growing scale.

Scalability Testing, is the testing of a software application to measure its capability to scale up or scale out in terms of any of its non-functional capability.

Performance, scalability and reliability testing are usually grouped together by software quality analysts.

The main goals of scalability testing are to determine the user limit for the web application and ensure end user experience, under a high load, is not compromised. For example, can a web page be accessed in a timely fashion with limited delay in response. Another goal is to check the server can cope i.e. Will the server crash if it is under a heavy load? 

Dependent on the application that is being tested, different parameters are tested. If a webpage is being tested, the highest possible number of simultaneous users would be tested. Also dependent on the application being tested is the attributes that are tested - these can include CPU usage, network usage or user experience.

Successful testing will project most of the issues which could be related to the network, database or hardware/software.

 

Fail-Over Testing / Fault-Tolerance Tests
Testing that the software corresponding to failures while involving two or more physical nodes, in this type of tests, we need to validate that in case of failure the users can continue with their work and the application data/services are moved to the available “Active” node.

Fault Tolerant refers to the ability to respond to an unforeseen catastrophe on the application landscape. This facet of performance is extremely crucial when you have a load balanced environment, active/passive DR backups, clustered JVMs, have your application on cloud in different availability zones. If you do not make sure that the application under test is still performant enough when one of the nodes go down, be ready for a surprise after it rolls into production. Same is the case with active/passive DR backups. In these cases, a lot depends on how you manage your user sessions. Different approaches exist with different outcomes depending on whether a session is managed on a web/app server or a database server.

Fault tolerance tests can be as simple as removing the cable out of one of the servers to actively working alongside the tests to bring down a few nodes, switching over to DR on the fly. This test is targeted at specific application types and implementations and is not a mandatory one for all applications.

 

Redundancy Tests
Testing the application redundancy mechanism in case of load, the main goal is to determine the effectiveness of the balancing system. 
 
Volume Tests

Testing the application with different amount of data, the main goal here is to understand the limits of our application and demonstrate how a specific amount of data can affect the predefined requirements.

 

Performance Testing types

 

 

 
About The Author: 

 


 

.

Performance Analysis

List mode
Table mode
Icons mode

Displaying 1 - 52 of 52

Title Testing Objectives Since Price Quotes Ping
eggPlant Functional Application Life Cycle Tracking, Cross Platform Testing, Desktop testing, Image Based Testing, Embedded Software, Image Comparison, Java testing, Mobile Testing, Network testing, Performance Analysis, Web testing, Cross-Browser testing 2,008 Active
Parasoft SOAtest API testing, Banking Process validation, DataBase testing, Interoperability testing, Stress Testing, Message testing, Performance Analysis, Protocol Testing, Security Testing, SOA testing, Web Monitoring, Web testing, Cross-Browser testing 2,002 Call manufacturer for more details Active
NeoLoad Stress Testing, Mobile Testing, Performance Analysis, Web testing 2,005 Please Call Manufacturer for more details, April 2012: Total Unit price start at $1300, January 2012: Unit price start at $1860 Active
StormRunner Load Mobile Testing, Performance Analysis, Web testing 2,013 Not Published Active
Fiddler Load Capabilities, Performance Analysis, Security Testing, Web testing, Cross-Browser testing 2,003 Free use Active
Silk Performer Load Capabilities, Mobile Testing, Performance Analysis, SAP Testing, Web testing 1,994 Active
WebLOAD Load Capabilities, Stress Testing, Mobile Testing, Performance Analysis, Web testing 1,993 Call Manufacturer for details Active
JMeter API testing, Load Capabilities, Stress Testing, Network testing, Performance Analysis, Server testing, SOA testing, Web testing 1,999 Free Use Active
AppLoader Desktop testing, Image Based Testing, EMail Testing, Image Comparison, Java testing, Load Capabilities, Stress Testing, Performance Analysis, SAP Testing, Web testing 2,000 Please Call Manufacturer for more details... Active
QF-Test Desktop testing, Image Comparison, Java testing, Load Capabilities, Performance Analysis, SAP Testing, Test Cases Statistics, Test Design, Web testing, Cross-Browser testing 2,001 Developer license (Test creation and execution): One GUI-technology included. 1,995 €, GUI-technology extensions: Purchase now or later on demand. 500 €, Maintenance (obligatory for initial license purchase): All Upgrades + support for 1 year with one GUI-technology. 480 €, Runtime license (Test execution): Substantial volume discounts for load testing. 995 € Active
SandStorm Stress Testing, Mobile Testing, Performance Analysis, Protocol Testing 2,011 Active
StresStimulus SAP Testing, Mobile Testing, Network testing, Performance Analysis, Web testing, Cross-Browser testing, Test Cases Statistics, Server testing, DataBase testing, Load Capabilities, Stress Testing, Java testing 2,006 September 2015: Free version is available, September 2015: 250 VUs, Pro Edition, Permanent license - $870, September 2015: 10,000 VUs, Ent. Edition, weekly pass - $2,130 Active
LoadUI NG Pro Network testing, Performance Analysis, Web testing 2,008 Active
Test Studio Desktop testing, Java testing, Mobile Testing, Performance Analysis, Test Cases Statistics, Test Design, Web testing, Cross-Browser testing 2,002 Test Studio Web & Desktop $2,499 perpetual license, include: Automated testing for Web and Desktop Visual Studio plugin, Test Studio Load $999 perpetual license, include: Load testing web UI Load testing web services, Test Studio Ultimate $3,499 perpetual license include: Test Studio Web & Desktop, Mobile, Load, Performance + API testing Active
HammerDB DataBase testing, Stress Testing, Performance Analysis 2,003 Free use Active
WAPT Load Capabilities, Stress Testing, Performance Analysis, Web testing 2,010 2016: WAPT, $700 (server + 2 agents), 2016: WAPT PRO, $1200 (server + 2 agents) Active
Load Tester DataBase testing, Data Migration Testing, Java testing, Stress Testing, Mobile Testing, Network testing, Performance Analysis, Cross-Browser testing 1,999 1 Month Lease $750 Active
SiteScope Performance Analysis, Web testing 2,008 Call Manufacturer Active
Loadster Load Capabilities, Stress Testing, Network testing, Performance Analysis, SOA testing, Web testing 2,004 2015: Loadster Cloud-1000 units : $79, 2015: Loadster Cloud-10000 units: $499 , 2015: Loadster Engine 3.x : $699 Active
BlazeMeter Desktop testing, Stress Testing, Performance Analysis, Web testing, Cross-Browser testing 2,011 October 2014 :Basic - $249/month/1000 users, October 2014 :Pro - $649/month/5000 users, October 2014 :Pro Plus - $1349/month/10,000users Active
eggPlant Performance Load Capabilities, Stress Testing, Performance Analysis, SOA testing, Web testing 2,008 Active
RTI Performance Analysis 1,999 Active
TestMaker Desktop testing, Performance Analysis, SOA testing, Web testing 2,000 Active
CLIF Performance Analysis, Stress Testing 2,007 Free Use Active
mAutomate Mobile Testing, Performance Analysis 2,011 Active
Topaz Data Flow Analysis, Performance Analysis 2,015 Active
Runscope API testing, Performance Analysis, SOA testing, Web Services, Web Monitoring, Web testing 2,013 Active
Tsung Protocol Testing, Network testing, Performance Analysis, SOA testing, Web Services, Web testing, Web Monitoring, Server testing, Load Capabilities, Stress Testing, Memory Analysis, Runtime Error Detection, Memory Leak Detection 2,001 Free Use 2015
OpenSTA Load Capabilities, Stress Testing, Performance Analysis, Test Cases Statistics, Web testing 2,003 Free use 2007
OctoPerf Load Capabilities, Stress Testing, Performance Analysis 2,015 2015 - Basic 39€ (3 purchased minimum) , 2015 - Advanced 499€, 2015 - Enterprise 999€ Active
AQtime Pro Code Analysis, Code Coverage testing, Performance Analysis 2,011 2016: Node-Locked (Named User) Licenses = $599, 2016: CONCURRENT USER LICENSE = $1899 Active
Apica ZebraTester Load Capabilities, Stress Testing, Performance Analysis, Web testing 2,015 Active
StressTester Mobile Testing, Network testing, Performance Analysis, SOA testing, Web testing, Server testing, Message testing, DataBase testing, Database Upgrade Testing, Load Capabilities, Stress Testing 2,003 Nov 2015 : Controller including 25 Virtual Users - $2,049, Nov 2015 : Controller including 100 Virtual Users - $4,950, Nov 2015 : Controller including 500 Virtual Users - $15,939 Active
TestQuest 10 Mobile Testing, Performance Analysis, Load Capabilities, Stress Testing 1,999 Active
MStone Protocol Testing, Performance Analysis, Message testing, Load Capabilities, Stress Testing, EMail Testing 2,005 Free Use 2013
PureLoad Stress Testing, Performance Analysis, Web testing 1,998 PureLoad Web - $5000/500 virtual clients, PureLoad Enterprise - $12000/500 virtual clients, PureLoad Enterprise, Developers Edition - $1250/10 virtual clients Active
mPulse Metrics Analysis, Performance Analysis, SOA testing 2,006 July 2015: Lite - Free/ 1 user, July 2015: Pro - $99/month Active
AppPerfect LoadTest Java testing, Stress Testing, Performance Analysis, Web testing 2,003 Feb 2015 : Desktop licence : $299/ 25 VU, Feb 2015 : Floating Licence : $399/ 25 VU, Feb 2015: Additional Virtual Users : $1/ VU Active
LoadTracer Load Capabilities, Stress Testing, Performance Analysis, Web testing 2,001 Call Manufacturer for details Active
Monkop Security Testing, Mobile Testing, Network testing, Performance Analysis, Metrics Analysis, Memory Analysis 2,014 Nov 2015: Solo - Free, Nov 2015: Basic - $90 / month / 5 Multi-Device executions per month , Nov 2015: Pro - $349 / month / 30 Multi-Device executions per month Active
TestWise Code Analysis, Code Coverage testing, Code Review, Static Code Analysis, Performance Analysis 2,002 Call Manufacturer for details 2011
FlowLevel Java testing, Stress Testing, Performance Analysis, Protocol Testing, Server testing 2,004 Active
jClarity Illuminate Performance Analysis 2,014 April 2015: 14-day free trial, April 2015: Basic - $24 Monthly / Per Server*, April 2015: Pro - $119 Monthly / Per Server*, April 2015: Enterprise - Contact Manufacturer for details Active
AgileLoad Load Capabilities, Memory Analysis, Mobile Testing, Performance Analysis, Web testing 2,006 May 2015: 50 Virtual users - $24 / day, May 2015: Free License - upto 10 virtual users Active
Loadzen Stress Testing, Network testing, Performance Analysis, Server testing, Web testing 2,012 Free use: upto 25 virtual users Active
Commercial Tools
SmarteLoad Load Capabilities, Stress Testing, Performance Analysis, Web testing 2,007 Call Manufacturer for details Active
OpenWebLoad Performance Analysis, Web testing 2,001 Free use 2001
AppAssist Mobile Testing, Performance Analysis 2,014 Active
PurpleStrike RT Load Capabilities, Performance Analysis 2,014 Active
JAR Load Load Capabilities, Back-end Stress, Stress Testing, Performance Analysis, Server testing, Uptime Testing, Web testing 2,010 Active
LoadFocus API testing, Load Capabilities, Performance Analysis, Website Speed Testing, Web Monitoring 2,013 Not Published Active

MANUFACTURERS Wallboard

Testing tool manufacturers world-wide list
10Levels ABID CONSULTING AccelQ Accord Software ActiMind AdaCore
AdaLog AgileLoad AgileWay Agitar Algorismi ALL4TEC
Andreas Kleffel Android Apache Apica Apollo Systems AppAssist.mobi
Applitools AppPerfect Appsee ApTest Assertible Assure
Atlassian AutoIt Consulti .. Automation Anyw .. Automation Cons .. Axosoft Aztaz Software
Backtrace I/O Badboy BlazeMeter Borvid BrowserStack BSQUARE
BStriker Intern .. CA Technologies Canonical Canoo Engineeri .. Catch Software CelestialTeapot
Chris Mallett Cleanscape ClicTest CloudQA Codeborne CodeCentrix
CodePlex projec .. Codoid Cogitek Compuware Configure IT Conflair
ConSol Core Services Coronys Ltd Countersoft CresTech Softwa .. CrossBrowserTes ..
Crosscheck Netw .. Crowdsourced Te .. Cucumber Ltd Cyara Cygnet Infotech DareBoost
Databene Datamatics Glob .. DevExpress DTM soft Dynatrace LLC EasyQA
Eclipse EkaTechserv Elvior Emmanuel Jorge Empirix EPAM Systems
Equafy Esterel Technol .. eXept Software .. Experitest Finaris Froglogic
FrontEndART Ltd GeneXus GitHub project gnoso Google Code Pro .. GrammaTech
Gurock Software HelpSystems HENIX Hewlett Packard .. Hexawise High-Tech Bridg ..
Hiptest Hitex IBM Rational imbus Shanghai Impetus Inflectra
informUp InTENSO - IT Ex .. Ipswitch Jamo Solutions Janova JAR Technologie ..
JBoss Developer jClarity Jellly.io JetBrains Jively jQuery foundati ..
JS Foundation Jspresso Kanoah KMS Technology Kualitee LDRA Limited
Litmus LoadFocus Loadster Perfor .. MarathonITE Marketcircle Marketcircle
Maveryx Meliora Ltd Micro Focus Sof .. Microsoft Mobile Labs Mobile1st
Mockaroo, LLC Monkop Mozila MSys Technologi .. Navicat NeoTys
Neowise Softwar .. NetCart NORIZZK.COM Novosync Mobili .. NRG Global NTT Resonant
OC Systems Odin Technology OpCord Oracle Orcanos Original Softwa ..
OW2 PANAYA Parasoft PassMark Patterson Consu .. Perfecto Mobile
Pivotal, Inc. Plutora Postman (API To .. PractiTest PrimaTest Process One
Programming Res .. Psoda PureLoad PushToTest Python Q-Assurance
QA Systems QACube QASymphony QAWorks QMetry Quali
Qualitia Softwa .. Quality First S .. Quotium RadView Softwar .. Ranorex RedLine13
Reflective Solu .. ReQtest RevDeBug Robotium Tech Rogue Wave Soft .. Rommana Softwar ..
RTTS ruby-doc.org Runscope Sandklef GNU La .. Sauce Labs Seapine Softwar ..
SeleniumHQ Sencha Sensiple Siemens PLM Sof .. SmartBear Softw .. SmarteSoft
SOASTA SoftLogica Softomotive Softsmith Solution-Soft SonarSource
Sourceforge Spirent Technol .. SQS Software Qu .. Square Stimulus Techno .. Swifting AB
Synopsys T-komp T-Plan TechExcel TechTalk Telerik By Prog ..
Tellurium Test Collab Test Goat Test Recon TestCaseLab testCloud.de Gm ..
TestCraft Techn .. Testenium Testim.io TestingBot TestLodge Testmunk
Testomato TestOptimal TestPlant TestPro Testsigma Techn .. Testuff
The Core Bankin .. The MathWorks The Open Group thePHP.cc Thoughtbot Thoughtworks
Tigris.org Time Simulator Top-Q Trace Technolog .. TrendIC TRICENTIS
Tritusa Pty Ltd TWD Solutions P .. TypeMock Tyto Software Ubertesters UniTESK
Universal Test .. Usetrace Ltd Utrecht Univers .. Validata Group Vanamco AG Vector Software
Veracode Verifaya Corpor .. Verit VersionOne Viewlike.us Vornex Inc.
Watir.com WcfStorm Soluti .. We Are Mammoth Web Performance .. Wintask Wireshark Found ..
Worksoft Xceptance XK72 Xpand IT XQual ZAPTEST
Zeenyx Software .. Zephyr Zeta Software zutubi pty

Theme by Danetsoft and Danang Probo Sayekti