Performance Testing Tools Limitations 711d1q

Comparison between QALoad, WebLOAD, OpenSTA and LoadRunner

Documented By: - Sravan Tadakamalla Comparison Between QALoad, WebLOAD, OpenSTA and LoadRunner LoadRunner

QALoad

Virtual Generator

Script Development Workbench, Player

Controller

Conductor

Analysis

Analyze

WebLOAD

OpenSTA

WebLOAD IDE

Modeler

WebLOAD Console

Commander

SCRIPT GENERATORS: Virtual Generator (VuGen)(LoadRunner): VuGen is used to record the flow of the application called scripts. The entire recording part is done in the VuGen including any enhancements in the scripts (parameterization-including the defined variables which contains the values which will be substituted at the RunTime, correlation-handle the dynamic values, content checks, & sleep time inclusion). The recording can be done in many protocols (e.g. HTTP/HTML for Web application). LoadRunner s nearly 44 different protocols. Before recording, we need to set the recording options as per our requirements. Recording options can be available from the start record wizard or from the Tools menu. After recording, the script consists of three parts, viz. v_init, Action, and v_end. The v_init contains that part of script which is the starting of the script (e.g. into any eCommerce site), Action contains that part which the actually performs his operation inside the website (e.g. going through the shopping site and buying different products), v_end contains the part when is leaving the site. The reason why the script is divided is, whenever we want the script to Run for no. of iterations only the Action part is to be executed repeatedly. After completing all the enhancements for the script, we do a sample Run in the VuGen. Before running we will set the Run-Time Settings for the script. Run-Time Settings can be available from the Tools Menu option. These Run-Time Settings includes the options like setting the script to execute for no. of iterations and the time to wait between each iteration, amount of think time to include in the script, log options, browser to be used, proxy if any and so on. Parameterization is done by selecting and right-clicking the string to be parameterized and going through the wizard by giving the values to be substituted and a variable which contain them.

Comparison between QALoad, WebLOAD, OpenSTA and LoadRunner Correlation is done basically in three ways in LoadRunner. They are 1. Autocorrelation (Ctrl+F8) 2. Manual Correlation(using web_reg_save( ) ) 3. Correlation Studio In correlation studio we create some rules, by giving the left boundary and right boundary, and these rules are automatically checked and correlate the values while we record a new script. This happens when we include them through the Record Options. Script Development WorkBench (QALoad): The recording part is done in the script development workbench same as VuGen of LoadRunner. Here we will get the script and that is to be enhanced and Run in the Player. Player is a different component from WorkBench. We need to the script and Run in the Player.  

  



•

Here, the Recording Options can be found in the Session Menu -> Record.. option. Runtime Settings are seen as Playback Options and these are seen with the vistree(Visual Navigator Tree). Parameterization can also be done using this view's menu items. The parameter values can be taken from the Datapool Files. Parameterization is also called Variablization (in previous versions). Correlation is done by using the Extract String wizard or manually. We have Parameterization Rules which acts similar to Correlation Studio in LoadRunner. Here, vistree view can be considered as tree view of LoadRunner. In workbench, to get the script we need to right click on some part of the vistree and click on the view script. The tool does not give us the editable script and to get this, we need to save the current script with some other name either by using save as option from File Menu or right clicking on the vistree and selecting Create Editable Script option. As we have three sections of the script named v_init, Action, and v_end in LoadRunner we have Transaction setup,Transaction Loop, Transaction cleanup in QALoad workbench. In workbench, while recording we have checkpoints in place of transactions.

The recorded script is first available as a capture file with .cap as extension, which then to be converted to vistree. We have two different conversion options that we can decide after recording the script( here capturing the script), they are HTML & HTTP forms. HTTP is same as URL based recording in LoadRunner. In LoadRunner’s VuGen, we can Run the script for number of iterations by single , where as in QALoad’s WorkBench, we can Run the script for only one iteration and this test-run is called validating the script. To test-run the script for no. of iterations, we make use of Player. In LoadRunner we record and test-run the script in the VuGen itself but in QALoad to test-run the script Player is used. Here in Player we validate the script to check whether it is ready to be Run in the Conductor for multiple iterations. WebLOAD IDE(WebLOAD): {Issues: • There was some problem with the correlation of the dynamic values. This problem is due to the trial version of the WebLOAD tool. Where as in licensed version there will be a separate option when

Comparison between QALoad, WebLOAD, OpenSTA and LoadRunner •

we right click on the code in the DOM view. Also the report creation is not so friendly. There are a lot of statistics to be considered. Here there is no separate tool for analyzing in the trial version. }

In WebLOAD, recording is done in IDE same as VuGen. Here we will get the script and we have two types of views, Agenda View & JavaScript View. Agenda View is same as Tree View and JavaScript View is same as Script View of LoadRunner. • Unlike LoadRunner and QALoad, here in WebLOAD, we do not have parts in the script. • One demerit of WebLOAD is we can not include transaction names at the time of recording; instead we have to manually drag and drop the Begin Transaction, End Transaction icons after the particular URL which are available in the Load Tab of the Toolbox available in left pane. We also have timers called Set Timer, Send Timer which acts as transaction start and end resp. and also sends the time to the Console while Load test. • We can Run the script in the IDE to check whether the script is ready for the Load test. • We call compilation process as Check Syntax in IDE. • Parameterization is done by dragging and dropping the Global Input File in the Agenda Tree from the General Tab of the Toolbox in the left pane. • Correlation is done by catching the hidden fields in the DOM view and extracting that values and asg to a variable. Modeler(OpenSTA): Modeler works the same way as VuGen. Modeler is used for recording the scripts. TEST EXECUTION ENVIRONMENTS: Controller(LoadRunner): We create the Load scenario and Run it in the Controller. The scenario is the predefined method for the test to Run. This scenario can include one or more number of scripts. The scenario which includes more than one script is called Group Scenario. When we Run the group scenario, all the scripts are being Run simultaneously with assigned no. of s. The scenarios can be of two types based on the functionalities: Manual Scenario & Goal-Oriented Scenario. Goal-Oriented Scenario: In the Goal-Oriented scenario, s can set the pre-requisites of the test like maximum number of hits to be achieved, throughput, etc. These can be called as goals of the test to be achieved. Manual Scenario: In the manual scenario, s don’t have the choice of setting the goals. The scenario will Run normally. In Manual Scenario, we have percentile mode option. Here we also have scheduled tests through which we can Run the test at a later time. LoadGenerator: The load (no. of vs) can be distributed among other systems using the LoadGenerator. LoadGenerator can assign the load to different machines, and simulate a different group of s performing different activities on the application being tested at the same time. And this

Comparison between QALoad, WebLOAD, OpenSTA and LoadRunner ‘different group of s’ belongs to different machines. Reporting: The LoadRunner’s Controller is capable of generating graphs which show different metrics like transactions per second, hits per second, etc. separately. LoadRunner sends the statistics of the test to the Analysis in the form of reports. Debugging: We can see the flow of the scripts running when we select the ‘view test log’ by rightclicking on the script name. We can monitor the sever performance by using the Monitors functionality in the LoadRunner’s Controller. Monitors allow us to mention the server’s resources to be monitored while test is running. At the end of the Load Test, we will get the statistics of the server resources through monitors. We can also select Databases to monitor. Conductor(QALoad): The Conductor of QALoad is used to Run the Load Test. This serves us similar to the Controller of LoadRunner. The term we use in conductor is session in place of scenario of controller. We don’t find the scenario types based on the functionalities here, as in LoadRunner. QALoad has ‘batch test’ which is the collection of one or more sessions. We can add one or more no. of scripts to the single test session. The batch test will execute the scripts in it linearly one after the other and not simultaneously as in LoadRunner. We Run the Load test in the Conductor by configuring the machine by giving the machine name or IP address manually. In conductor we can assign the VUs (vs) to each machine. We can configure and add different machines for test using the configure machines wizard. This is same as what LoadGenerator does in the LoadRunner. Here in conductor instead of ramp up and ramp down time we will specify the time interval, starting VU’s and ending VU’s. It automatically takes the time. The reports generated for the test in the Console are sent to the Analyze module and we can also view the graphs in the Console while test is running. And as soon as we exit out of the test, the Analyze module will open showing the graphs and reports. We have Monitors functionality in QALoad’s Conductor to monitor the server performance while the Load Test is running. Apart from this functionality, QALoad also comes up with some additional tools/modules like Application Vantage and Server Vantage. These modules give the advantage for the Testers in running Load Tests and for Performance Tuning issues. Console(WebLOAD): In Console, we call the scenario of test as Agenda. An Agenda is a test-script used to test our application. WebLOAD s:

Comparison between QALoad, WebLOAD, OpenSTA and LoadRunner • •

Single Agendas- scenarios which contain single test script. Mix of Agendas- a combination of Agendas, each simulating a different group of s performing different activities on the application being tested at the same time.

We don’t find the scenarios’ types based on the functionalities here as in LoadRunner. Console is the module which acts as both Controller and Analysis modules of LoadRunner. In Console, we can Run the test and simultaneously we can generate the reports. We also have different tools like Performance Measurement Manager (PMM) and Report Manager which are additional tools used for monitoring the server and better analysis of the test. (These are available for licensed versions) There is an advantage with this tool which allows us to save the Load Session by which we can open in a later time and add reports for the load session. We use the template and load session in Console, where template is the settings applied to Run the load session and load session is the resultant test. That is, template is similar to scenario in LoadRunner, when we save the template the scripts allocated and the load balancing between them is saved. Whereas load session is like the resultant report we get in the Analysis after the test using which we analyze the test. We can export the reports to Microsoft Excel sheet and also as a HTML document. REPORT ANALYZING ENVIRONMENT: Analysis: The Analysis module of the LoadRunner is used for analyzing the test we ran in the Controller. We can analyze the test by using the different graphs available and the reported values for different metrics. Using the Analysis module, we can create the custom reports using different metrics we consider for the particular application. We can export the reports to Microsoft Word document and also as a HTML document. Analyze: The Analyze module works in the similar way as LoadRunner’s Analysis module. We can export the reports to Microsoft Word document and also as a HTML document.