Copyright 2004 © IDEALX S.A.S. -  http://tsunami.idealx.org/

IDX-Tsunami User's manual

Version: 1.10
Date : July 20, 2004

IDX-Tsunami user's manual

Table of Contents

1  Introduction

1.1  What is IDX-Tsunami ?

IDX-Tsunami is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP and Jabber servers.

It is distributed under the GNU General Public License version 2.

1.2  What is Erlang and why is it important for IDX-Tsunami ?

IDX-Tsunami main strength is its ability to simulate a huge number a simultaneous user from a single CPU. When used on cluster you can generate a really impressive load on a server with a modest cluster, easy to set-up and to maintain.

IDX-Tsunami is developed in Erlang and this is where the power of IDX-Tsunami relies.

Erlang is a concurrency-oriented programming language. Tsunami is based on the Erlang OTP (Open Transaction Platform) and inherits several characteristics from Erlang: More informations on Erlang on http://www.erlang.org and http://www.erlang-projects.org/

1.3  IDX-Tsunami background

History: IDX-Tsunami has been used for very high load tests: IDX-Tsunami has been used at:

2  Features

2.1  IDX-Tsunami main features

2.2  HTTP related features

2.3  Jabber related features

2.4  Complete reports set

Measures and statistics produced by Tsunami are extremely feature-full. They are all represented as a graphic. IDX-Tsumami produces statistics regarding: Note that IDX-Tsunami take care of the synchronisation process by itself. Gathered statistics are «synchronized».

It is possible to generate graphs during the benchmark as statistics are gathered in real-time.

2.5  Highlights

IDX-Tsunami has several advantages over other injection tools:

3  Installation

This package has only be tested on Linux. It should work on Erlang supported platforms (Solaris, *BSD, Win32 and MacOS-X).

3.1  Dependencies

3.2  Compilation


 
./configure
 make
 make install

3.3  Configuration

The main configuration file is ~/.idx-tsunami/idx-tsunami.xml ( there is a sample file /usr/share/doc/idx-tsunami/examples/idx-tsunami.xml).

Log files are saved in ~/.idx-tsunami/log/ . A new subdirectory is created for each test using the current date as name (~/.idx-tsunami/log/20040217-09:40 for ex.)

3.4  Feedback

Use the idx-tsunami mailing list (see http://lists.idealx.org/info/idx-tsunami) if you have suggestions or questions about IDX-Tsunami.

For any questions concerning commercial support, ask solutions@idealx.com.

4  HTTP benchmark approach

  1. Record scenario: start the recorder with: idx-tsunami recorder, and then configure your browser to use IDX-Tsunami proxy recorder (the listen port is 8090).
  2. Edit / organize scenario
  3. Write small code for dynamic parts if needed and place dynamic mark-up in the scenario
  4. Test and adjust scenario to have a nice progression of the load. This is highly dependent of the application and of the size of the target cluster. Calculate the normal duration of the scenario and use the interarrival time between users and the duration of the phase to estimate the number of simultaneous users for each given phase.
  5. Launch benchmark with your first application parameters set-up: idx-tsunami start
  6. wait for the end of the test or stop by by hand with idx-tsunami stop
  7. Analyse results, change parameters and relaunch another benchmark

5  Understanding idx-tsunami.xml configuration file

5.1  File structure

Scenarios are enclosed into idx-tsunami tags:
 
<?xml version="1.0"?>
<!DOCTYPE idx-tsunami SYSTEM "idx-tsunami-1.0.dtd" [] >
<idx-tsunami loglevel="info" dumptraffic="false">
...
</idx-tsunami>

5.2  Clients and server

Scenarios start with a clients (Tsunami cluster) and server definition:
 
  <clients>
     <client host="louxor" weight="1" maxusers="500">
         <ip value="10.9.195.12"></ip>
         <ip value="10.9.195.13"></ip>
     </client>
     <client host="memphis" weight="3" maxusers="250" cpu="2">
         <ip value="10.9.195.14"></ip>
     </client>
  </clients>

  <server host="10.9.195.1" port="8080" type="tcp"></server>

Several virtual IP can be used to simulate more machines. This is very useful when a load-balancer use the client's IP to distribute the traffic among a cluster of servers.

In this example, a second machine is used in the Tsunami cluster, with a higher weight, and 2 cpus. Two Erlang virtual machines will be used to take advantage of the number of CPU.

The server is the entry point into the cluster (Only one server should be defined).

5.3  Monitoring

Scenarios can contain optional monitoring informations. For example, here is a cluster monitoring definition based on Erlang agents, for a cluster of 6 computers:
 
  <monitoring>
    <monitor host="geronimo" type="erlang"></monitor>
    <monitor host="bigfoot-1" type="erlang"></monitor>
    <monitor host="bigfoot-2" type="erlang"></monitor>
    <monitor host="f14-1" type="erlang"></monitor>
    <monitor host="f14-2" type="erlang"></monitor>
    <monitor host="db" type="erlang"></monitor>
  </monitoring>

The type keyword snmp can replace the erlang keyword, if SNMP monitoring is preferred. They can be mixed. erlang is the default value for monitoring.

Note: For Erlang monitoring, monitored computers need to be accessible through the network. SSH needs to be configured to allow connection without password on.

5.4  Defining the load progression

The load progression is set-up by defining several arrival phases:
 
  <arrivalphase phase="1" duration="10" unit="minute">
    <users interarrival="2" unit="second"> </users>
  </arrivalphase>

  <arrivalphase phase="2" duration="10" unit="minute">
    <users interarrival="1" unit="second"> </users>
  </arrivalphase>

  <arrivalphase phase="3" duration="10" unit="minute">
    <users interarrival="0.1" unit="second"> </users>
  </arrivalphase>

5.5  Default values

Default values can be set-up globally: thinktime between requests in the scenario and ssl cipher algorithms. These values overrides those set in session configuration tags. 
  <default name="thinktime" value="3" random="false"/>
  <default name="ssl_ciphers" 
           value="EXP1024-RC4-SHA,EDH-RSA-DES-CBC3-SHA"/>

Default values for specific protocols can be defined. Here is an example of default values for Jabber:
 
  <default type="ts_jabber" name="global_number" value="5" />
  <default type="ts_jabber" name="userid_max" value="100" />
  <default type="ts_jabber" name="domain" value="jabber.org" />
  <default type="ts_jabber" name="username" value="glop" />
  <default type="ts_jabber" name="passwd" value="glop" />

5.6  Sessions

Sessions define the content of the scenario itself. They describe the requests to execute.
 
  <session name="http-example" popularity="70" type="ts_http">

    <request> <http url="/" method="GET" version="1.1">
                    </http> </request>
    <request> <http url="/images/logo.gif"
               method="GET" version="1.1" 
               if_modified_since="Fri, 14 Nov 2003 02:43:31 GMT">
              </http></request>

    <thinktime value="20" random="true"></thinktime>

    <transaction name="index_request">
     <request><http url="/index.en.html"
                          method="GET" version="1.1" >
              </http> </request>
     <request><http url="/images/header.gif"
                          method="GET" version="1.1">
              </http> </request>
    </transaction>

    <thinktime value="60" random="true"></thinktime>
    <request>
      <http url="/" method="POST" version="1.1"
               contents="bla=blu">
      </http> </request>
    <request>
       <http url="/bla" method="GET" version="1.1"
             contents="bla=blu&name=glop">
       <www_authenticate userid="Aladdin"
                         passwd="open sesame"/></http>
    </request>
  </session>

  <session name="backoffice" popularity="30" ...>
  ... </session>

The popularity is the frequency of this type of session. This is used to decided which session a new user will execute. The sum of all session's popularity must be 100.

This example show several features of the HTTP protocol support in Tsunami: GET and POST request, basic authentication, transaction for statistics definition, ...

Here is an example of a session definition for the Jabber protocol: 
 <session popularity="70" name="jabber-example" type="ts_jabber">

    <request> <jabber type="connect" ack="no_ack" />
                    </request>
    <thinktime value="2"></thinktime>
    <transaction name="authenticate">
      <request> <jabber type="authenticate" ack="local">
                      </jabber> </request>
    </transaction>

    <request> <jabber type="presence" ack="no_ack"/> 
                </request>
    <thinktime value="2"></thinktime>

    <request> <jabber type="register" ack="no_ack" 
                     id="new"></jabber> </request>

    <transaction name="roster">
      <request><jabber type="iq:roster:set" ack="no_ack" 
                              destination="offline" />
      </request>
      <request><jabber type="presence:roster" ack="no_ack"
                              destination="previous"/>
      </request>
      <request><jabber type="iq:roster:set" ack="no_ack"
                             destination="online"/>
      </request>
      <request><jabber type="iq:roster:get" ack="no_ack">
                  </jabber> </request>
    </transaction>
    <thinktime value="2"></thinktime>

    <request> <jabber type="chat" ack="no_ack"
                           size="56"/></request>
    <thinktime value="30"></thinktime>

    <transaction name="global_msg">
    <request> <jabber type="chat" ack="global" size="56"
                           destination="random"/></request>
    </transaction>

    <thinktime value="30"></thinktime>

    <transaction name="online">
    <request> <jabber type="chat" ack="no_ack" size="16"
                          destination="online"/></request>
    </transaction>
    <thinktime value="30"></thinktime>

    <transaction name="offline">
      <request> <jabber type="chat" ack="no_ack" size="56"
                             destination="offline"/><request>
    </transaction>

    <thinktime value="30"></thinktime>

    <transaction name="close">
      <request> <jabber type="close" ack="local">
                </jabber></request>
    </transaction>

  </session>

  <session popularity="30" name="jabber-example2" ...>
  ... </session>

5.7  Dynamic substitutions

Dynamic substitution are mark-up placed in element of the scenario. For HTTP, this mark-up can be placed in basic authentication (www_authenticate tag: userid and passwd attributes), URL (to change GET parameter) and POST content.

Those mark-up are of the form %%Module:Function%%. Substitutions are executed on a request-by-request basis, only if the request tag has the attribute subst="true".

When a substitution is asked, the substitution mark-up is replaced by the result of the call to the Erlang function: Module:Function(Pid).

Here is an example of use of substitution in a Tsunami scenario:
 
<session name="rec20040316-08:47" popularity="100" type="ts_http">
 <request subst="true">
  <http url="/echo?symbol=%%symbol:new%%" method="GET">
  </http></request>
</session>

Here is the Erlang code of the module used for dynamic substitution:
 
-module(symbol).
-export([new/1]).

new(Pid) ->
    case random:uniform(3) of
        1 -> "IBM";
        2 -> "MSFT";
        3 -> "RHAT"
    end.

As you can this, writing scenario with dynamic substitution is trivial.

If you want to set unique id, you can use the function ts_user_server:get_unique_id. 
<session name="rec20040316-08:47" popularity="100" type="ts_http">
 <request subst="true">
  <http url="/echo?id=%%ts_user_server:get_unique_id%%" method="GET">
  </http></request>
</session>

5.8  Dynamic variables

In some cases, you may want to use a value given by the server in a response later in the session, and this value is dynamically generated by the server for each user. For this, you can use <dyn_variable> in the scenario

Let's take an example with HTTP. You can easily grab a value in a HTML form like: 
<form action="go.cgi" method="POST">
<hidden name="random_num" value="42"></form>
</form>


 <request>
   <http url="/testtsunami.html" method="GET" version="1.0"></http>
   <dyn_variable name="random_num" ></dyn_variable>
 </request>

Now random_num will be set to 42 for the user session. It's value will be replace in all mark-up of the form %%_random_num%% if and only if the request tag has the attribute subst="true", like:
 
    <request subst="true">
      <http url='/go.cgi' version='1.0' 
      contents='username=nic&amp;random_num=%%_random_num%%&amp;op=login' 
      content_type='application/x-www-form-urlencoded' method='POST'>
      </http>
    </request>
If the dynamic value is not a form variable, you can set a regexp by hand, for example to get the title of a HTML page: 
    <request>
      <http url="/testtsunami.html" method="GET" version="1.0"></http>
      <dyn_variable name="mytitlevar" 
                    regexp="&lt;title&gt;\(.*\)&lt;/title&gt;"/>
    </request>

6  Statistics and reports

Available stats: HTTP specific stats:

6.1  Generating the report

cd to the log directory of your test (say ~/.idx-tsunami/log/20040325-16:33/) and use the script analyse_msg.pl:
 
/usr/lib/idx-tsunami/bin/analyse_msg.pl --stats idx-tsunami.log  --html --extra  --plot

6.2  Tsunami summary

images/tsunami-report.png

Figure 1: Report

6.3  Graphical overview

images/tsunami-graph.png

Figure 2: Graphical output

7  References

8  Acknowledgements

The first version of this document is based on a talk given by Mickael Rémond4 during an Object Web benchmarking workshop in April 2004 (more info at http://jmob.objectweb.org/).

A  Frequently Asked Questions

A.1  IDX-tsunami crash when I start it

Does your Erlang system has ssl support enabled ?

to test it: 
  > erl
  Eshell V5.2  (abort with ^G)
  1> ssl:start()
  you should see 'ok'

A.2  IDX-tsunami still doesn't start ...

Most of the time, when a crash happened at startup without any traffic generated, the problem arise because the main Erlang controller node cannot create a "slave" Erlang virtual machine. The message looks like:
 
===============================================
=ERROR REPORT==== 4-May-2004::22:38:26 ===
** Generic server ts_config_server terminating
** Last message in was {'$gen_cast',{newbeam,myshortname,[]}}
** When Server state == {state,{config,
                                     undefined,
                                     5,
                                     full,
                                     undefined,
                                     [{client,
                                          "myshortname",
                                          2.00000,
                                          5,
                                          [{10,68,133,140}]}],
                                     {server,"foo.net",80,gen_tcp},
                                     [],
                                     [{arrivalphase,
                                          1,
                                          60,
                                          undefined,
                                          undefined,
                                          5.00000e-5,
                                          infinity}],
                                     undefined,
                                     [{session,
                                          1,
                                          100,
                                          ts_http,
                                          parse,
                                          true,
                                          undefined}],
                                     14,
                                     3,
                                     7,
                                     6,
                                     "negociate"},
  "/home/username/.idx-tsunami/log/20040204-18:32",
                               undefined,
                               0,
                               undefined,
                               2.00000}
** Reason for termination ==
** {{badmatch,{error,timeout}},
    [{ts_config_server,handle_cast,2},
     {gen_server,handle_msg,6},
     {proc_lib,init_p,5}]}

IDX-Tsunami launches a new erl virtual machine to do the actual injection even when you have only one machine in the injection cluster. This is because it need to by-pass some limit with the number of open socket from a single process (1024 most of the time). The idea is to have several system processes (Erl beam) that can handle only a small part of the network connection from the given computer. When the maxclient limit (simultaneous) is reach, a new Erlang beam is launched and the newest connection can be handle by the new beam).

The problem is that the Erlang slave module cannot start a local slave node. It tries to start it with the short node name "myshortname" (erl -sname myshortname). If this fails the injection process cannot start. Most of the time, adding the short name with the correct IP address in the /etc/hosts file is sufficient to make it work.

Note that you do not need to use the 127.0.0.1 address in the config file. It will not work if you use it as the injection interface. The shortname of your client machine should not refer to this address.

A.3  IDX-tsunami still crash/fails when I start it !

First look at the log file ~/.idx-tsunami/log/XXX/tsunami_controller@yourhostname' to see if there is a problem.

Remember that you can validate your configuration file using the provided DTD !

If you see nothing wrong, you can compile idx-tsunami with full debugging: recompile with make debug , and don't forget to set the loglevel to "debug" in the XML file.

To start the debugger or see what happen, start IDX-Tsunami with the debug argument instead of start. You will have an erlang shell on the tsunami_controller node. Use toolbar:start(). to launch the graphical tools provided by Erlang.

A.4  What is the format of the stats file idx-tsunami.log ?


 
# stats: dump at 1083694995
stats: users 11 11
stats: request 41 1.03289 0.125108 1.59802 0.901978
stats: connect 41 0.220170 6.67110e-2 0.494019 0.171997
stats: users_count 11 11
stats: page 24 6.80416 17.2794 80.4609 0.958984
stats: size 26818 26818
stats: 404 7 7
stats: 200 20 20
# stats: dump at 1083695005
stats: users 21 21
stats: request 113 1.03980 0.110650 1.59802 0.791016
stats: connect 118 0.197619 4.26037e-2 0.494019 0.163940
stats: users_count 10 21
stats: page 52 2.72266 1.74204 80.4609 0.791016
stats: size 78060 104878
stats: 404 15 22
stats: 200 51 71
 ...

the format is, for request, page, session:

# stats:'name' count(during the last 10sec), mean, stdvar, max, min

or for HTTP returns code, size ...

# stats:'name' count(during the last 10sec), totalcount(since the beginning)

A.5  How can i specify the number of concurrent users ?

You can't. But it's on purpose: the load generated by IDX-Tsunami is dependent on the arrival time between new clients. Indeed, once a client has finished his session in idx-tsunami, it stops. So the number of concurrent users is a function of the arrival rate and the mean session duration.

For example, if your web site has 1000 visits/hour, the arrival rate is 1000/3600 = 0.2778 visits/second. If you want to simulate the same load, set the inter-arrival time is to 1/0.27778 = 3.6 sec (<users interarrival="3.6" unit="second"> in the arrivalphase node in the XML config file).

A.6  SNMP monitoring doesn't work ?!

There is a small bug in the snmp\_mgr module (R9C-0 release). You have to apply this patch to make it work. This is fixed in version R9C-1 and up.
 
--- lib/snmp-3.4/src/snmp_mgr.erl.orig  2004-03-22 15:21:59.000000000 +0100
+++ lib/snmp-3.4/src/snmp_mgr.erl       2004-03-22 15:23:46.000000000 +0100
@@ -296,6 +296,10 @@
     end;
 is_options_ok([{recbuf,Sz}|Opts]) when 0 < Sz, Sz =< 65535 ->
     is_options_ok(Opts);
+is_options_ok([{receive_type, msg}|Opts]) ->
+    is_options_ok(Opts);
+is_options_ok([{receive_type, pdu}|Opts]) ->
+    is_options_ok(Opts);
 is_options_ok([InvOpt|_]) ->
     {error,{invalid_option,InvOpt}};
 is_options_ok([]) -> true.

1
http://www.erlang-projects.org/Members/mremond/events/dossier_de_presentat/block_10766817551485/file
2
http://www.editions-eyrolles.com/php.accueil/Ouvrages/ouvrage.php3?ouv_ean13=9782212110791
3
http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf
4
mickael.remond@erlang-fr.org

Copyright © 2004 IDEALX S.A.S.. « Linux » is the property of Linus Torvalds. All other trademarks belong to their respective owners.

This document was translated from LATEX by HEVEA.