Copyright 2004 © IDEALX S.A.S. -  http://tsunami.idealx.org/

IDX-Tsunami User's manual

Version: 1.17
Date : September 5, 2005

IDX-Tsunami user's manual

Table of Contents

1  Introduction

1.1  What is IDX-Tsunami ?

IDX-Tsunami is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP and Jabber servers.

It is distributed under the GNU General Public License version 2.

1.2  What is Erlang and why is it important for IDX-Tsunami ?

IDX-Tsunami main strength is its ability to simulate a huge number a simultaneous user from a single CPU. When used on cluster you can generate a really impressive load on a server with a modest cluster, easy to set-up and to maintain.

IDX-Tsunami is developed in Erlang and this is where the power of IDX-Tsunami relies.

Erlang is a concurrency-oriented programming language. Tsunami is based on the Erlang OTP (Open Transaction Platform) and inherits several characteristics from Erlang: More informations on Erlang on http://www.erlang.org and http://www.erlang-projects.org/

1.3  IDX-Tsunami background

History: IDX-Tsunami has been used for very high load tests: IDX-Tsunami has been used at:

2  Features

2.1  IDX-Tsunami main features

2.2  HTTP related features

2.3  Jabber related features

2.4  Complete reports set

Measures and statistics produced by Tsunami are extremely feature-full. They are all represented as a graphic. IDX-Tsunami produces statistics regarding: Note that IDX-Tsunami take care of the synchronization process by itself. Gathered statistics are «synchronized».

It is possible to generate graphs during the benchmark as statistics are gathered in real-time.

2.5  Highlights

IDX-Tsunami has several advantages over other injection tools:

3  Installation

This package has only be tested on Linux. It should work on Erlang supported platforms (Solaris, *BSD, Win32 and MacOS-X).

3.1  Dependencies

3.2  Compilation


 
./configure
 make
 make install

3.3  Configuration

The main configuration file is ~/.idx-tsunami/idx-tsunami.xml ( there is a sample file /usr/share/doc/idx-tsunami/examples/idx-tsunami.xml).

Log files are saved in ~/.idx-tsunami/log/ . A new subdirectory is created for each test using the current date as name (~/.idx-tsunami/log/20040217-09:40 for ex.)

3.4  Feedback

Use the idx-tsunami mailing list (see http://lists.idealx.org/info/idx-tsunami) if you have suggestions or questions about IDX-Tsunami.

4  HTTP benchmark approach

  1. Record scenario: start the recorder with: idx-tsunami recorder, and then configure your browser to use IDX-Tsunami proxy recorder (the listen port is 8090). A session file will be created. For HTTPS recording, use http://{ instead of https:// in your browser.
  2. Edit / organize scenario
  3. Write small code for dynamic parts if needed and place dynamic mark-up in the scenario
  4. Test and adjust scenario to have a nice progression of the load. This is highly dependent of the application and of the size of the target server(s). Calculate the normal duration of the scenario and use the interarrival time between users and the duration of the phase to estimate the number of simultaneous users for each given phase.
  5. Launch benchmark with your first application parameters set-up: idx-tsunami start
  6. Wait for the end of the test or stop by hand with idx-tsunami stop (reports can also be generated during the test (see § 7) : the statistics are updated every 10 seconds). For a brief summary of the current activity, use idx-tsunami status
  7. Analyse results, change parameters and relaunch another benchmark

4.1  benchmarking a HTTP proxy server

By default, the HTTP plugin is used to benchmark HTTP servers. But you can also benchmark HTTP Proxy server. To do that, you must add :
 
  <default type="ts_http" name="http_use_server_as_proxy" value="true"></default>

5  Jabber benchmark approach

This paragraph explain how to write a session for Jabber.

There are two difference between HTTP and Jabber testing:
  1. There is no recorder for Jabber, so you have to write your sessions by hand (an example is provided in 6.6).
  2. the jabber plugin do not parse XML; instead it use packets acknowledgement. The rest of this paragraph will explain this feature.
Since the jabber plugin do not parse XML (historically, it was for performance reasons), and also due to the bidirectional nature of the jabber protocol, you must have a way to say when a request is finished. There are 3 possibilities:
ack=local
as soon as a packet is received from the server, the request is consider over. Hence if you use a local ack with a request that do not require a response from the server (presence for ex.), it will wait forever (or until a timeout is reached).
ack=no_ack
as soon as the request is send, it is consider over (do not wait for incoming data)
ack=global
synchronized users. its main use is for waiting for all users to connect before sending messages. To do that, set a request with global ack (it can be the first presence msg:
 
   <request> <jabber type="presence" ack="global"/> </request>
)

You also have to specify the number of users to be connected:
 
<default type="ts_jabber" name="global_number" value="100"></default>

To be sure that exactly global_number users are started, add the 'maxnumber' attribute to 'users'
 
    <users maxnumber="100" interarrival="1.0" unit="second"></users>

If you do not specify maxnumber, the global ack will be reset every global_number users

6  Understanding idx-tsunami.xml configuration file

6.1  File structure

Scenarios are enclosed into idx-tsunami tags:
 
<?xml version="1.0"?>
<!DOCTYPE idx-tsunami SYSTEM "/usr/share/idx-tsunami/idx-tsunami-1.0.dtd" [] >
<idx-tsunami loglevel="info" dumptraffic="false">
...
</idx-tsunami>

6.2  Clients and server

Scenarios start with clients (IDX-Tsunami cluster) and server definitions:
 
  <clients>
     <client host="louxor" weight="1" maxusers="500">
         <ip value="10.9.195.12"></ip>
         <ip value="10.9.195.13"></ip>
     </client>
     <client host="memphis" weight="3" maxusers="250" cpu="2">
         <ip value="10.9.195.14"></ip>
     </client>
  </clients>

  <server host="10.9.195.1" port="8080" type="tcp"></server>

Several virtual IP can be used to simulate more machines. This is very useful when a load-balancer use the client's IP to distribute the traffic among a cluster of servers.

In this example, a second machine is used in the Tsunami cluster, with a higher weight, and 2 cpus. Two Erlang virtual machines will be used to take advantage of the number of CPU.

The server is the entry point into the cluster (Only one server should be defined).

6.3  Monitoring

Scenarios can contain optional monitoring informations. For example, here is a cluster monitoring definition based on Erlang agents, for a cluster of 6 computers:
 
  <monitoring>
    <monitor host="geronimo" type="erlang"></monitor>
    <monitor host="bigfoot-1" type="erlang"></monitor>
    <monitor host="bigfoot-2" type="erlang"></monitor>
    <monitor host="f14-1" type="erlang"></monitor>
    <monitor host="f14-2" type="erlang"></monitor>
    <monitor host="db" type="erlang"></monitor>
  </monitoring>

The type keyword snmp can replace the erlang keyword, if SNMP monitoring is preferred. They can be mixed. erlang is the default value for monitoring. (Note: SNMP is currently not working with erlang R10B, use R9C-2 if you need snmp monitoring).

Note: For Erlang monitoring, monitored computers need to be accessible through the network. SSH needs to be configured to allow connection without password on. You must use the same version of Erlang/OTP on all nodes otherwise it may not work properly !

6.4  Defining the load progression

The load progression is set-up by defining several arrival phases:
 
  <arrivalphase phase="1" duration="10" unit="minute">
    <users interarrival="2" unit="second"> </users>
  </arrivalphase>

  <arrivalphase phase="2" duration="10" unit="minute">
    <users interarrival="1" unit="second"> </users>
  </arrivalphase>

  <arrivalphase phase="3" duration="10" unit="minute">
    <users interarrival="0.1" unit="second"> </users>
  </arrivalphase>

With this setup, during the first 10 minutes of the test, a new user will be created every 2 seconds, then during the next 10 minutes, a new user will be created every seconds, and for the last 10 minutes, 10 users will be generated every seconds. The test will finish when all users have ended their session.

The load generated in terms of HTTP requests / seconds will also depend on the mean number of requests within a session (if you have a mean value of 100 requests per session and 10 new users per seconds, the theoretical average throughput will be 1000 requests/ sec).

6.5  Default values

Default values can be set-up globally: thinktime between requests in the scenario and ssl cipher algorithms. These values overrides those set in session configuration tags if override is true. 
  <default name="thinktime" value="3" random="false" override="true"/>
  <default name="ssl_ciphers" 
           value="EXP1024-RC4-SHA,EDH-RSA-DES-CBC3-SHA"/>

6.5.1  Default values: Jabber

Default values for specific protocols can be defined. Here is an example of default values for Jabber:
 
  <default type="ts_jabber" name="global_number" value="5" />
  <default type="ts_jabber" name="userid_max" value="100" />
  <default type="ts_jabber" name="domain" value="jabber.org" />
  <default type="ts_jabber" name="username" value="myuser" />
  <default type="ts_jabber" name="passwd" value="mypasswd" />

Using these values, users will be myuserXXX where XXX is an integer in the interval [1:userid_max] and passwd mypasswdXXX

6.5.2  Default values: HTTP

For HTTP, you can set the UserAgent values (available since Idx-Tsunami 1.1.0), using a frequency for each value (the sum of all frequencies must be equal to 100)
 
  <default type="ts_http" name="user_agent">
    <user_agent frequency="80">
       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Galeon/1.3.21
    </user_agent>
    <user_agent frequency="20">
      Mozilla/5.0 (Windows; U; Windows NT 5.2; fr-FR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4
    </user_agent>
  </default>

6.6  Sessions

Sessions define the content of the scenario itself. They describe the requests to execute.
 
  <session name="http-example" popularity="70" type="ts_http">

    <request> <http url="/" method="GET" version="1.1">
                    </http> </request>
    <request> <http url="/images/logo.gif"
               method="GET" version="1.1" 
               if_modified_since="Fri, 14 Nov 2003 02:43:31 GMT">
              </http></request>

    <thinktime value="20" random="true"></thinktime>

    <transaction name="index_request">
     <request><http url="/index.en.html"
                          method="GET" version="1.1" >
              </http> </request>
     <request><http url="/images/header.gif"
                          method="GET" version="1.1">
              </http> </request>
    </transaction>

    <thinktime value="60" random="true"></thinktime>
    <request>
      <http url="/" method="POST" version="1.1"
               contents="bla=blu">
      </http> </request>
    <request>
       <http url="/bla" method="GET" version="1.1"
             contents="bla=blu&name=glop">
       <www_authenticate userid="Aladdin"
                         passwd="open sesame"/></http>
    </request>
  </session>

  <session name="backoffice" popularity="30" ...>
  ... </session>

The popularity is the frequency of this type of session. This is used to decide which session a new user will execute. The sum of all session's popularity must be 100.

This example show several features of the HTTP protocol support in Tsunami: GET and POST request, basic authentication, transaction for statistics definition, ...



Here is an example of a session definition for the Jabber protocol: 
 <session popularity="70" name="jabber-example" type="ts_jabber">

    <request> <jabber type="connect" ack="no_ack" /> </request>

    <thinktime value="2"></thinktime>

    <transaction name="authenticate">
      <request> <jabber type="authenticate" ack="local"> </jabber> </request>
    </transaction>

    <request> <jabber type="presence" ack="no_ack"/> </request>

    <thinktime value="2"></thinktime>

    <transaction name="roster">
      <request><jabber type="iq:roster:get" ack="local"> </jabber> </request>
    </transaction>

    <thinktime value="30"></thinktime>

    <transaction name="online">
    <request> <jabber type="chat" ack="no_ack" size="16" destination="online"/></request>
    </transaction>
    <thinktime value="30"></thinktime>

    <transaction name="offline">
      <request> <jabber type="chat" ack="no_ack" size="56" destination="offline"/><request>
    </transaction>

    <thinktime value="30"></thinktime>

    <transaction name="close">
      <request> <jabber type="close" ack="local"> </jabber></request>
    </transaction>
  </session>

6.7  Dynamic substitutions

Dynamic substitution are mark-up placed in element of the scenario. For HTTP, this mark-up can be placed in basic authentication (www_authenticate tag: userid and passwd attributes), URL (to change GET parameter) and POST content.

Those mark-up are of the form %%Module:Function%%. Substitutions are executed on a request-by-request basis, only if the request tag has the attribute subst="true".

When a substitution is asked, the substitution mark-up is replaced by the result of the call to the Erlang function: Module:Function(Pid, DynData) where Pid is the erlang process id of the current virtual user and DynData the list of all Dynamic variables (Warn: before version 1.1.0, the argument was just the Pid !).

Here is an example of use of substitution in a Tsunami scenario:
 
<session name="rec20040316-08:47" popularity="100" type="ts_http">
 <request subst="true">
  <http url="/echo?symbol=%%symbol:new%%" method="GET">
  </http></request>
</session>

Here is the Erlang code of the module used for dynamic substitution:
 
-module(symbol).
-export([new/1]).

new({Pid, DynData}) ->
    case random:uniform(3) of
        1 -> "IBM";
        2 -> "MSFT";
        3 -> "RHAT"
    end.

As you can this, writing scenario with dynamic substitution is trivial.

If you want to set unique id, you can use the built-in function ts_user_server:get_unique_id. 
<session name="rec20040316-08:47" popularity="100" type="ts_http">
 <request subst="true">
  <http url="/echo?id=%%ts_user_server:get_unique_id%%" method="GET">
  </http></request>
</session>

6.7.1  Reading external file

New in 1.0.3: A new experimental module ts_file_server is available. You can use it to read external files. For example, if you need to read user names and passwd from a csv file, you can do it with it.

You have to add this in the xml config file: 
 <default name="file_server"  value="/tmp/userlist.csv"></default>

Now you can build you own function to use it, for example:
 
-module(readcsv).
-export([user/1]).

user(Pid)->
    {ok,Line} = ts_file_server:get_next_line(), 
    [Username, Passwd] = string:tokens(Line,";"),
    "username=" ++ Username ++"&amp;passwd=" ++ Passwd.

in your session, use something like: 
  <request subst="true">
    <http url='/login.cgi' version='1.0' contents='%%readcsv:user%%&amp;op=login'
    content_type='application/x-www-form-urlencoded' method='POST'>
    </http>
  </request>

Two functions are available: ts_file_server:get_next_line and ts_file_server:get_random_line

6.8  Dynamic variables

In some cases, you may want to use a value given by the server in a response later in the session, and this value is dynamically generated by the server for each user. For this, you can use <dyn_variable> in the scenario

Let's take an example with HTTP. You can easily grab a value in a HTML form like: 
<form action="go.cgi" method="POST">
<hidden name="random_num" value="42"></form>
</form>


 <request>
   <http url="/testtsunami.html" method="GET" version="1.0"></http>
   <dyn_variable name="random_num" ></dyn_variable>
 </request>

Now random_num will be set to 42 during the user's session. It's value will be replace in all mark-up of the form %%_random_num%% if and only if the request tag has the attribute subst="true", like:
 
    <request subst="true">
      <http url='/go.cgi' version='1.0' 
      contents='username=nic&amp;random_num=%%_random_num%%&amp;op=login' 
      content_type='application/x-www-form-urlencoded' method='POST'>
      </http>
    </request>
If the dynamic value is not a form variable, you can set a regexp by hand, for example to get the title of a HTML page: 
    <request>
      <http url="/testtsunami.html" method="GET" version="1.0"></http>
      <dyn_variable name="mytitlevar" 
                    regexp="&lt;title&gt;\(.*\)&lt;/title&gt;"/>
    </request>

6.9  Checking the server's response

With the attribute match in a request tag, you can check the server's response against a given string. If it matches, this will increment the match counter, if it does not match, the nomatch counter will be incremented.

For example, let's say you want to test a login page. If the login is ok, the server will respond with Welcome ! in the HTML body, otherwise not. To check that: 
 <request match="Welcome !">
      <http url='/login.php' version='1.0' method='POST' 
       contents='username=nic&amp;user_password=sesame'
       content_type='application/x-www-form-urlencoded' >
 </request>

7  Statistics and reports

Available stats: HTTP specific stats:

7.1  Generating the report

cd to the log directory of your test (say ~/.idx-tsunami/log/20040325-16:33/) and use the script analyse_msg.pl:
 
/usr/lib/idx-tsunami/bin/analyse_msg.pl --stats idx-tsunami.log  --html --extra  --plot

7.2  Tsunami summary

images/tsunami-report.png

Figure 1: Report

7.3  Graphical overview

images/tsunami-graph.png

Figure 2: Graphical output

8  References

9  Acknowledgments

The first version of this document was based on a talk given by Mickael Rémond4 during an Object Web benchmarking workshop in April 2004 (more info at http://jmob.objectweb.org/).

A  Frequently Asked Questions

A.1  IDX-tsunami crash when I start it

Does your Erlang system has ssl support enabled ?

to test it: 
  > erl
  Eshell V5.2  (abort with ^G)
  1> ssl:start()
  you should see 'ok'

A.2  IDX-tsunami still doesn't start ...

Most of the time, when a crash happened at startup without any traffic generated, the problem arise because the main Erlang controller node cannot create a "slave" Erlang virtual machine. The message looks like:
 
===============================================
=ERROR REPORT==== 4-May-2004::22:38:26 ===
** Generic server ts_config_server terminating
** Last message in was {'$gen_cast',{newbeam,myshortname,[]}}
** When Server state == {state,{config,
                                     undefined,
                                     5,
                                     full,
                                     undefined,
                                     [{client,
                                          "myshortname",
                                          2.00000,
                                          5,
                                          [{10,68,133,140}]}],
                                     {server,"foo.net",80,gen_tcp},
                                     [],
                                     [{arrivalphase,
                                          1,
                                          60,
                                          undefined,
                                          undefined,
                                          5.00000e-5,
                                          infinity}],
                                     undefined,
                                     [{session,
                                          1,
                                          100,
                                          ts_http,
                                          parse,
                                          true,
                                          undefined}],
                                     14,
                                     3,
                                     7,
                                     6,
                                     "negociate"},
  "/home/username/.idx-tsunami/log/20040204-18:32",
                               undefined,
                               0,
                               undefined,
                               2.00000}
** Reason for termination ==
** {{badmatch,{error,timeout}},
    [{ts_config_server,handle_cast,2},
     {gen_server,handle_msg,6},
     {proc_lib,init_p,5}]}

IDX-Tsunami launches a new erl virtual machine to do the actual injection even when you have only one machine in the injection cluster. This is because it need to by-pass some limit with the number of open socket from a single process (1024 most of the time). The idea is to have several system processes (Erl beam) that can handle only a small part of the network connection from the given computer. When the maxclient limit (simultaneous) is reach, a new Erlang beam is launched and the newest connection can be handle by the new beam).

The problem is that the Erlang slave module cannot start a local slave node. It tries to start it with the short node name "myshortname" (erl -sname myshortname). If this fails the injection process cannot start. Most of the time, adding the short name with the correct IP address in the /etc/hosts file is sufficient to make it work.

Note that you do not need to use the 127.0.0.1 address in the config file. It will not work if you use it as the injection interface. The shortname of your client machine should not refer to this address.

New in 1.1.0: If you don't use the distributed feature of Tsunami and have trouble to start a remote beam on a local machine, you can set the 'use_controller_vm' attribute to true, for ex.:
 
  <client host="mymachine" use_controller_vm="true">

A.3  IDX-tsunami still crash/fails when I start it !

First look at the log file ~/.idx-tsunami/log/XXX/tsunami_controller@yourhostname' to see if there is a problem.

If you see nothing wrong, you can compile idx-tsunami with full debugging: recompile with make debug , and don't forget to set the loglevel to "debug" in the XML file.

To start the debugger or see what happen, start IDX-Tsunami with the debug argument instead of start. You will have an erlang shell on the tsunami_controller node. Use toolbar:start(). to launch the graphical tools provided by Erlang.

A.4  What is the format of the stats file idx-tsunami.log ?


 
# stats: dump at 1083694995
stats: users 11 11
stats: request 41 1.03289 0.125108 1.59802 0.901978
stats: connect 41 0.220170 6.67110e-2 0.494019 0.171997
stats: users_count 11 11
stats: page 24 6.80416 17.2794 80.4609 0.958984
stats: size 26818 26818
stats: 404 7 7
stats: 200 20 20
# stats: dump at 1083695005
stats: users 21 21
stats: request 113 1.03980 0.110650 1.59802 0.791016
stats: connect 118 0.197619 4.26037e-2 0.494019 0.163940
stats: users_count 10 21
stats: page 52 2.72266 1.74204 80.4609 0.791016
stats: size 78060 104878
stats: 404 15 22
stats: 200 51 71
 ...

the format is, for request, page, session:

# stats:'name' count(during the last 10sec), mean, stdvar, max, min

or for HTTP returns code, size ...

# stats:'name' count(during the last 10sec), totalcount(since the beginning)

A.5  How can i specify the number of concurrent users ?

You can't. But it's on purpose: the load generated by IDX-Tsunami is dependent on the arrival time between new clients. Indeed, once a client has finished his session in idx-tsunami, it stops. So the number of concurrent users is a function of the arrival rate and the mean session duration.

For example, if your web site has 1000 visits/hour, the arrival rate is 1000/3600 = 0.2778 visits/second. If you want to simulate the same load, set the inter-arrival time is to 1/0.27778 = 3.6 sec (<users interarrival="3.6" unit="second"> in the arrivalphase node in the XML config file).

A.6  SNMP monitoring doesn't work ?!

SNMP currently doesn't work with erlang R10B and up. Use erlang R9C-2 if you need SNMP.

There is a small bug in the snmp\_mgr module in old Erlang release (R9C-0). You have to apply this patch to make it work. This is fixed in erlang R9C-1 and up.
 
--- lib/snmp-3.4/src/snmp_mgr.erl.orig  2004-03-22 15:21:59.000000000 +0100
+++ lib/snmp-3.4/src/snmp_mgr.erl       2004-03-22 15:23:46.000000000 +0100
@@ -296,6 +296,10 @@
     end;
 is_options_ok([{recbuf,Sz}|Opts]) when 0 < Sz, Sz =< 65535 ->
     is_options_ok(Opts);
+is_options_ok([{receive_type, msg}|Opts]) ->
+    is_options_ok(Opts);
+is_options_ok([{receive_type, pdu}|Opts]) ->
+    is_options_ok(Opts);
 is_options_ok([InvOpt|_]) ->
     {error,{invalid_option,InvOpt}};
 is_options_ok([]) -> true.

1
http://www.erlang-projects.org/Members/mremond/events/dossier_de_presentat/block_10766817551485/file
2
http://www.editions-eyrolles.com/php.accueil/Ouvrages/ouvrage.php3?ouv_ean13=9782212110791
3
http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf
4
mickael.remond@erlang-fr.org

Copyright © 2004 IDEALX S.A.S.. « Linux » is the property of Linus Torvalds. All other trademarks belong to their respective owners.

This document was translated from LATEX by HEVEA.