Toad World® Forums

Runing agents concurrently


#1

Our SQL Server based product has to support thousands of concurrent users. With only a limited number of physical machines available for testing, we have turned to Benchmark Factorys virtual user environment to get an idea of how many concurrent users we can actually handle.

We used SQL Server Profiler to record an actual client session and imported the resulting script into BMF (5.5.1). We set the latency for every transaction to 0 (No Delay) and then added some think time to a few transactions scattered thru the script to closer mimic the timing of our actual client session.

Running the script (in Text File Import Mode) with multiple users on a single agent produces an easily recognized CPU usage pattern on the server. When we run BMF with multiple agent machines, we see this CPU pattern repeating sequentially, once for each agent.

It does not appear as if BMF is running the agents concurrently. Am I missing something or doing something wrong? Any suggestions would be greatly appreciated.


#2

When Benchmark Factory creates a Job from a SQL Server trace file it will create a replay test that contains User Scenarios. These User Scenarios will execute the transactions in order each time they are run. How the job will run when the load is increased will depend upon the changes that you made to the script, and the number of actual agents running the test should not affect how the Job is run.
When you modified your replay test to use more users, did you change the Users number on the Replay tests Transaction tab? If you changed the Executions number as well, the user scenarios will run through the transactions repeatedly as specified by that number. If you did this, it would cause the repeated CPU pattern you are seeing.

Tracey


#3

Hi Tracy,

For every test, I set the number of users in the Transactions tab of the script and then submit as a new job. The script is always the same; the number of Executions is always one. The only change from run to run is the number of users, and sometimes, the number of remote agents that are started before the test.

Bob


#4

Something else you can check are the latency values on the user load tests. If they are something other than No Delay, that may cause the user scenarios to start at different times.
Would it be possible for you to send me the script, so I can take a look?


#5

The script has 1604 transactions. All but a few have latency set to no delay. The exceptions are:
Query # 5, 1276 ms
Query # 59, 29 ms
Query # 72, 29 ms
Query # 94, 47 ms
Query # 95, 30 ms
Query #436, 59 ms
Query # 1285, 48 ms
Query # 1345, 89674 ms
Query # 1353, 2095 ms
Query # 1431, 35 ms
Query # 1526, 214 ms
Query # 1545, 42 ms
Query # 1604, 65 ms
These have think time as shown (all Absolute) to mimic the processing of our app.


#6

Are the latency values of your user scenarios set to Absolute as well?

Does this only happen when the agents are on remote machines? Could it be a network issue? You can try running multiple agents on the same machine to see if it happens then as well.

How long does it take for your job to run? Is it very quick like less than a minute, or does it take several minutes or longer?
Tracey


#7

The User Scenario Latency is set to No Delay. The problem only appears to happen with remote agents, but we are on a fast 1gb network. Jobs take a long time to run (sometimes even on one box). Minutes can tick by with no apparent activity (something I am waiting for a response from Quest about).

Is it your experience that Jobs are reliably fast and agents on separate machines run concurrently?


#8

I have been doing more testing and I am getting very inconsistent results running the same tests on the same hardware, from good, to slow, to locking up. The slow tests appear to run sequentially, while I have seen quicker tests run concurrently. My real problem would appear to be whatever is causing the huge variations in performance.


#9

You may want to add some Real-Time counters to your job or use monitoring software like Spotlight on SQL Server to see what your server is doing when you are encountering these performance issues.

Also, if you are running one User Scenario with multiple users you may be having table locking problems. If you SQL Transactions are performing database update tasks and multiple users are trying to execute them at the same time, then locking may become a problem. If this is the problem I have two ideas you can try. First, you can try adding a non absolute latency to the user scenario so that they start at slightly different times. Secondly, you can randomize your SQL transactions using BFScripts so that the virtual users are not executing the exact same SQL.


#10

Im experiencing seemingly random delays and lockups before and after the db sampling, so I dont think my problems are in SQL Server, but there are two updates to the same row, column that I did not realize were there (thanks). Ill remove these and retest.