Toad World® Forums

Cycling observed in Quest Benchmark factory output.


#1

I am running Quest Benchmark factory TPC-C test against Oracle 11g R2 RAC database. The test runs fine and I observe almost perfect scaling till higher user load is reached.
After that, one iteration produces very good TPS and the next iteration output drops significantly. What can I make out from this output ? I don’t think it’s a storage or database issue as throughput on the alternate iterations is very good. Is this some kind of a known issue ?

Can any of you please advise me on this ?

-Shashi


#2

Without having some additional statistics it is hard to know what caused the drop, but I have seen large drops in TPS in other test runs including those for RAC systems. I have not seen rises and drops though like you are mentioning. My advice would be to capture some monitoring statistics to see what is happening. There are no known issues with BMF that can cause the type or results you are seeing.


#3

When I check AWR report, I see “enq: row lock contention” as the top event. I can’t think it’s storage issue because, the performance is good on one iteration and drops on the next iteration.

If it was a continuous drop, maybe I could have thought that was something related to storage. If I can get performance on one iteration, why is that such a drop during my next iteration.

I have also attached a sample chart below which shows this exact behavior.

[cid:image001.png@01CB2751.51A5DD90]

As you can see, not sure what happens once we reach higher iteration. The graph is linear till 6000 and then observed huge cycling as shown above…

-Shashi


#4

I was looking at some other runs and I found a run with RAC that has somewhat of the peak and valleys that you are seeing, but not to the degree you get with every other TPS is still linear. I think it is interesting that the only run I could fine with this is also on a RAC system. How is your interconnect between nodes and how many nodes?
RAC_10Node_TPS.jpeg


#5

Thanks for the response.

I always observe this kind of cycling issues with Quest BMF on RAC config. At higher userload, I observe this. Mine is a 2-Node RAC and I have three 1Gig interfaces trunked together for RAC interconnect.

It’s very frustrating as Oracle purely puts blame on Quest and the AWR suggests “enq: row lock contention” which is application related.

If I contact Quest, they say it’s all related to Oracle database :-). Till now, I am clueless and it will be very helpful if you can advise me on this.

-Shashi


#6

I’ve got same issue as you. my configuration is 11gR2 RAC, 4 node.

no idea on how to resolve it but it’s something Toad needs to deal with as this wait event is much more relevant to the code they developed to do the TPCC testing.

please let me know if they helped you to fix the problem.


#7

I have not received any response from Quest till now. This is really a frustrating issue. Oracle support will not accept this as the AWR report clearly indicates that it is something related to application.

Please let me know if you get any resolution for this issue ?

-Shashi


#8

Hi Shashi,

I’m thinking of installing the new version of Benchmark Factory 6.1.1 to see if the new version would be better. if it’s not, I will send the Quest Support the support bundle which could be created from the new version.

since my test is still in progress, if you can, can you install that and try another run? hopefully they fixed this issue in the new version.

Thanks,
Betty


#9

I have successfully done 10 to 20 node RAC clusters with 11g and have not had any problems or issues when using BMF. BUT - and this is a huge BUT - you must not just create a default RAC database on a default OS setup with default network configuration issues (such as frame sizes). Basically BMF is doing exactly what you want - it’s showing you that the defaults are typically all wrong as setup for you out of the box. The only two things you need to define in BMF are the #users and thus set the scale size to 1’th that size. So 10,000 users requires setting scale size to 1,000 - or results will be skewed by locks and deadlocks (if it even runs to completion). I would need a heck of a lot more info than this discussion to assist further. If you want to email me offline - it’s bert.scalzo@quest.com. Just understand that if most of the questions are about proper database and OS setup for doiing RAC - I’m going to give you as mnuch help as I can - but free consulting stops off many times quicker than what some people need. So you may still get incomplete answers for your needs - because I am time constrained and my job allows for little pro-bono Oracle help. Sorry - just warning you is all so that your expectations are properly set - in that I’ll try my best within reason :slight_smile: