Toad World® Forums

Is it possible for Benchmark Factory to run the same exact test to many DBs


#1

I recently began to experienment with this software and find it quite amazing compared to coming up with our own benchmarks. One thing I am unclear of is let’s say I set up a TPC-B benchmark and run it against Oracle10g1, once it’s finished run it against Oracle11g1, the finally against Oracle11g2. All three databases are on the same host. However I do see different results. I would expect at least some consistency between the 11g databases but so far I have not.

Is it fair to assume that Benchmark Factory should be running the same benchmark each time the job is submitted? Or is there some other way to accomplish the task I am trying to do which is to compare two identical databases and expect similiar results. I’ll continue reading the documentation to see if such a scenario is mentioned.

Thanks!
Eric


#2

BMF takes great care to ensure that every time a test is executed that it is the same. For example all random number generators are seeded with the virtual user ID at the start of a test so that each time the test is run the sequence of numbers is the same for a particular virtual user, but each virtual user will have a different sequence. So your assumption is correct. So what you are doing is correct for comparing different version of Oracle. One best practice is to make sure that you are recreating the TPC-B database between test runs since the TPC-B transactions are making changes to the database which can affect performance between runs if not recreated.


#3

Thanks! The key thing is the recreate objects as you mentioned. Once I did that I noticed more consistent results, or at least more consistent than I was getting yesterday.

Thanks again,
E