Toad World® Forums

Compare Results With Other Systems?

So I have been playing with Benchmark Factory and have run some tests but am not really sure where to find something published to compare my results with. Things like Response Time, etc. Somewhere there has to be some results. I have spent a good amount of time searching the web but have come up empty handed. What is a good response time?

I understand that there are many variables in determining that and what one may consider ‘good’ another may not. Please don’t simply reply with ‘it depends’. If you could guide me in the right direction, perhaps a site that lists something, that would be helpful. I find it hard to believe that there is a tool that gives one results but there is no way of knowing what constitutes acceptable and not good.

Identifying an acceptable response time is dependent on your application, users expectations and any service level agreements you may have in place. Because of that, there is no single number that specifies what an acceptable response time is. Case in point, if you have a web site that is processing orders in real time, your response time needs to be a lot faster than if you have a decision support system where you may be processing millions of rows to answer a single question. You might try narrowing your focus to applications that are similar to yours to come up with valid comparisons. Most of the response times I have seen published relate to web based applications.

One thing you need to consider also is that the response time of your database is only one piece of your total response time which in turn is reflective of your user experience.

If you are using the TPC benchmarks you can compare your results to the results published on their website. For instance, TPC-C benchmarks are published and they use transactions per minute (tpmC) as their reporting metric. Just take Benchmark Factory’s transactions per second number and multiply by 60.

So the answer is no, this tool gives one numbers but those numbers are utterly useless because one is not able to easily find out what those numbers mean.

I suppose I expected that somewhere there would be something published like:

4 CPU/32GB RAM with xbrand SAN with 12 15k spindles at RAID10 and I get x throughput.

or anything…anything.

Jeez, there has to be something that tells me whether or not these numbers coming out of BMF mean, otherwise the numbers and the product are not worth much.

There is something like that - if you look on the TPC’s website here: www.tpc.org/…/tpcc_results.asp you will see a list of hardware configurations and their test results. As for storage, it’s a bit trickier since there any thousands of server - storage pairings and publishing a set of benchmarks for each combination would be a massive undertaking and pretty expensive. I would look for a similar server on the TPC website and then see if someone has run a benchmark on your storage system using a TPC benchmark.

There’s another aspect to this –what I like to refer to as the treadmill. Read this blog:

http://www.toadworld.com/products/benchmark-factory/b/weblog/archive/2013/04/01/benchmarking-reality-vs-perception.aspx

The doctor goes to medical school to learn what’s good vs. bad and how to apply that for different people, age groups, weights, etc. The doctor does not expect
the lowly treadmill to do that – nor does he even expect or trust some diagnostic monitor – he applies his skill, wisdom and insights.

BMF is just the brain dead treadmill. The doctor needs a performance tool like spotlight to begin to divine the medical facts. Right now it sound like you want
a doctor who can only read what something else tells him – and that’s not much of a physician
J

From: jxs2151 [mailto:bounce-jxs2151@toadworld.com]

Sent: Monday, February 17, 2014 3:42 PM

To: bmfstandards@toadworld.com

Subject: RE: [Benchmark Factory for Databases - Standard Benchmarks Discussion Forum] Compare Results With Other Systems?

RE: Compare Results With Other Systems?

Reply by jxs2151

So the answer is no, this tool gives one numbers but those numbers are utterly useless because one is not able to easily find out what those numbers mean.

I suppose I expected that somewhere there would be something published like:

4 CPU/32GB RAM with xbrand SAN with 12 15k spindles at RAID10 and I get x throughput.

or anything…anything.

Jeez, there has to be something that tells me whether or not these numbers coming out of BMF mean, otherwise the numbers and the product are not worth much.

To reply, please reply-all to this email.

Stop receiving emails on this subject.

Or
Unsubscribe from Benchmark Factory - Standard Benchmarks
notifications altogether.

Benchmark Factory for Databases - Standard Benchmarks Discussion Forum

Flag
this post as spam/abuse.

Bert,

Thanks for taking the time to reply. I understand that BMF is not going to tell me everything I need to know about database system performance. However, some guidelines,rough guidelines or even directions to where to find the numbers would be a start.

I am not asking anyone to diagnose my system, merely pointers to something to compare it with. Heck, even Spotlight has some guidance about what is good and what is bad (alarms).

Also, if BMF is simply a treadmill then why have it generate reports?

I’ll spend more time with Spotlight but if I wanted a simple load generator I could have stuck with SQLIO. If the purpose of BMF is to generate a more standardized, repeatable load that one can use for comparison, the comparison numbers would be nice.

The best advice I can give is to reiterate what Kevin said - go to tpc.org, find the published test results for a similar system, and then compare your findings to those. Just remember to read appendix A in their results where they must divulge any DDL changes they make - which most published results have. The reason BMF goves some light weight reports is the same as a treadmill that will say distance ran, calories burned, etc. We have to provide some info. If it we up to me I’d remove all those reports and report nothing - but then most people would complain that it tells them nothing. So not sure what right answer is. I hear what you’re asking and I do get it - but BMF aint going to fill in those gaps. Sorry.