Toad World® Forums

TPC-E Users


#1

I am a bit confused about how many users to select for the TCP-E Benchmark. I read through the TPC-E spec, and could not find a specification for the number of users to use to run the benchmark. Wouldn’t it bean “apples to oranges” comparison if one company pulished thier TPC-E TPS results with say - 10 users, and another company published thier TPC-E TPS results with 100 users. I mean should’nt TPC-E specify a standard workload in order to get an apples to apples comparison. How can the results of a TPC-E be auditable if the number of users utilizing the database is variable?

Also - what it the “Benchmark Scale Factor”.

Thanks much for any help


#2

BMF users doing standard benchmarks should read (not scan) the spec for that benchmark. BMF automates “portions” of the spec - it’s up to the user to handle the rest. For example BMF does not partition tables - because BMF cannot read your hardware config (LUNS vs. spindles) to choose the perfect design. That requires human expertise.

As for the # of users and database size - that’s entirely up to you. User #1 with a modest server would choose much more conservative or smaller settings than user #2 with a killer server.

Here are some comments directly from the spec:

The number of customers defined for the brokerage firm can be varied to represent the workloads of
different size businesses.

The transaction load generated to service customer accounts and to interact with financial markets
drives the throughput of the TPC-E benchmark. To increase the throughput, more customers and their
associated data must be configured. The cardinality of the CUSTOMER table is the basis of the TPC-E
database size and scaling. CUSTOMER table cardinality is determined based on the transaction
throughput metric requirements defined in Clause 6.6.7.

This last note comes from section 2.6 - which directly addresses your questions. You should at least read and know this part so you can set BMF properly. For example one sentence says:

The Scale Factor is the number of customer rows per single Transaction-Per-Second-E (tpsE). The Scale
Factor for Nominal Throughput is 500.

BMF simply tries to offer you choices within the spec’s context - but it does not explain them.

I’ve written numerous blogs on www.toadworld.com about benchmarking - and the #1 mistake most people make is in the preparation. BMF is not a single push button tool - because there is no such tool. As the following blog points out - BMF does about 1/10th of the real work - you do the rest …

http://www.toadworld.com/BLOGS/tabid/67/EntryID/373/Default.aspx

here are some others to read …

http://www.toadworld.com/BLOGS/tabid/67/EntryID/98/Default.aspx

http://www.toadworld.com/BLOGS/tabid/67/EntryID/139/Default.aspx

http://www.toadworld.com/BLOGS/tabid/67/EntryID/370/Default.aspx

http://www.toadworld.com/BLOGS/tabid/67/EntryID/225/Default.aspx

http://www.toadworld.com/BLOGS/tabid/67/EntryID/378/Default.aspx