The first and critical step is to read and know the TPC-H spec. All answers are always in the spec.
Section 5.3 execution rules, specifically sub0section 188.8.131.52 says “To measure the performance of a system using the TPC Benchmark™ H, the test sponsor will execute runs composed of:
- A power test, to measure the raw query execution power of the system when connected with a single active user. In this test, a single pair of refresh functions are executed exclusively by a separate refresh stream and scheduled before and after the execution of the queries;
- A throughput test, to measure the ability of the system to process the most queries in the least amount of time. In this test, several pairs of refresh functions are executed exclusively by a separate refresh stream and scheduled as defined by the test sponsor.
Section 5.4.1 defines TPC-H Power as “the inverse of the geometric mean of the timing intervals” and shows two alternative, complex formulas to compute TPC-H Power@Size.
Section 5.4.2 defines TPC-H Throughput Numerical Quantity as “the ratio of the total number of queries executed over the length of the measurement interval” and shows the complex formulas to compute TPC-H Throughput@Size.
Section 5.4.3 defines The TPC-H Composite Query-Per-Hour Performance Metric as follows:
- TPC-H Price-per-QphH@Size = $/QphH@Size
BMF produces the raw numbers to plug into the equations to calculate the values. BMF cannot report the Section 5.4.6’s Required Reporting Components for several reasons – including licensing/royalties reasons and because you have to enter the hardware/software costs into the equation as well. So users have to generate their own scores using the input values from BMF and the spec formulas.
Now once you calculate those final results – BMF does not tell you how to interpret them or what to do next. Read my blog Benchmarking – Reality vs. Perception – where BMF is the “treadmill” in the cardiac stress test example. The treadmill does just one thing – it stresses the system. In order to then monitor, diagnose, prescribe corrections, apply them and retest – you need other tools.