Toad World® Forums

RAC Scalability


#1

We have BMF 5.0 with 4 node RAC.

The schema is created with 12000 warehouses for a total size of little over 1TB.

We are seeing very little or poor scaling when going to 2 Node test.

There is extreme contention on C_WAREHOUSE, C_DISTRICT tables.

and on C_ORDER_I1 and C_ORDER_LINE_I1 indexes.

The two tables were fixed by setting PCTFREE=98.

With the indexes, we tried 4 hash partitions with negative effect.

Please let us know if BMF is sending different rows from different warehouses

to different nodes in the Cluster?

Currently, that seems to be the issue. AWR and ADDM reports arrive at the same conclusion.

Any help will be deeply appreciated.

Thanks,

Prabhakaran.


#2

Each virtual user is assigned a warehouse id based on the maximum userload that the test will go to as well as the TPC-C scale factor for the test. That warehouse ID that is used by the virtual user for many of the TPC-C queries, but there are a few of the queries that, per spec, goes to remote warehouse ID (warehouse ID not assigned to the virtual user) such as the New Order transaction.

There is nothing inside implementation that says maps the specific warehouse ID with specific nodes. If you are using BMF load balancing in the Oracle profile, the virtual users are assigned a node to connect to via a TNSNAMES entry via the specified percentage. Are you using BMF load balancing?

Here is a link to a RAC scalability test that we did with Dell that may help.

http://www.quest.com/success_stories/Dell-Quest.pdf


#3

Yes, we are using BMF Load Balancing.
Based on the AWR/ADDM reports, it looks like both nodes
are likely going against the same row or blocks.

This can be seen by the extreme contention
for gc buffer busy. buffer busy waits on the
objects listed previously.

Thanks, Prabhakaran.


#4

You may want to verify that the scale factor set in the menu EDIT - SETTINGS and then the benchmarks tab is still set to 12000. The scale factor used for the creates are stored in the benchmark creation object and can be different than the default.


#5

prabhakaran,
One thing you can test is to uncheck BMF Load Balancing and let Oracle handle the load balancing.

Also, can you explain how are your storage LUNS are set up?
Are you using ASM or RAW devices?


#6

Hi ,

We tried 2 runs with the Edit->Settings and a warehouse number of 12000.
There was no improvement.

We are using BMF load balancing. Even with Oracle load balancing,the queries
are likely to be sent round-robbin to each node.

We are using ASM . There are 50 disks of 63 GB each. The schema that we are running the BMF on is 12000 warehouses or approx 1TB ( including indexes ).

Here is the section of the latest AWR report,
Global Enqueue Statistics DB/Inst: TCEDB/tcedb3 Snaps: 952-953

Statistic Total per Second per Trans


gcs blocked converts 7,320 7.9 0.2
gcs blocked cr converts 12,607 13.6 0.3

Blocked converts is the biggest indication that both nodes are very very likely going
after the same rows. Causing a huge degradation in scalability by BMF.

Thanks, Prabhakaran.

Added the details of the ges statistics.

Message was edited by: prabhakaran