Monitoring Oracle 10g RAC with Quest Spotlight on RAC - Part I

Tuesday Jan 24th 2006 by Tarry Singh

Tarry Singh offers a peek into Quest Spotlight on RAC, now out of beta and soon to go GA. SoRAC provides detailed diagnostic information by drilling down individual instances and merging the appropriate calculations into one interface.

You have heard the hype around a particular technology. It's cool and everyone wants a piece of it. Then it's the post-hype era for a technology where it is getting serious attention from the Big Players and eventually its commoditization. Breakthrough technologies, like Oracle RAC or Server Partitioning (fondly known as Virtualization), will eventually be commonplace. Therefore, it is time for your DBA to monitor several oracle RACs. The scenario could range from a RAC farm, as massive as 36-Node Oracle RAC, to a 2-Node Oracle RAC. Being confronted with questions like these is very normal to a curious DBA:

  • How is my cluster performing?
  • How is my Database performance as a whole?
  • Contention issues disk, io, cluster, interconnect?
  • Latency problems within HSI (High Speed Interconnects)? Are there poor settings? Are my NIC's are set properly? Or is it a poorly configured RAC?
  • Inter-node communication issues? Are too many "Hot Blocks" being transferred across nodes due to poor database design?
  • Poor load balancing?

As you can see, the list is endless. You have migrated to a heavy duty Oracle RAC and you see performance drop. Unfortunately, you cannot see exactly what might be going wrong.

Introducing Spotlight on RAC (SoRAC)

It is time to say hello to SoRAC (Spotlight on RAC by Quest Software) which is well out of beta phase and very soon to be GA. I had the pleasure of participating in the beta program and was rather pleased with the results (even though it was still in beta phase).

SoRAC provides detailed diagnostic information by drilling down individual instances and merging the appropriate calculations into one interface called "SoRAC Homepage." And it's all live!

Overview of Spotlight on Oracle RAC

What is this tool capable of?

It provides three classes of functionality to a RAC DBA.…

  • You get to see the physical architecture. As you see below...

    You can clearly see all of the essential components of a cluster: the individual instances, the interconnect and the I/O subsystem are displayed, including the essential data flows between them. The great thing about this tool is that you can paint a picture of your RAC in your head, enhance your understanding of RAC, and localize problems. The above example runs against my 2-node RHEL 4.2 Oracle 10g R2 RAC on VMware GSX Server.

  • Calculation of diagnostics and presenting them back to the interface as a whole. For instance, you might have latch contention on one node on a 6-node RAC and the aggregate of all the latches might just be fine for your RAC to function adequately.
  • Several RAC specific alarms such as:
    1. Uneven Load Distribution to the instances.
    2. Overhead due to cluster maintenance.
    3. High latency problems within the HSI (High Speed Interconnects).
    4. Excessive HSI communication (might point to partitioning data across nodes, for instance a typical hybrid , meaning OLTP (Online Transaction Processing) /DSS (Decision Support System) might require a data partitioning on the DSS part of that RAC database).
    5. Block Corruption / Orphaned blocks across instances.
    6. OCFS problems / Shared Disk (Whether on SAN/NAS) contention.

What Platform does it support?

The initial release will still provide support to a RACs running on Unix/Linux based systems such as RHEL/Solaris. There is theoretical support for a typical 36-node cluster but formally, you monitor a 16-node cluster with ease.

You will, however, need an Oracle Client software installation and you can get it here although you will need to adjust your TNSNAMES.ora file, as this will be used to make a connection to your RAC Service.

Getting Spotlight on RAC

Spotlight on RAC should be available very soon. (So I heard from the Marketing Manager) and then you should be able to download an evaluation from here .

Installing and Setting up

Installation is rather simple.

  • Install your Oracle Client Software.

  • Install SoRAC

  • Setting up the TNSNAMES.ora

Click for larger image

  • Connecting to your RAC

Click for larger image

Upon double clicking your connection, you get to fill in the Mycluster Properties file:

Click for larger image

Above, you will see the connection string drop down for both your Oracle RAC Service and ASM instances. The minimum that you need is the Oracle Connection String. SoRAC connects to the instance and queries view gv$instance for all RAC instances and it queries the view gv$active_instances to display the active instances in the cluster.

The Oracle Username and Password is the same as that required for Spotlight on Oracle – SoRAC and SoO share the same Oracle user. You will be prompted to create a user or use an existing user. Follow the steps below to create a new user.

Step 1: Wizard opens up

Step 2: It connects to an instance to upgrade the user

Step 3: Create a new user

Step 4: We will call the new user spotrac.

Step 5: I pick tablespaces "Users" and "TEMP", you can however create your own tablespace as well.

Step 6: Creating user and configuring...

Step 7: Done.

All other fields are optional. If the cluster uses ASM, then you can fill the Monitor ASM section. ASM is managed via an Oracle instance, which must be entered here. It only supports the user "SYS" as we know. This section must be filled out to populate the ASM spinner, associated drilldown and ASM Service Time alarm.

Connecting/Verifying the Cluster

Step 1: Double clicking the connection we get our instances presented to us...

Click for larger image

Step 2: Click test and then OK

Step 3: Calibration Settings


In the next and the final article of "Monitoring Oracle RAC with Spotlight on RAC" (SoRAC), we will discuss in detail how the Alarm (Oracle Aggregated Alarms, Cluster Balance Alarms, Cluster Latency and Overhead Alarms, Global Cache Alarms, ASM Alarms) and Drilldowns work. Here you have seen that the homepage or the monitoring pane of SoRAC is very intuitive. You as a busy DBA will spend a lot less time on complex problems, as a graphical overview will point you in the right direction.

» See All Articles by Columnist Tarry Singh

Mobile Site | Full Site