Oracle's new ADR with command interface shows promise for DBAs to quickly investigate diagnostic information.
In Part I of this series, we took a very high-level look at Oracle Database 11g release of ADR. We visited some of ADR's infrastructure, parameters, and even how to extract some diagnostic data with a very simplistic ADRCI command. In this article, I'd like to take a slight step back and visit some key definitions to ADR. By doing this I think we can all gain a level-set on terms and proceed more effectively in the next part of this series.
What is the Automatic Diagnostic Repository (ADR)
ADR is a simple file-based (directory structured) repository that contains diagnostic data generated from Oracle. It is a collection of items such as trace files, dumps, alert logs, etc. that can be used by administrators to investigate problems and the health of the database. ADR is the new unified directory structure that will hold all diagnostic data from all Oracle products and componentssimplifying the search and correlation of information across all aspects of a single or multiple database instance.
What is ADR Command Interpreter (ADRCI)
ADRCI is a command line utility that serves as the interface between you the administrator and the automatic diagnostic repository (ADR). You can do such things as view diagnostic data, view reports, view alert logs, and even package the diagnostic information for sending to Oracle support. While we only looked at a very simplistic set of commands in part 1, the ADRCI is actually a very robust interface with a rich command set. Much like SQL*Plus, you can create and execute scripts to handle a variety of daily DBA type tasks.
What is a Problem
Within ADR, a problem is nothing more than a critical error that has occurred within an Oracle product or component such as the database. These are the typical ORA- errors you would normally search through the alert log for. An important aspect of ADR is that each problem is individually tracked and assigned a unique problem ID.
What is an Incident
Since a problem could happen many times, an incident is a single occurrence of the problem. Individual incidents are also tracked within ADR and are given a unique numeric incident ID within ADR. Behind the scene, ADR will put together all the diagnostic data about the particular incident and store this packaged information in an ADR subdirectory created uniquely for that incident. The process flow for when an incident occurs is something like this:
1. Entry is made in the alert log
2. Incident is sent to Oracle Enterprise Manager
3. Diagnostic data is gathered about the incident
4. Incident ID is assigned
5. Incident and gathered data is stored in ADR subdirectory
What is Incident Flood Control
Often times, as you can guess, a problem has the potential to generate hundreds of incidents if left un-checked. The generation of diagnostic data could be enormous and possibly congesting or creating an issue (space, performance, etc.) in ADR. Because of this, ADR has a built-in mechanism, flood control, which is applied to excessive incident generation after thresholds have been met. After the thresholds have been met, ADR will only generate an alert log entry and record the event in ADRbypassing the packaging of diagnostic data in another subdirectory of the ADR repository.
Current threshold levels to trigger flood control include and cannot be modified:
1. When 5 incidents occur for the same problem key in one hour
2. When 25 incidents have occurred for the same problem key in a single day
3. When 50 incidents for the same problem key occur in one hour
4. When 250 incidents for the same problem key occur in one day
What is a Problem Key
Every problem is assigned a problem key. The problem key is a text string that includes the error code and possibly some error parameters. Problem keys are not unique across problems and any two incidents that are considered to have the same root cause will have the same problem key.
What is an Incident Package
We have already sort of talked about how ADR will gather associated diagnostic information for an incident. An incident package makes use of the Incident Packaging Service (IPS) to collect diagnostic data, plus any additional information you might want to add such as external files, before sending the incident to Oracle Support. How much do you want to bet Oracle Support was frustrated and bombarded with un-related information? This packaging is probably a good thing for them and you as it will keep all related incident information together and easy to transport.
What is Finalizing
An incident package is really just a logical collection of diagnostic and external information. An incident package can continually be added to, modified, and even have pieces removed before it is actually saved to a physical format (zip). The finalizing processes signals the completion of the modification of an incident package and can now be physically created and saved into a zip file for transport.
Just like the Oracle database, with ORACLE_BASE and ORACLE_HOME, ADR has an ADR Base directory that is the base directory for all ADR Home directories. It is the root directory that all diagnostic information is branched off from. It is set by the DIAGNOSTIC_DEST initialization parameter.
Much like ORACLE_HOME, ADR Home is the location on disk where ADR maintains a directory for all diagnostic data related to a single instance or Oracle product. So if you had two Oracle instances on the same machine you would see two different directories under the ADR Base directory. This is quite handy because some ADRCI commands will work across ADR homes at the same time and allow you to research problems for all instances that are on a single machine.
Because ADRCI commands can operate across all ADR homes under a single ADR Base, Homepath can be used to point at specific ARD Homes within the ADR Base. A couple of interesting behaviors for Homepath include:
1. When Homepath points to a single ADR home, only that ADR home is considered current
2. When Homepath points to a directory above an ADR home, all ADR homes under the directory are considered current
3. When Homepath is NULL, all ADR homes under ADR base are considered current
This article should have gotten you familiar with the definitions surrounding the use of ADR and ADRCI. You should have a good understanding of key components and be ready to dive into the wealth of diagnostic information contained within the repository.