Synopsis. Oracle Database 11g Release 2 makes it much simpler to configure and incorporate many of the grid computing features that were only available in a Real Application Clusters (RAC) clustered database environment in previous releases for a single-instance Oracle database. This article the first in this series - will demonstrate how to install and configure a new Oracle 11g Release 2 (11gR2) Grid Infrastructure home as the basis for the majority of these grid computing features.
Its been a few months since I summarized the incredible array of new features that Oracle has introduced as part of Oracle Database Release 11gR2, and in that span of time, Ive been experimenting with those features as Ive built a new infrastructure for experimentation. Among the most intriguing new features is the consolidation of Automatic Storage Management (ASM) with Oracle Clusterware (OC) into a pragmatic and sensible arrangement called the Oracle Grid Infrastructure (GI). As Ill demonstrate in this article, the venerable Oracle Universal Installer (OUI) utility gets a welcome update in this release, but first Ill need to perform quite a bit of system administration work before we can invoke it and explore its new features.
First ... A Word About The (Computing) Environment. Ive made some long-desired changes to my home offices personal computing infrastructure so that I can manage my workload effectively and efficiently with my favorite virtualization environment, VMWare:
- Ive upgraded to Oracle Enterprise Linux (OEL) 5 Update 2 (kernel 2.6.18-92.el5) for my base computing platform, a home-grown gaming server with 4GB of memory running an AMD Opteron dual-core processor.
- Ive also finally moved up to VMWare Workstation Version 7.0.0 for all my VMWare endeavors, and though I still occasionally long for the freedom of VMWare Server 2.0a (as in free!), Ive found that Workstation is just as stable and that it works extremely well with OEL as both its host and guest OS.
Setting Up For Oracle 11gR2 Grid Infrastructure
Im going to implement my 11gR2 Grid Infrastructure via a series of Oracle best practices that Ive encountered over the years and have gleaned through a thorough reading of Oracles technical documentation. Ill be using raw disk partitions for configuring all of the ASM disks that will eventually comprise the various ASM disk groups needed for my demonstrations.
Creating the Required Raw Partitions. The Oracle 11gR2 Grid Infrastructure leverages ASM to store multiple copies of the Oracle Clusterware Registry (OCR) file, multiple Voting Disks, and of course the ASM disk groups disks themselves. Since the maximum number of logical partitions that can be created within any one extended partition is 12, Ive created two VMWare virtual disks sized at 18.5 GB and 11.0 GB, respectively. Heres the output from the terminal session during which I used the Linux fdisk command to create the remaining logical partitions:
[root@11gR2Base ~]# fdisk -l /dev/sde Disk /dev/sde: 19.3 GB, 19327352832 bytes 255 heads, 63 sectors/track, 2349 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sde1 1 2349 18868311 5 Extended /dev/sde5 1 281 2257069+ 83 Linux /dev/sde6 282 562 2257101 83 Linux /dev/sde7 563 843 2257101 83 Linux /dev/sde8 844 1124 2257101 83 Linux /dev/sde9 1125 1405 2257101 83 Linux /dev/sde10 1406 1686 2257101 83 Linux /dev/sde11 1687 1967 2257101 83 Linux /dev/sde12 1968 2248 2257101 83 Linux [root@11gR2Base ~]# fdisk -l /dev/sdf Disk /dev/sdf: 12.0 GB, 12079595520 bytes 255 heads, 63 sectors/track, 1468 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdf1 1 1468 11791678+ 5 Extended /dev/sdf5 1 281 2257069+ 83 Linux /dev/sdf6 282 562 2257101 83 Linux /dev/sdf7 563 843 2257101 83 Linux /dev/sdf8 844 1124 2257101 83 Linux /dev/sdf9 1125 1405 2257101 83 Linux
Assigning Raw Partitions to Block Device Endpoints. Oracle has recommended for some time that block devices are a much better choice for the ASM file system, especially since Ive occasionally heard rumors that support for traditional raw devices allocated through the /etc/sysconfig/rawdevices configuration file may be reduced or disappear in the future.
For this and all future Oracle 11gR2 features demonstrations, Ive configured a special service, losetup, that will construct, configure and allocate virtual block devices during server startup. For the losetup script to work properly, however, note that I also needed to increase the default number of loopback devices from eight to 16; I did this by adding the following line to the /etc/modprobe.conf system configuration file, and then rebooting the server to make sure it took effect:
options loop max_loop=16
Listing 1.1 shows the losetup script I used to complete the assignment of raw partitions to virtual block devices. After I copied the script to file /etc/init.d/losetup, I then registered the new service (as the root user) via chkconfig:
#> chmod 775 /etc/init.d/losetup #> chkconfig losetup --add #> chkconfig losetup on #> chkconfig losetup --list
After rebooting the server, heres the result of implementing the losetup script the successful allocation of block devices as shown below:
[root@11gR2Base ~]# ls -la /dev/xv* lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdb -> /dev/loop1 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdc -> /dev/loop2 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdd -> /dev/loop3 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvde -> /dev/loop4 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdf -> /dev/loop5 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdg -> /dev/loop6 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdh -> /dev/loop7 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdi -> /dev/loop8 lrwxrwxrwx 1 root root 10 Feb 7 20:49 /dev/xvdj -> /dev/loop9 lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdk -> /dev/loop10 lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdl -> /dev/loop11 lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdm -> /dev/loop12 lrwxrwxrwx 1 root root 11 Feb 7 20:49 /dev/xvdn -> /dev/loop13
Configuring and Implementing ASMLIB
To keep my ASM configuration simple to manage, Ill also use the Oracle ASM disk management drivers that ASMLIB provides to stamp each target mount point before actually creating ASM disks and disk groups. First, Ill confirm that the oracleasm drivers appropriate to my OS kernel version have indeed been installed:
[root@11gR2Base ~]# rpm -qa | grep oracleasm oracleasm-2.6.18-92.el5xen-2.0.4-1.el5 oracleasm-2.6.18-92.el5-2.0.4-1.el5 oracleasm-2.6.18-92.el5debug-2.0.4-1.el5 oracleasm-support-2.0.4-1.el5
Excellent! My system administrator took care of this when she installed Oracle Enterprise Linux 5 Update2; otherwise, Id be forced to remind her to download the appropriate ORACLEASM drivers and then install them on my server. However, it was necessary to make sure that the connection to the appropriate oracleasm RPMs was available, and that took a little extra manipulation as shown in the output below:
[root@11gR2Base ~]# /usr/lib/oracleasm/oracleasm_debug_link 2.6.18-92.el5 $(uname -r) oracleasm_debug_link: Target exists [root@11gR2Base ~]# ls -l /lib/modules/$(uname -r)/kernel/drivers/addon/oracleasm total 576 -rw-r--r-- 1 root root 579514 May 23 2008 oracleasm.ko [root@11gR2Base ~]# /etc/init.d/oracleasm configure -i Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets (''). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface : oracle Default group to own the driver interface : dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Loading module "oracleasm": [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ] [root@11gR2Base ~]# /etc/init.d/oracleasm status Checking if ASM is loaded: [ OK ] Checking if /dev/oracleasm is mounted: [ OK ]
Stamping Candidate Disks With ASMLIB. Now that ASMLIB is configured properly, its time to apply ASMLIB stamps to each virtual device via the createdisk command as shown below. This makes it much simpler to configure and manage ASM disks without having to use complex mount point naming conventions:
[root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK1 /dev/xvdb [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK2 /dev/xvdc [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK3 /dev/xvdd [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK4 /dev/xvde [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK5 /dev/xvdf [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK6 /dev/xvdg [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK7 /dev/xvdh [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ASMDISK8 /dev/xvdi [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK1 /dev/xvdj [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK2 /dev/xvdk [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK3 /dev/xvdl [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK4 /dev/xvdm [root@11gR2Base ~]# /etc/init.d/oracleasm createdisk ACFDISK5 /dev/xvdn
Finally, Ill invoke ASMLIBs listdisks command to confirm that all disks have been correctly stamped and are now ready for use in concert with my upcoming Grid Infrastructure installation:
[root@11gR2Base ~]# /etc/init.d/oracleasm listdisks ACFDISK1 ACFDISK2 ACFDISK3 ACFDISK4 ACFDISK5 ASMDISK1 ASMDISK2 ASMDISK3 ASMDISK4 ASMDISK6 ASMDISK7 ASMDISK8