Oracle Database 11gR2: Building an ASM Clustered File System (ACFS)

Thursday Mar 25th 2010 by Jim Czuprynski
Share:

Oracle Database 11g Release 2 (11gR2) integrates a new ASM-based file system -- the ASM Clustered File System -- that offers the ability to store files other than database specific files like online redo logs, control files, and datafiles. This article demonstrates how to install and configure a new Oracle 11g Release 2 (11gR2) Grid Infrastructure home as the basis for the majority of these grid computing features.

Synopsis. As its integration of grid computing features that had only been available in a Real Application Clusters (RAC) clustered database environment, Oracle Database 11g Release 2 (11gR2) integrates a new ASM-based file system – the ASM Clustered File System – that offers the ability to store files other than database specific files like online redo logs, control files, and datafiles. This article – the next in this series - will demonstrate how to install and configure a new Oracle 11g Release 2 (11gR2) Grid Infrastructure home as the basis for the majority of these grid computing features.

Oracle Database 11gR2: Installing Grid Infrastructure explained how to:

  • Configure ASMLIB for block devices
  • Install the new Oracle 11gR2 Grid Infrastructure components in single-server mode
  • Create an ASM instance using the Grid Infrastructure tools

Continuing along this same course, I’ll next illustrate how to:

  • Install and configure the new Oracle 11gR2 ASM Clustered File System (ACFS)
  • Create a new Oracle 11gR2 database home within an ACFS file system
  • Use myriad new features of the ASM command-line utility, ASMCMD, to manage all facets of an ASM instance, ASM disk groups, and ASM disks

Installing ACFS: Preparation

Now that I’ve got an Automatic Storage Management (ASM) instance configured, I’ll turn my attention to setting up a new ASM disk group that uses 11gR2’s new ASM Cluster File System (ACFS) as its underlying file system. First, I’ll verify whether my ASM instance has been successfully restarted after I rebooted my host server:

[oracle@11gR2Base ~]$ ps -ef | grep +ASM
oracle   13568     1  0 11:30 ?        00:00:00 asm_pmon_+ASM
oracle   13570     1  0 11:30 ?        00:00:00 asm_vktm_+ASM
oracle   13574     1  0 11:30 ?        00:00:00 asm_gen0_+ASM
oracle   13576     1  0 11:30 ?        00:00:00 asm_diag_+ASM
oracle   13578     1  0 11:30 ?        00:00:00 asm_psp0_+ASM
oracle   13580     1  0 11:30 ?        00:00:06 asm_dia0_+ASM
oracle   13582     1  0 11:30 ?        00:00:00 asm_mman_+ASM
oracle   13584     1  0 11:30 ?        00:00:00 asm_dbw0_+ASM
oracle   13586     1  0 11:30 ?        00:00:00 asm_lgwr_+ASM
oracle   13588     1  0 11:30 ?        00:00:00 asm_ckpt_+ASM
oracle   13590     1  0 11:30 ?        00:00:00 asm_smon_+ASM
oracle   13592     1  0 11:30 ?        00:00:00 asm_rbal_+ASM
oracle   13594     1  0 11:30 ?        00:00:02 asm_gmon_+ASM
oracle   13596     1  0 11:30 ?        00:00:00 asm_mmon_+ASM
oracle   13598     1  0 11:30 ?        00:00:00 asm_mmnl_+ASM
oracle   14318 14217  0 17:09 pts/2    00:00:00 grep +ASM

[oracle@11gR2Base ~]$ . oraenv
ORACLE_SID = [orcl] ? +ASM
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/grid is /u01/app/oracle

[oracle@11gR2Base ~]$ srvctl status asm
ASM is running on 11gr2base

Excellent news! My ASM instance is running after the server was rebooted, and this means that the setup for Oracle 11gR2 High Availability Services (HAS) is still intact. If you’ve worked with Oracle Clusterware (OC) and Real Application Cluster (RAC) databases in prior Oracle releases, by the way, you’ll notice that I used the venerable srvctl OC command to confirm the status of my ASM instance. This illustrates that Oracle Clusterware is indeed imbedded within the Grid Infrastructure components of Oracle 11gR2. (I’ll be covering a range of new srvctl commands in my next article as we explore how an Oracle single-instance database is managed within the 11gR2 Grid Infrastructure.)

Configuring ACFS For Server Reboots. Before I can proceed with the creation of an ACFS-based Oracle Home, I need to verify that the corresponding ACFS module and its service, oracleacfs, is also running. I’ll connect as the root user and then issue the lsmod command to search for that service:

#> lsmod | grep oracle
oracleasm              81576  1 

Unfortunately, neither oracleacfs nor its counterpart oracleadvm service that Oracle uses to manage ASM Dynamic Volume Management (ADVM), is loaded right now because I haven’t yet implemented the proper acfsload startup script. Much like the losetup script in the prior article, I’ve built a new script, /etc/init.d/acfsload, as shown below, and I’ll activate this script as a service for automatic reboot using the Linux chkconfig command:

#> view /etc/init.d/acfsload

#!/bin/sh
# chkconfig: 2345 30 31
# description: Automatically load Oracle 11gR2 ACFS drivers during system reboot
/u01/app/oracle/product/11.2.0/grid/bin/acfsload start -s

#> chmod 775 acfsload
#> chkconfig --add acfsload
#> chkconfig --list acfsload

acfsload        0:off   1:off   2:on    3:on    4:on    5:on    6:off

Once I constructed and then executed this script manually, I can confirm that the proper ACFS and ADVM modules are now loaded:

#> lsmod | grep oracle
oracleacfs            877320  0 
oracleadvm            221760  0 
oracleoks             276880  2 oracleacfs,oracleadvm
oracleasm              81576  1 

Installing ACFS: Creating a New ASM Disk Group

My next step is to configure ACFS, and to do that I’ll first build a new ASM disk group just to contain this new file system. Instead of using the asmca utility to build this disk group in interactive mode as I did in my prior article, however, this time I’ll invoke it in command line mode. I can view a brief listing of all the things that asmca can perform in this mode by invoking the asmca –help command from the same terminal window, as shown in Listing 2.1.

To build a new disk group named ACFS, I’ll invoke asmca with the –creatediskGroup directive.

$> asmca -silent -createDiskGroup 
-diskString '/dev/oracleasm/disks/*' 
-diskGroupName ACFS 
-diskList /dev/oracleasm/disks/ACFDISK1
          /dev/oracleasm/disks/ACFDISK2,
         /dev/oracleasm/disks/ACFDISK3 
-redundancy EXTERNAL 
-au_size 1 
-sysAsmPassword oracle

And here’s the resulting entries from the +ASM instance’s alert log:

Sun Mar 07 22:46:59 2010
NOTE: updated gpnp profile ASM diskstring: /dev/oracleasm/disks/*
ALTER SYSTEM SET asm_diskstring='/dev/oracleasm/disks/*' SCOPE=BOTH SID='*';
SQL> CREATE DISKGROUP ACFS EXTERNAL REDUNDANCY  DISK '/dev/oracleasm/disks/ACFDISK1',
'/dev/oracleasm/disks/ACFDISK2',
'/dev/oracleasm/disks/ACFDISK3' 
	ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */ 
NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/ACFDISK1)
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/ACFDISK2)
NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/ACFDISK3)
NOTE: initializing header on grp 2 disk ACFS_0000
NOTE: initializing header on grp 2 disk ACFS_0001
NOTE: initializing header on grp 2 disk ACFS_0002
NOTE: initiating PST update: grp = 2
kfdp_update(): 13 
Sun Mar 07 22:47:00 2010
kfdp_updateBg(): 13 
NOTE: group ACFS: initial PST location: disk 0000 (PST copy 0)
NOTE: PST update grp = 2 completed successfully 
NOTE: cache registered group ACFS number=2 incarn=0x17288ec8
NOTE: cache began mount (first) of group ACFS number=2 incarn=0x17288ec8
NOTE: cache opening disk 0 of grp 2: ACFS_0000 path:/dev/oracleasm/disks/ACFDISK1
NOTE: cache opening disk 1 of grp 2: ACFS_0001 path:/dev/oracleasm/disks/ACFDISK2
NOTE: cache opening disk 2 of grp 2: ACFS_0002 path:/dev/oracleasm/disks/ACFDISK3

< . . . some lines removed for sake of brevity . . .>

NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2
SUCCESS: diskgroup ACFS was mounted
SUCCESS: 
	CREATE DISKGROUP ACFS EXTERNAL REDUNDANCY  DISK '/dev/oracleasm/disks/ACFDISK1',
'/dev/oracleasm/disks/ACFDISK2',
'/dev/oracleasm/disks/ACFDISK3' ATTRIBUTE 'compatible.asm'='11.2.0.0.0',
	'au_size'='1M' /* ASMCA */
Sun Mar 07 22:47:09 2010
NOTE: diskgroup resource ora.ACFS.dg is online

I’ll use asmca to confirm that creation of the new ACFS ASM disk group has been successful and to modify any necessary ASM initialization parameters. As Figure 2.1 shows, ASM disk group ACFS was successfully created:

Invoking ASMCA
Figure 2.1. Invoking ASMCA

However, I also need to verify that the proper compatibility settings for the ACFS disk group are in place:

Verifying Compatibility Settings

Figure 2.2. Verifying Compatibility Settings

Excellent! These settings are correct. So now, I’ll create a new mount point within ACFS that will eventually host my new Oracle 11gR2 database home, as shown below:

Creating a New Oracle Database Home Within an ACFS Volume

Figure 2.3. Creating a New Oracle Database Home Within an ACFS Volume

For the location of the new ACFS mount point, I’ll choose the new ACFS ASM disk group as the target for the ACFS volume created above …

Selecting a Destination for New ACFS Volume Figure 2.4. Selecting a Destination for New ACFS Volume

... and the end result is the successful volume assignment as shown in Figure 2.5 below.

Completed ACFS Volume Configuration
Figure 2.5. Completed ACFS Volume Configuration

Managing ACFS: The ACFSUTIL Utility

I can verify the status of any ACFS-controlled mount point with the acfsutil command-line utility. For example, the acfsutil info fs command shows me the current state of the existing ACFS file system, including where and how it’s been mounted, its current availability, how much free disk space is available, whether any software redundancy is in place, and even the stripe size used:

#> acfsutil info fs
/u01/app/oracle/acsfmounts/acfs_db1
    ACFS Version: 11.2.0.1.0.0
    flags:        MountPoint,Available
    mount time:   Sun Mar  7 22:56:43 2010
    volumes:      1
    total size:   6442450944
    total free:   6391668736
    primary volume: /dev/asm/acfs_db1-152
        label:                 
        flags:                 Primary,Available,ADVM
        on-disk version:       39.0
        allocation unit:       4096
        major, minor:          252, 77825
        size:                  6442450944
        free:                  6391668736
        ADVM diskgroup         ACFS
        ADVM resize increment: 268435456
        ADVM redundancy:       unprotected
        ADVM stripe columns:   4
        ADVM stripe width:     131072
    number of snapshots:  0
    snapshot space usage: 0

Listing 2.2 shows a detailed list of acfsutil commands that are currently available. I’m going to dive much deeper into acfsutil in the next article in this series when I demonstrate how to create an ACFS file system snapshot (FSS) for faster patching of Oracle homes with virtually zero downtime.

Mounting an ACFS File System Automatically After Server Reboot. As you can see from my invocation of the Linux ls command below, the new ACFS file system is actually not yet mounted:

#> ls -la /u01/app/oracle/acsfmounts/*
total 8
drwxr-x--- 2 oracle dba 4096 Mar  7 22:54 .
drwxr-x--- 3 oracle dba 4096 Mar  7 22:54 ..

To mount this new ACFS file system at the appropriate mount point, I’ll simply invoke the mount command as the root user, but note that for the first time, I’ll specify a new file system type of acfs:

#> mount -t acfs /dev/asm/acfs_db1-152 /u01/app/oracle/acsfmounts/acfs_db1/

#> mount
/dev/sda1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda3 on /tmp type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /u01/app/oracle/product type ext3 (rw)
/dev/sdc1 on /stage type ext3 (rw)
/dev/sdd1 on /u01/app/oracle/oradata type ext3 (rw)
/dev/sdh1 on /u01/app/oracle/oraFRA type ext3 (rw)
/dev/sdi2 on /DONTTOUCH type ext3 (rw)
/dev/mapper/vg1-lvfra on /u01/app/flash_recovery_area type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
.host:/ on /mnt/hgfs type vmhgfs (rw,ttl=1)
none on /proc/fs/vmblock/mountPoint type vmblock (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
oracleasmfs on /dev/oracleasm type oracleasmfs (rw)
/dev/asm/acfs_db1-152 on /u01/app/oracle/acsfmounts/acfs_db1 type acfs (rw)

And one other thing that I must not forget to do before I reboot my server: I’ll need to add an entry to my server’s /etc/fstab configuration file to insure that my eventual new Oracle 11gR2 database home is actually mounted before I attempt to start the database within that home:

LABEL=/                 /                       ext3    defaults        1 1
LABEL=/tmp              /tmp                    ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0
/dev/sdb1               /u01/app/oracle/product ext3    defaults        0 0
/dev/sdc1               /stage                  ext3    defaults        0 0
/dev/sdd1               /u01/app/oracle/oradata ext3    defaults        0 0
/dev/sdh1               /u01/app/oracle/oraFRA  ext3    defaults        0 0
/dev/sdi2               /DONTTOUCH              ext3    defaults        0 0
/dev/vg1/lvfra          /u01/app/flash_recovery_area    ext3    defaults        0 0

/dev/asm/acfs_db1-152   /u01/app/oracle/acfsmounts/acfs_db1     acfs   defaults        0 0

Installing a New Oracle 11g Release 2 Database Home

Just as with the installation of a new Oracle 11gR2 Grid Infrastructure home, I’ll once again use the Oracle Universal Installer (OUI) utility to install a new Oracle 11gR2 database home. I simply invoked the standard runInstaller OUI script, this time from the /stage/11.2.0/database/ directory from my staging disk. I won’t bore you with my responses to every single prompt on every OUI panel here; however, I’ve included all of the panels and my responses in Listing 2.3 starting with Figure 2.6.

Once all of the installation panels have been completed appropriately, the new Oracle database home has been successfully propagated, and I’ve run the standard root.sh script from within the new Oracle 11gR2 database home directory, I’m at last ready to create a new Oracle 11gR2 database, or even upgrade an existing 11gR1 database to release 11gR2 … but that’s a topic for my next article.

Next Steps

We’re making excellent progress … but it’s going to get a lot more interesting (and a lot less tedious) in the next article in this series, as I’ll demonstrate how to:

  • Upgrade an existing Oracle 11gR1 database to Oracle 11gR2 using an ACFS-resident Oracle Home
  • Use new ASMCMD command options to selectively monitor ASM disk performance and ASM file system content

References and Additional Reading

Before you proceed to experiment with any of these new features, I strongly suggest that you first look over the corresponding detailed Oracle documentation before trying them out for the first time. I’ve drawn upon the following Oracle Database 11g Release 2 documents for this article’s technical details:

E10881-02 Oracle Database 11gR2 New Features

E10592-03 Oracle Database 11gR2 SQL Language Reference

E10700-01 Oracle Database 11gR2 DataGuard Concepts and Administration

E10595-05 Oracle Database 11gR2 Administrator’s Guide

E10713-03 Oracle Database 11gR2 Concepts

E10820-02 Oracle Database 11gR2 Reference

E10500-02 Oracle Database 11gR2 Storage Administrator’s Guide

E10837-02 Oracle Database 11gR2 VLDB and Partitioning Guide

» See All Articles by Columnist Jim Czuprynski

Share:
Home
Mobile Site | Full Site
Copyright 2017 © QuinStreet Inc. All Rights Reserved