Oracle DBA in an Oracle Applications World - Part 2

Wednesday May 28th 2008 by Steve Callan

Steve Callan presents a simple – but effective – means of replicating your production environment for the first time on a new machine.

In a prior article about E-Business Suite, I provided an overview of the cloning process, which is essentially three steps: gather, copy and implement. Virtually all of the documentation regarding cloning assumes you already have replicants in place. Replicants you ask? Like the ones in Blade Runner? The ones who are “visually indistinguishable from [the production environment] – are used for dangerous and degrading work in [your test, dev and QA environments]?” Well, almost the same. What I mean here is that your test/dev/QA machines are virtually indistinguishable from those used in production. But, how do you get there?

What follows next is a simple – but effective – means of replicating your production environment for the first time on a new machine. First, let’s take care of some semantics. With respect to the non-production servers, regardless of their purpose (test, development or QA), let’s refer to them generically as the clone server(s) or clone(s).

Assuming the source and clone servers are physically similar, the basic requirement to get started is to install/match the operating system version or kernel, create the same mount points/partitions on the clones, and create the same (key) user accounts of oracle and applmgr (plus any other “people” accounts). Networking/IP address/ports are other setup areas that need to be addressed, which really isn’t any different than adding a new server into your environment.

But, you ask, don’t I have to install Oracle on the clone database server? You can if you want to, but it is easier to just copy the Oracle RDBMS and Apps software installation footprints from the sources to the clones. This isn’t just ORALCE_BASE and below; you’ll need to get pointer files such as oratab and oraInst.loc. With a direct copy, there’s no need to recompile the binaries.

Relative to the clones, the paths for the software installations are more than likely to be local. Special files (datafiles, redo logs, controlfiles, archived redo logs) may not be. You have to find a home or new set of paths for these files. If we assume new locations (because the source files are on a network file system or SAN) are required, what is an easy way to make the name/path changes? One way is to use “file name convert” parameters within the init.ora file. Another way is to (and you have to do this anyway) use the newly created controlfile, specifically, the SQL script used to create a new controlfile.

On the source database, backup the controlfile to a usable version/file that can be edited. You’ll be using the new SID name in that file, and that NEWSID value is what ties the log and datafiles to the controlfile which in turn is what binds the controlfile(s) to the init.ora file.

The clones can have different host names. Reconfiguring the cloned installations to recognize the new hostnames takes place during the implementation. The database name can (and more than likely) be different as well. Don’t forget to edit the init.ora file for other changes such as archive log and dump locations.

To provide some perspective on cloning the database tier, look at it this way: what if your production database server needs to be replaced, or you were just cloning the database (forget about this being for Oracle Apps, for the moment) onto a different server for development or QA users? It’s the same thing here. Recover the database via using a new init.ora file and re-create the controlfiles. If the copy of the source was taken while the database was open, and assuming archivelog mode, perform recovery on the clone.

Applying the configuration scripts

Running the RapidClone Perl script starts an interview process. Default/current values are shown and you can accept or reject the suggested value. If you reject, you will be prompted to enter new/updated information. One thing to keep in mind is the difference between a SID or server named config file versus a SID or server named file system path. By copying files over from the sources, you will inherit path names reflecting the source. What matters is that configuration files are named reflecting the new SID or server/host name.

If you make a mistake, or if the script encounters an error, simply re-run the script. Sample output (it looks the same on the apps tier and database tier) from the script is shown below, including an error. Key words to look for (replace with your values) are OLD and NEWSID, the_password, domain name, and host/server name.

:oldappsvr:/u003/app/applmgr/OLDSID/comntop/clone/bin>perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:

Provide the values required for creation of the new APPL_TOP Context file.
Do you want to use a virtual hostname for the target node (y/n) [n] ?:
Target system database SID [OLDSID]:NEWSID
Target system domain name [xyz.com]:
Target system database server node [oldsvr]:newsvr
Target system database domain name [xyz.com]:
Does the target system have more than one application tier server node (y/n) [y] ?:y
Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:
Target system APPL_TOP mount point [/u003/app/applmgr/OLDSID/appltop]:
Target system COMMON_TOP directory [/u003/app/applmgr/OLDSID/comntop]:
Target system 8.0.6 ORACLE_HOME directory [/u003/app/applmgr/OLDSID/8.0.6]:
Target system iAS ORACLE_HOME directory [/u003/app/applmgr/OLDSID/iAS]:
Do you want to preserve the Display set to oldappssvr:0.0 (y/n) [y] ?:n
Target system Display [newappsvr:0.0]:
Location of the JDK on the target system [/usr/j2se/j2sdk1.4.2_13]:/usr/jdk/jdk1.5.0_15
Target system JRE_TOP [/usr/j2se/j2sdk1.4.2_13]:/usr/jdk/jdk1.5.0_15
Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:
Web Listener port is 8002
Complete port information available at 
UTL_FILE_DIR on database tier consists of the following directories.
1. /usr/tmp
2. /usr/tmp
3. /u002/app/oracle/product/OLDSID/9.2.0/appsutil/outbound/NEWSID_newsvr
4. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1]:
Creating the new APPL_TOP Context file from :
The new APPL_TOP context file has been created :
Log file located at /u003/app/applmgr/OLDSID/comntop/clone/bin/CloneContext_04180252.log
Running Rapid Clone with command:
        perl adclone.pl java=../jre/1.3.1 mode=apply 
        component=appsTier method=CUSTOM 
        showProgress contextValidated=true
The 806 installer uses the oratab file.
The file oratab needs to exist at:
Rapid Clone will use the oraInst.loc file.
The file oraInst.loc needs to exist at:
Please run the script /tmp/orainstRoot.sh as root
Restart adclone.pl after performing required corrections.
 ERROR: Failed to execute adclone.pl
 Please check logfile.

So, which tier was being implemented via the adclone.pl script? Several ways to tell, but one of the easiest is based on the questions regarding these four locations:

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:
Target system APPL_TOP mount point [/u003/app/applmgr/OLDSID/appltop]:
Target system COMMON_TOP directory [/u003/app/applmgr/OLDSID/comntop]:
Target system 8.0.6 ORACLE_HOME directory [/u003/app/applmgr/OLDSID/8.0.6]:
Target system iAS ORACLE_HOME directory [/u003/app/applmgr/OLDSID/iAS]:

Had it been the database tier, the questions would have been like:

Number of DATA_TOP's on the target system [4]:
Target system DATA_TOP 1 [/u004/app/oradata/OLDSID]:
Target system DATA_TOP 2 [/u005/app/oradata/OLDSID]:
Target system DATA_TOP 3 [/u008/app/oradata/OLDSID]:
Target system DATA_TOP 4 [/u009/app/oradata/OLDSID]:

Additionally, the database tier questions in the interview leave no doubt as to which tier you’re dealing with. I left the error in place to show the effect of what happens when ancillary files (those not normally under ORACLE_BASE) are missing.

What if some of the servers (in the Apps context, not the machine context) are split across the Apps server (machine or node) and the database server? Everything on the Apps node is run as applmgr, and everything run on the database node – for the database component – is run as oracle. You will need to have applmgr as an account (and owner) of the applications tier components installed on this node. In other words, you run the adclone script twice, once for the oracle user/owner, and once for the applmgr user/owner.

One way to see if the apps tier is split across physical nodes is to examine the state of processes at the end of the log file generated by adstrtal.sh script.

The report format is:
  <Service>                                      <Script>         <Status>
  Oracle Apache Server NEWSID_newappsvr             adapcctl.sh      Started
  OracleTNSListener80APPS_NEWSID_newappsvr          adalnctl.sh      Started
  OracleFormsServer-Forms60NEWSID_newappsvr         adfrmctl.sh      Started
  Oracle Metrics Client NEWSID_newappsvr            adfmcctl.sh      Started
  Oracle Metrics Server NEWSID_newappsvr            adfmsctl.sh      Started
  Oracle Fulfillment Server NEWSID_newappsvr        jtffmctl.sh      Started
  Oracle Discoverer services NEWSID_newappsvr       addisctl.sh      Started
  Oracle Restricted Apache Server NEWSID_newappsvr  adaprstctl.sh    Disabled
  Oracle Apache Server NEWSID_newappsvr for PL/SQL  adapcctl.sh      Disabled
  Oracle TCF SocketServer NEWSID_newappsvr          adtcfctl.sh      Disabled
  OracleConcMgrNEWSID_newappsvr                     adcmctl.sh       Disabled
  OracleReportServer-Rep60_NEWSID                   adrepctl.sh      Disabled
  Oracle ICSM NEWSID_newappsvr                      ieoicsm.sh       Disabled
  Oracle iProcurement Bulk Loader NEWSID_newappsvr  icxblkctl.sh     Disabled
ServiceControl is exiting with status 

Where is the Concurrent Manager running? Given that it is disabled (see OracleConcMgrNEWSID_newappsvr), probably not on the server named newappsvr. It would be reasonable to assume that it is running on the database server. The corresponding log file there shows:

The report format is:
  <Service>                                      <Script>         <Status>
  OracleTNSListener80APPS_NEWSID_newsvr          adalnctl.sh      Started
  OracleConcMgrNEWSID_newsvr                     adcmctl.sh       Started
  OracleReportServer-Rep60_OCVREP                adrepctl.sh      Started
  Oracle Apache Server NEWSID_newsvr             adapcctl.sh      Disabled
  Oracle Restricted Apache Server NEWSID_newsvr  adaprstctl.sh    Disabled
  Oracle Apache Server NEWSID_newsvr for PL/SQL  adapcctl.sh      Disabled
  Oracle TCF SocketServer NEWSID_newsvr          adtcfctl.sh      Disabled
  OracleFormsServer-Forms60NEWSID_newsvr         adfrmctl.sh      Disabled
  Oracle Metrics Client NEWSID_newsvr            adfmcctl.sh      Disabled
  Oracle Metrics Server NEWSID_newsvr            adfmsctl.sh      Disabled
  Oracle ICSM NEWSID_newsvr                      ieoicsm.sh       Disabled
  Oracle Fulfillment Server NEWSID_newsvr        jtffmctl.sh      Disabled
  Oracle iProcurement Bulk Loader NEWSID_newsvr  icxblkctl.sh     Disabled
  Oracle Discoverer services NEWSID_newsvr       addisctl.sh      Disabled
ServiceControl is exiting with status 0

The output also reflects the scripts used to start/stop each component individually.

One other key piece of information concerns ports, and by extension, URLs. The URL used to get to the Apps logon page will be the same as what is in production, except for one item, and that is the host or server name. If you kept the same port assignments, just update the base URL to reflect the new server name. Port assignments are recorded in the portpool.lst file under $APPL_TOP/admin/out/NEWSID_newappsvr. Port 8002 was used in production, and 8002 will be used on the clone.

[applmgr newappsvr] $ more portpool.lst
Web Listener Port                      :  8002
Database Port                          :  1523
RPC Port                               :  1628
Reports Port                           :  7002
OPROC Manager Port                     :  8102
Web PLSQL Port                         :  8202
Servlet Port                           :  8800
Forms Listener Port                    :  9002
Metrics Server Data Port               :  9102
Metrics Server Req. Port               :  9202
JTF Fulfillment Server Port            :  9302
Map Viewer Servlet Port                :  9802
OEM Web Utility Port                   :  10002
VisiBroker OrbServer Agent Port        :  10102
MSCA Server Port                       :  10202
MSCA Dispatcher Port                   :  10302
OACORE Servlet Port Range              :  16020-16029
Discoverer Servlet Port Range          :  17020-17029
Forms Servlet Port Range               :  18020-18039
TCF port                               :  -1
XMLSVCS Servlet Port Range             :  19020-19029
Java Object Cache Port                 :  12345

In Closing

Cloning an Oracle Apps environment using RapidClone is relatively easy once the replicants have been created/established. Assuming no changes to the apps tier (e.g., new forms), a subsequent clone may only consist of cloning the database. The tricky part is getting the very first clone in place, and with that, you have the overhead of performing steps with more complexity. But, as I hope I have done in this article, once the overall process has been explained, you can see that making the first clone isn’t that hard to do.

» See All Articles by Columnist Steve Callan

Mobile Site | Full Site