DB2 Instance and Fast Communication Manager

Thursday Dec 11th 2003 by Marin Komadina
Share:

The process of tuning an ESE DB2 database is automatic--almost. Several components still require human intervention and the Fast Communication Manager is one of them. Learn about the global DB2 memory structure and FCM tuning parameters for a SUN Solaris platform.

The process of tuning an ESE DB2 database is almost automatic, at least according to the DB2 manuals and Web announcements. However, much of this tuning still requires human intervention. The Fast Communication Manager is one of the components that need hand tuning. FCM is used to control the information flow between database partitions as well as to manage optimal performance for shared nothing architecture. IBM is keeping the details for the FCM interconnection interface under cover as much as possible, with the intention to make it self managed in the future. However, version 8.1 still requires some configuration. In this article, I will explain the global DB2 memory structure and FCM tuning parameters for a SUN Solaris platform.

This article covers:

  • DB2 Memory Model
  • DBM Instance Memory Structure
  • Fast Communication Manager (FCM)
  • Conclusion

DB2 Memory Model

Click for larger image

The DB2 memory model has the following main components:

Database Manager Instance Memory (DBM Shared Memory) is the database manager's (DBM) allocated memory, and is used for the DB2 data structures manipulation and instance activity handling. Information regarding database instance memory handling is hard to find. DB2 version 7.2 has no ability to control instance memory allocation, while there exists in DB2 version 8.1 a new DBM configuration parameter instance_memory for defining suitable instance memory allocation:

Parameter

Configurable Online

Automatic

instance_memory

No

Yes

The range of values for the DBM instance memory allocation are between 32 KB and 2GB or between 8 and 524 288 4K pages.

# db2 get db cfg | grep DATABASE_MEMORY
Size of database shared memory (4KB)  (DATABASE_MEMORY) = AUTOMATIC

By default, DB2 will automatically (AUTOMATIC) adjust instance memory allocation to reflect current resource requirements. The allocated, internally calculated memory will be acquired on instance activation and freed on instance deactivation.

Database Global Memory (DB Shared Memory) is the database allocated memory area used by all applications connected to the same database. DB global memory is allocated on database activation and cleared on database deactivation. Several shared, dynamically tunable areas build global memory. Included memory areas are buffer pools, lock list, database and utility heap, package cache, and catalog cache. Additionally, each database in the instance has a private global memory block. The minimum reserved, per database shared memory allocation is defined with the DBM parameter database_memory.

Parameter

Configurable Online

Automatic

database_memory

No

Yes

The range of values for the DB database memory allocation are between 0 KB and 16TB or between 0 and 4 294 967 295 4K pages.

DB2 will calculate the necessary memory requirements on database activation and therefore automatically adjust the database_memory parameter.

# db2 get db cfg | grep DATABASE_MEMORY
Size of database shared memory (4KB)  (DATABASE_MEMORY) = AUTOMATIC

The parameter DATABASE_MEMORY is provided only for DB2 version 8.1 and above.

Application Memory is the memory segment used for internal data exchange between the DB2 database and connected applications. Application memory is global, shared or private and is allocated on demand. Application memory size is controlled by the defined maximum concurrent active applications (maxappls) in the database.

# db2 get db cfg | grep MAXAPPLS
Max number of active applications            (MAXAPPLS) = 40        --> version 7.2
Max number of active applications            (MAXAPPLS) = AUTOMATIC --> version 8.1

DB2 version 7.2 has 40 as a default number of active applications while version 8.1 defaults MAXAPPLS to AUTOMATIC value.

Agent Private Memory is an application service memory. This memory is allocated by the database agents serving connected database applications. The agent private memory includes sort heaps, cursor information and session context area. Common agent's memory allocation is limited by lower values of the DB parameters maxappls and maxagents.

# db2 get dbm cfg | grep MAXAGENTS
Max number of existing agents               (MAXAGENTS) = 400

DBM Instance Memory Structure

DBM Instance Memory (usually called Global Control Block) has the following internal structure:

Click for larger size

Database Manager Instance memory elements:

  • Monitor Heap Size
  • Directory Cache
  • Audit Buffer
  • FCM Memory Allocations
  • Backup and Restore Buffer

Every DB2 instance has the same shown structure. Backup and restore buffers are allocated and deallocated on demand and are not permanently present. The ESE DB2 database with enabled intra_partition or inter-partition parallelism (logical or physical partitioned DB2 database) has allocated an additional memory segment for the Fast Communication Manager (FCM). FCM provides communication support between database partitions.

The database's instance memory allocation is controlled with the following parameters:

Database System Monitor Heap Size

Parameter

Configurable Online

Automatic

mon_heap_sz

No

No

The Monitor Heap Size is the memory segment used for collecting monitor data. The Monitor heap size is measured in 4KB pages.

# db2 get dbm cfg | grep MON_HEAP_SZ
Database monitor heap size (4KB) (MON_HEAP_SZ) = 56 

Activities such as snapshots, activating or resetting monitor switches and event monitors will populate the memory buffer allocated on the instance startup. Having the DBM setting mon_heap_size=0 will prevent any collection of monitored data. For a heavily used database with many applications connected and a high demand for collecting monitor information, may cause a monitor buffer overflow. Such situations with an inadequate monitoring buffer will result in an error condition, entries in the database alert log and failures on starting additional monitor sessions. The default monitor heap size is 56 in 4KB pages or 224KB. The maximum size for the mon_heap_size is 240 MB or 60000 in 4KB pages.

Directory cache support or database directory listings

Parameter

Configurable Online

Automatic

dir_cache

No

No

The DBM configuration parameter, dir_cache enables database directory caching. Database directory is the common name for the system db directory, local database directory, node directory and DCS directory. Each of them is used for storing determined content:

  • system database directory contains information related to all known databases
  • local database directory contains information related to all local databases
  • node directory contains connectivity information related to the remote databases
  • DCE directory contains connectivity information for the host databases

Database directory information caching reduces connection delay and directory file I/O.

DBM will automatically govern the shared directory cache, while applications have to manage a separate, private directory cache. The shared cache is populated during DBM (instance) startup.

# db2 get dbm cfg | grep DIR_CACHE
Directory cache support (DIR_CACHE) = YES

After startup, all local and remote database and node entries are cached in the shared cache. Any newly created entry that results from catalog operations (database catalog, uncatalog, create or drop) would not be cached in the shared directory cache. Only on the next DBM restart will new entries be cached. By default, dir_cache parameter is always enabled.

Listing node directory entries for the test DB2 instance:

db2 => list node directory  

Node Directory
 
 Number of entries in the directory = 2
 
Node 1 entry:
 	
 Node name                      = ARTIST01
 Comment                        = Article DB2 Server
 Protocol                       = TCPIP
 Hostname                       = ARTIST01.zg.tel.hr
 Service name                   = 50000
 
Node 2 entry:
 
 Node name                      = HOST99
 Comment                        =
 Protocol                       = TCPIP
 Hostname                       = IBMHOST01.zg.tel.hr
 Service name                   = 51100

Listing system database directory entries for the test DB2 instance:

db2 => list database directory 

 System Database Directory

 Number of entries in the directory = 3

Database 1 entry:

 Database alias                  = ARTIST
 Database name                   = ARTIST
 Node name                       = JUPITER0001
 Local database directory        = /export/users/db2inst1/node0/artist
 Database release level          = 9.00
 Comment                         = ARTICLE DB2
 Directory entry type            = Indirect
 Authentication                  = SERVER
 Catalog node number             = 0

Database 2 entry:

 Database alias                  = HOST01
 Database name                   = HOST01
 Node name                       = SATURN00010
 Database release level          = 9.00
 Comment                         = IBM Host DB2
 Directory entry type            = Remote
 Authentication                  = DCS_ENCRYPT
 Catalog node number             = -1

Database 3 entry:
 
 Database alias                  = MARIN
 Database name                   = MARIN
 Local database directory        = /export/users/db2inst1/node0/artist
 Database release level          = 9.00
 Comment                         = ALIAS FOR LOCAL DB 
 Directory entry type            = Indirect
 Authentication                  = SERVER
 Catalog node number             = 0

Remark following specific entries in the system database directory:

1- ARIST (Directory entry type=Indirect) local database entry

2- HOST01 (Directory entry type=Remote) remote Host DB2 database entry

3- MARIN (Directory entry type=Indirect) local database alias

Audit Buffer Size

Parameter

Configurable Online

Automatic

audit_buf_sz

No

No

The Audit Buffer Size parameter specifies the size for the audit buffer, created during DBM start in the instance control block and used for audit information caching.

# db2 get dbm cfg | grep AUDIT_BUF_SZ
Audit buffer size (4KB)                  (AUDIT_BUF_SZ) = 0

The default value for audit_buf_sz parameter is 0. When this parameter is set to the default of zero (0), DBM will not cache the audit records but will write them ddirectly, synchronously to disk. The Audit buffer values range from a minimum of 0 to a maximum of 65000 in 4KB pages, resulting in a maximum audit buffer size of 254MB.

Backup and Restore Default Size

Two additional DBM parameters need to be taken into computation for the DBM global control block. These are the internal instance backup and restore buffers. They behave fully dynamically, and are allocated and deallocated only during backup and restore operations. Each call for the backup or restore utility will open a buffer and at the end of the operation close it. Such a buffering speeds the whole process and shortens the backup/recovery time. The default buffer size is predefined with the instance parameters, or it is specified on the command line, together with the database call for the backup/restore utility.

Parameter

Configurable Online

Automatic

Backbufsz

No

No

restbufsz

No

No

Default settings for the instance backup and restore buffer size:

# db2 get dbm cfg | grep BUFSZ
Backup buffer default size (4KB) 
           (BACKBUFSZ) = 1024
Restore buffer default size (4KB)
           (RESTBUFSZ) = 1024

The default backup buffer size is 1024 in 4KB pages, or 4MB. Backup buffer working range resulting from min 32KB to max 2GB.

The default restore buffer size is 1024 in 4KB pages, or 4MB. Restore buffer working range resulting from min 64KB to max 2GB.

FCM Shared Memory and FCM Buffer Pool

A separate memory area is allocated for the fast communication manager. The main FCM memory segments are FCM shared memory and FCM Buffer pool.

Fast Communication Manager (FCM)

The DB2 Fast Communication Manager or FCM for short is a specific IBM EDU (engine dispatchable unit) or UNIX process, that is responsible for handling database inter-parallel node communication in multimode database configuration. Inter node data and message traffic circulates over FCM shared memory structures between database partitions.

Every database partition has an associated db2fcmdm FCM deamon process, which is started during the DB2 instance start.

Let's look at the DB2 process tree (db2ptree) for a specific database configuration, with one physical and three logical nodes (partitions):

$ db2ptree
6836  -ksh
  6849  db2ptree
14174  db2wdog 0
  14180  db2sysc 0
    14196  db2gds 0
      14215  db2resyn 0
      14216  db2srvlst 0
    14197  db2fcmdm 0  -> FCM EDU or FCM deamon for partition 0
    14202  db2pdbc 0
    14203  db2ipccm 0
      14221  db2agent (idle) 0
    14204  db2tcpcm 0
      14235  db2agent (idle) 0
    14217  db2panic (idle) 0
14189  db2wdog 1
  14190  db2sysc 1
    14198  db2gds 1
      14207  db2resyn 1
      14208  db2srvlst 1
      26550  db2agent (idle) 1
    14199  db2fcmdm 1   -> FCM EDU or FCM deamon for partition 1
    14205  db2pdbc 1
    14206  db2ipccm 1
    14209  db2panic (idle) 1
14194  db2wdog 2
  14195  db2sysc 2
    14200  db2gds 2
      14212  db2resyn 2
      14213  db2srvlst 2
      26557  db2agent (idle) 2
    14201  db2fcmdm 2 -> FCM EDU or FCM deamon for partition 2
    14210  db2pdbc 2
    14211  db2ipccm 2
    14214  db2panic (idle) 2

FCM Configuration Files

During instance start, the FCM deamon process will read communication parameters from the operating system service file /etc/service as well as configuration parameters from the database configuration file, db2nodes.cfg. The dedicated port group used exclusively for FCM communication is configured in the service file:

% more /etc/services | grep DB2	
DB2_db2inst1               50000/tcp -> start FCM address
DB2_db2inst1_END           50002/tcp -> end FCM address 

Following the given example, separate TCP/IP ports (50000, 50001, and 50002) are used as FCM communication ports. Every database partition has an associated unique communication port. Each database partition in a DB2 mulitpartitioned database has the same entries in the service and database configuration files on all physical machines. Correctly defined communication ports are critical for successful, fast communication manager functioning. If they are not defined correctly, new partitions cannot be added to an existing multipartition database configuration, nor can a non-partitioned database be transformed to partitioned one.

The database configuration parameter file db2nodes.cfg contains the following partitions information:

>> cat db2nodes.cfg
0 ARTIST01 0  - > first logical DB2 partition
1 ARTIST01 1  - > second logical DB2 partition
2 ARTIST01 2  - > third logical DB2 partition

The described file consists of three entries, one per logical partition.

Once established, communication between partitions became a main lifeline for multipartition database activity.

FCM Parameters

Table with all DBM (instance) parameters related to the fast communication manager:

Parameter

Configurable Online

Automatic

fcm_num_buffers

Yes

No

fcm_num_rqb

No (only ver. 7)

No

fcm_num_connect

No ( only ver. 7)

No

fcm_num_anchors

No ( only ver. 7)

No

List of FCM parameters with associated default values for partitioned DB2 7.2 database:

# db2 get dbm cfg | grep FCM
 No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) = 1024
 Number of FCM request blocks              (FCM_NUM_RQB) = 512
 Number of FCM connection entries      (FCM_NUM_CONNECT) = (FCM_NUM_RQB * 0.75) 
 Number of FCM message anchors         (FCM_NUM_ANCHORS) = (FCM_NUM_RQB * 0.75) 

List of FCM parameters with associated default values for partitioned DB2 8.1 database:

# db2 get dbm cfg | grep FCM
No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) = 4096

FCM parameters descriptions:

fcm_num_buffers - the amount of FCM communication buffers, with default value 4096 in 4KB pages for partitioned DB2 database.

fcm_num_rqb - the amount of FCM request blocks used for communication between the FCM daemon and agent or between agents. Default value for partitioned database is 2048 in 4KB pages.

fcm_num_connect - the number of FCM connection entries used for inter-agents data communication. The default value for partitioned databases totals 1536 in 4KB pages.

fcm_num_anchors = the message registers used for inter-agents message communication. The default value for partitioned databases totals to 1536 in 4KB pages.

Review of default values for FCM configuration parameters:

Parameter

Deault 32bit (4KB pages)

Default 64bit (4KB pages)

Min. 32 bit (4K pages)

Max. 32 bit (4KB pages)

Min. 64 bit (4KB pages)

Max. 64 bit (4KB pages)

fcm_num_buffers

4096

4096

128

65 300

128

524 288

fcm_num_rqb

 

2048

10,000

128

120 000

128

524 288

fcm_num_connect

1536

2048

128

120 000

128

524 288

fcm_num_anchors

1536

2048

128

120 000

128

524 288

To make the picture complete, I need to say a few words about other types of DB2 parallelism. DB2 has an additional option, called intra-partition parallelism, to handle parallel requests inside a single partition, . A special DBM parameter is provided:

# db2 get dbm cfg | grep INTRA
Enable intra-partition parallelism     (INTRA_PARALLEL) = NO

Activating intrapartition parallelism ( INTRA_PARALLEL=YES ) for (DB2
EE), causes
FCM to be started for a single partition database and is used to handle
parallelism for database agents. Without the setting, FCM will not be started for a
single node database, in which case we have following default FCM default values:

Parameter

INTRA_PARALLEL=Y with local clients only (4KB pages)

INTRA_PARALLEL=Y with local and remote clients (4KB pages)

fcm_num_buffers

512

1024

fcm_num_rqb

256

512

fcm_num_connect

192

384

fcm_num_anchors

192

384

Conclusion

The very first task, before starting with any FCM tuning activity, is to understand the internal DB2 working logic and to have an overview of the actual DBM settings. My preferred note of interrogation for any foreign database:

  • basic hardware and system configuration
  • number of database instances running on the system
  • number of database partition servers running on the system
  • number of database users
  • complexity and type of the applications
  • complexity of running queries

» See All Articles by Columnist Marin Komadina

Share:
Home
Mobile Site | Full Site
Copyright 2017 © QuinStreet Inc. All Rights Reserved