Quantcast
Channel: Oracle Database&EBS General – HELIOS BLOG
Viewing all articles
Browse latest Browse all 138

ORA-29702: error occurred in Cluster Group Service operation

$
0
0

I faced with that error on r12.1.3 EBS system. My db version is 11.2&2 node RAC system.

My first node is goes down by below errors:

opiodr aborting process unknown ospid (11075852) as a result of ORA-28
LMON (ospid: 63767522) detects hung instances during IMR reconfiguration
LMON (ospid: 63767522) tries to kill the instance 2. 
Please check instance 2’s alert log and LMON trace file for more details.
Tue Mar 19 10:58:36 2020
USER (ospid: 32900426): terminating the instance due to error 481
Tue Mar 19 10:58:36 2020
Errors in file /oracle11g/PROD00/db/diag/rdbms/PROD00/PROD001/trace/PROD001_lmon_63767522.trc:
ORA-29702: error occurred in Cluster Group Service operation
System state dump is made for local instance
System State dumped to trace file /oracle11g/PROD00/db/diag/rdbms/PROD00/PROD001/trace/PROD001_diag_9373174.trc
Instance terminated by USER, pid = 32900426


From PROD001_lmon_63767522.trc file:

*** 2020-03-19 10:55:00.531

* DRM RCFG called (swin 1)
CGS recovery timeout = 85 sec
Begin DRM(5108) (swin 1)

*** 2020-03-19 10:57:11.567
Name Service frozen

..
...
kjxgmpoll: terminate the CGS reconfig.
Error: Cluster Group Service reconfiguration takes too long <<< here is error 
LMON caught an error 29702 in the main loop 
error 29702 detected in background process <<< here is error code
ORA-29702: error occurred in Cluster Group Service operation

I see many drm quiesce hang messages in the lmon.trc files

find . -name "*lmon*.trc" |xargs grep -i "quiesce hang"
./oracle/PROD001_lmon_63767522.trc:* Request pseudo reconfig due to drm quiesce hang
./oracle/PROD001_lmon_63767522.trc:* Request pseudo reconfig due to drm quiesce hang
./oracle/PROD002_lmon_14221454.trc:* Request pseudo reconfig due to drm quiesce hang
./oracle/PROD002_lmon_14221454.trc:* Request pseudo reconfig due to drm quiesce hang
./oracle/PROD002_lmon_14221454.trc:* Request pseudo reconfig due to drm quiesce hang
./oracle/PROD002_lmon_14221454.trc:* Request pseudo reconfig due to drm quiesce hang
./oracle/PROD002_lmon_14221454.trc:* Request pseudo reconfig due to drm quiesce hang

Based on these, The issue does appear to be an occurrence of bug : 12879027 LMON gets stuck in DRM quiescence causing intermittent pseudo reconfiguration

To get the fix for the bug, please install the 11.2.0.3 patchset into the rdbms $ORACLE_HOME and then , apply on top, the 11.2.0.3.3 PSU, or higher/later PSUs

More details can be found at MOS note: Bug 12879027 – LMON gets stuck in DRM quiesce causing intermittent pseudo reconfiguration [ID 12879027.8]


Viewing all articles
Browse latest Browse all 138

Trending Articles