Forum Discussion

H_A_N_B_U's avatar
H_A_N_B_U
Level 3
10 years ago

VCS Database Service Group Service Group Fails After Initiating Offline

Hi,


We have 2 Storage Foundation for Oracle RAC 5.1SP1 servers running on AIX 5.3 Servers. We have Application Service Group and Database Service Group  which is both active active. I encountered a problem when i initiated manual offline of the Service group. Application Service Group is ok. But in the Database Service Group it failed and the 2 servers unpectedly restarted and i encountered split-brain condition.

Here's the engineA.log:

ServiceGroup_PRD Application successfull offiline

2015/06/24 01:17:14 VCS INFO V-16-1-50135 User root fired command: hagrp -offline ServiceGroup_PRD  Server1  from localhost

2015/06/24 01:17:14 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_PRD on system SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:17 VCS INFO V-16-1-10305 Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:19 VCS INFO V-16-1-10305 Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:19 VCS INFO V-16-1-10305 Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:20 VCS INFO V-16-1-10305 Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:20 VCS INFO V-16-1-10305 Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:24 VCS INFO V-16-1-10305 Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:24 VCS INFO V-16-1-10305 Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:32 VCS INFO V-16-1-10305 Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:32 VCS INFO V-16-1-10305 Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:33 VCS INFO V-16-1-10305 Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:34 VCS INFO V-16-1-10305 Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:35 VCS NOTICE V-16-1-10446 Group SERVICEGROUP_PRD is offline on system SERVER1

2015/06/24 01:17:35 VCS INFO V-16-6-15002 (SERVER1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/lvmvg_postoffline SERVER1 SERVICEGROUP_PRD   successfully

2015/06/24 01:17:35 VCS INFO V-16-6-0 (SERVER1) postoffline:Invoked with arg0=SERVER1, arg1=SERVICEGROUP_PRD

2015/06/24 01:17:36 VCS INFO V-16-6-15002 (SERVER1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline SERVER1 SERVICEGROUP_PRD   successfully

2015/06/24 01:17:41 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_PRD on system SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:45 VCS INFO V-16-1-10305 Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:45 VCS INFO V-16-1-10305 Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:47 VCS INFO V-16-1-10305 Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:47 VCS INFO V-16-1-10305 Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:47 VCS NOTICE V-16-1-10446 Group SERVICEGROUP_PRD is offline on system SERVER2

2015/06/24 01:17:47 VCS INFO V-16-6-15002 (SERVER2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/lvmvg_postoffline SERVER2 SERVICEGROUP_PRD   successfully

2015/06/24 01:17:47 VCS INFO V-16-6-0 (SERVER2) postoffline:Invoked with arg0=SERVER2, arg1=SERVICEGROUP_PRD2015/06/24 01:17:48 VCS INFO V-16-6-15002 (SERVER2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline SERVER2 SERVICEGROUP_PRD successfully

 

ServiceGroup_DBPRD Database fails to offline

2015/06/24 01:18:51 VCS INFO V-16-1-50135 User root fired command: hagrp -offline SERVICEGROUP_DBPRD  SERVER1  from localhost

2015/06/24 01:18:51 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_DBPRD on system SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ocr_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource quorum_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog1_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog2_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u02_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u03_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u04_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u05_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u06_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u07_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource voting_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:52 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:redolog1_mnt:offline:Umount Failed : Mount Point : /redolog1

2015/06/24 01:18:52 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:quorum_mnt:offline:Umount Failed : Mount Point : /quorum

2015/06/24 01:18:52 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog1_mnt:offline:Attempting fuser TERM : Mount Point : /redolog1

2015/06/24 01:18:52 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:quorum_mnt:offline:Attempting fuser TERM : Mount Point : /quorum

2015/06/24 01:18:53 VCS INFO V-16-1-10305 Resource ocr_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:18:53 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmocr (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:53 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:redolog2_mnt:offline:Umount Failed : Mount Point : /redolog2

2015/06/24 01:18:53 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog2_mnt:offline:Attempting fuser TERM : Mount Point : /redolog2

2015/06/24 01:18:55 VCS INFO V-16-1-10305 Resource cvmocr (Owner: unknown, Group: SERVICEGROUP_DBPRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:18:55 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u03_mnt:offline:Umount Failed : Mount Point : /u03

2015/06/24 01:18:55 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u03_mnt:offline:Attempting fuser TERM : Mount Point : /u03

2015/06/24 01:18:57 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u02_mnt:offline:Umount Failed : Mount Point : /u02

2015/06/24 01:18:57 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u02_mnt:offline:Attempting fuser TERM : Mount Point : /u02

2015/06/24 01:18:58 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u04_mnt:offline:Umount Failed : Mount Point : /u04

2015/06/24 01:18:58 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u04_mnt:offline:Attempting fuser TERM : Mount Point : /u04

2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog1_mnt:offline:Attempting fuser TERM : Mount Point : /redolog1

2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog2_mnt:offline:Attempting fuser TERM : Mount Point : /redolog2

2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:quorum_mnt:offline:Attempting fuser TERM : Mount Point : /quorum

2015/06/24 01:40:15 VCS NOTICE V-16-1-11022 VCS engine (had) started

2015/06/24 01:40:15 VCS INFO V-16-1-10196 Cluster logger started

2015/06/24 01:40:15 VCS NOTICE V-16-1-11050 VCS engine version=5.1

2015/06/24 01:40:15 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.00.0

2015/06/24 01:40:15 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.1-10/06/09-14:37:00

2015/06/24 01:40:15 VCS NOTICE V-16-1-10114 Opening GAB library

2015/06/24 01:40:16 VCS NOTICE V-16-1-10619 'HAD' starting on: SERVER1

2015/06/24 01:40:16 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms

2015/06/24 01:40:16 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms

2015/06/24 01:40:16 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message

2015/06/24 01:40:26 VCS INFO V-16-1-10077 Received new cluster membership

2015/06/24 01:40:26 VCS NOTICE V-16-1-10112 System (SERVER1) - Membership: 0x3, DDNA: 0x0

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING

2015/06/24 01:40:26 VCS NOTICE V-16-1-10086 System SERVER1 (Node '0') is in Regular Membership - Membership: 0x3

2015/06/24 01:40:26 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3

2015/06/24 01:40:26 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'SERVER2'

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER2 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER1 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER2 (Node '1') changed state from CURRENT_DISCOVER_WAIT to CURRENT_PEER_WAIT

2015/06/24 01:40:28 VCS NOTICE V-16-1-52006 UseFence=SCSI3. Fencing is enabled

2015/06/24 01:40:28 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:40:43 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:40:58 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:13 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:28 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:43 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:58 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing

 

The solution i did here was to run vxfenclearpre.

Is there any logs i need to check why the servers faults and restarted unexpectedly after service group offline?

 

Any help will appreciate.

Thanks.

 

AIX
business continuity
Cluster Server
Error messages
Tip-How to
Troubleshooting
  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    10 years ago

    You have an issue here. Three resources that are really referring to the same diskgroup. So if one offlines the DG the other will fault.

     

    CVMVolDg cvmquorum (

                                        CVMDiskGroup = oradgredolog1

                                        CVMVolume = { volredolog1 }

                                        CVMActivation = sw

                                        )

     

                      CVMVolDg cvmredolog1 (

                                        CVMDiskGroup = oradgredolog1

                                        CVMVolume = { volredolog1 }

                                        CVMActivation = sw

                                        )

     

                      CVMVolDg cvmredolog2 (

                                        CVMDiskGroup = oradgredolog1

                                        CVMVolume = { volredolog1 }

                                        CVMActivation = sw

                                        )

     

  • mikebounds's avatar
    mikebounds
    10 years ago

    Is your Oracle voting disk on /ocr or /voitng - if so this is your issue - you need to move this CFSMount and CVMVolDg resource to cvm group and add a cssd resource so you have something like below:

    ccsd2.png

    If you bring down voting disk before stopping cssd, Oracle will panic the box - you should be able to confirm this from the system log which should show that Oracle paniced the server

    Mike

  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    Level 6
    10 years ago

    Please post your main.cf.

     

    Also, Storage Foundation for Oracle RAC 5.1SP1 is very old. Might want to patch that.

  • mikebounds's avatar
    mikebounds
    Level 6
    10 years ago

    I would guess that  /quorum is maybe your Oracel Voting disks so when this was offined Oracle panics the mode.

    You should have the cssd resource dependent on the voting disk resouce so that cssd is stopped before offling voting disk.

    Mike

  • Sunil_Yadav's avatar
    Sunil_Yadav
    Level 4
    10 years ago

    Log snippet shared isn’t enough for RCA of the issues. Can you share more evidences(e.g. main.cf, engine log, syslog, etc.) for complete/accurate RCA of the issue.

     

    Thanks & Regards,
    Sunil Y

  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    Level 6
    10 years ago

    You have an issue here. Three resources that are really referring to the same diskgroup. So if one offlines the DG the other will fault.

     

    CVMVolDg cvmquorum (

                                        CVMDiskGroup = oradgredolog1

                                        CVMVolume = { volredolog1 }

                                        CVMActivation = sw

                                        )

     

                      CVMVolDg cvmredolog1 (

                                        CVMDiskGroup = oradgredolog1

                                        CVMVolume = { volredolog1 }

                                        CVMActivation = sw

                                        )

     

                      CVMVolDg cvmredolog2 (

                                        CVMDiskGroup = oradgredolog1

                                        CVMVolume = { volredolog1 }

                                        CVMActivation = sw

                                        )

     

  • mikebounds's avatar
    mikebounds
    Level 6
    10 years ago

    Is your Oracle voting disk on /ocr or /voitng - if so this is your issue - you need to move this CFSMount and CVMVolDg resource to cvm group and add a cssd resource so you have something like below:

    ccsd2.png

    If you bring down voting disk before stopping cssd, Oracle will panic the box - you should be able to confirm this from the system log which should show that Oracle paniced the server

    Mike

  • mikebounds's avatar
    mikebounds
    Level 6
    10 years ago

    Just to add to what Riaan said, normally you would have:

    • One diskgroup for Oracle voting disk
    • One diskrgoup per database and in this single diskrgoup you can have multiple volumes for archive, redo and data

    As aside, I would recommend using naming convention something like:

    /oracle/DB1/data1
    /oracle/DB1/data2
    /oracle/DB1/data3
    /oracle/DB1/redo1
    /oracle/DB1/redo1
    /oracle/DB1/archive

    in a diskgroup called something like DB1-dg

    /oracle/DB2/data1
    /oracle/DB2/data2

    etc.

    Mike

  • H_A_N_B_U's avatar
    H_A_N_B_U
    Level 3
    10 years ago

    Thanks Mike and Riaan,

     

    Thanks for sharing your findings. I'll give you an update regarding the issue.

     

    Paul

  • Sunil_Yadav's avatar
    Sunil_Yadav
    Level 4
    10 years ago

    Gentle reminder! This discussion is open for last 1 month and keeps popping in "Can you solve these?" section. If your query is resolved, then please mark appropriate comments as solution. 

    Thanks & Regards,
    Sunil Y