今天是2014-03-28，继续写rac的内容，今天写的是如何恢复损坏的表决磁盘（votedisk）和ocr。
首先查看目前状态：
[root@rac-two bin]# ./ocrcheckstatus of oracle cluster registry is as follows : version : 3 total space (kbytes) : 262120 used space (kbytes) : 3472 available space (kbytes) : 258648 id : 152256349 device/file name : +gidg device/file integrity check succeeded device/file not configured device/file not configured device/file not configured device/file not configured cluster registry integrity check succeeded logical corruption check succeeded[root@rac-two bin]# ./crsctl query css votedisk## state file universal id file name disk group-- ----- ----------------- --------- --------- 1. online af2fed5d6ac54f1bbf19e4982e364c57 (/dev/asm-diske) [gidg] 2. online 34b65f2e3db14f23bf88e689b55058b7 (/dev/asm-diskd) [gidg] 3. online 07bf3ce9af7d4ff6bf38d160255c671c (/dev/asm-diskc) [gidg]located 3 voting disk(s).[root@rac-two bin]# ./crsctl stat res -t--------------------------------------------------------------------------------name target state server state_details --------------------------------------------------------------------------------local resources--------------------------------------------------------------------------------ora.datadg.dg online online rac-one online online rac-two ora.gidg.dg online online rac-one online online rac-two ora.listener.lsnr online online rac-one online online rac-two ora.testdg.dg offline offline rac-one online offline rac-two ora.asm online online rac-one started online online rac-two started ora.gsd offline offline rac-one offline offline rac-two ora.net1.network online online rac-one online online rac-two ora.ons online online rac-one online online rac-two ora.registry.acfs online online rac-one online online rac-two --------------------------------------------------------------------------------cluster resources--------------------------------------------------------------------------------ora.listener_scan1.lsnr 1 online online rac-two ora.listener_scan2.lsnr 1 online online rac-one ora.listener_scan3.lsnr 1 online online rac-one ora.cvu 1 online online rac-one ora.oc4j 1 online online rac-one ora.rac-one.vip 1 online online rac-one ora.rac-two.vip 1 online online rac-two ora.rac.db 1 online online rac-two open 2 online online rac-one open ora.scan1.vip 1 online online rac-two ora.scan2.vip 1 online online rac-one ora.scan3.vip 1 online online rac-one
模拟votedisk和ocr损坏。
[root@rac-two bin]# dd if=/dev/zero of=/dev/asm-diske bs=1024 count=200;dd if=/dev/zero of=/dev/asm-diskd bs=1024 count=200;dd if=/dev/zero of=/dev/asm-diskc bs=1024 count=200;200+0 records in200+0 records out204800 bytes (205 kb) copied, 0.0884688 s, 2.3 mb/s[root@rac-two bin]# dd if=/dev/zero of=/dev/asm-diskd bs=1024 count=200;200+0 records in200+0 records out204800 bytes (205 kb) copied, 0.138348 s, 1.5 mb/s[root@rac-two bin]# dd if=/dev/zero of=/dev/asm-diskc bs=1024 count=200;200+0 records in200+0 records out204800 bytes (205 kb) copied, 0.103544 s, 2.0 mb/s
开始进行恢复：
停止crs资源。
[root@rac-two bin]# ./crsctl stop crscrs-2791: starting shutdown of oracle high availability services-managed resources on 'rac-two'crs-2673: attempting to stop 'ora.crsd' on 'rac-two'crs-2790: starting shutdown of cluster ready services-managed resources on 'rac-two'crs-2673: attempting to stop 'ora.listener.lsnr' on 'rac-two'crs-2673: attempting to stop 'ora.listener_scan1.lsnr' on 'rac-two'crs-2673: attempting to stop 'ora.gidg.dg' on 'rac-two'crs-2673: attempting to stop 'ora.registry.acfs' on 'rac-two'crs-2673: attempting to stop 'ora.rac.db' on 'rac-two'crs-2677: stop of 'ora.listener.lsnr' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.rac-two.vip' on 'rac-two'crs-2677: stop of 'ora.listener_scan1.lsnr' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.scan1.vip' on 'rac-two'crs-2677: stop of 'ora.rac-two.vip' on 'rac-two' succeededcrs-2672: attempting to start 'ora.rac-two.vip' on 'rac-one'crs-2677: stop of 'ora.registry.acfs' on 'rac-two' succeededcrs-2677: stop of 'ora.scan1.vip' on 'rac-two' succeededcrs-2672: attempting to start 'ora.scan1.vip' on 'rac-one'crs-2677: stop of 'ora.rac.db' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.datadg.dg' on 'rac-two'crs-2677: stop of 'ora.datadg.dg' on 'rac-two' succeededcrs-2676: start of 'ora.rac-two.vip' on 'rac-one' succeededcrs-2676: start of 'ora.scan1.vip' on 'rac-one' succeededcrs-2672: attempting to start 'ora.listener_scan1.lsnr' on 'rac-one'crs-2677: stop of 'ora.gidg.dg' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.asm' on 'rac-two'crs-2677: stop of 'ora.asm' on 'rac-two' succeededcrs-2676: start of 'ora.listener_scan1.lsnr' on 'rac-one' succeededcrs-2673: attempting to stop 'ora.ons' on 'rac-two'crs-2677: stop of 'ora.ons' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.net1.network' on 'rac-two'crs-2677: stop of 'ora.net1.network' on 'rac-two' succeededcrs-2792: shutdown of cluster ready services-managed resources on 'rac-two' has completedcrs-2677: stop of 'ora.crsd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.crf' on 'rac-two'crs-2673: attempting to stop 'ora.ctssd' on 'rac-two'crs-2673: attempting to stop 'ora.evmd' on 'rac-two'crs-2673: attempting to stop 'ora.asm' on 'rac-two'crs-2673: attempting to stop 'ora.drivers.acfs' on 'rac-two'crs-2673: attempting to stop 'ora.mdnsd' on 'rac-two'crs-2677: stop of 'ora.crf' on 'rac-two' succeededcrs-2677: stop of 'ora.evmd' on 'rac-two' succeededcrs-2677: stop of 'ora.mdnsd' on 'rac-two' succeededcrs-2677: stop of 'ora.asm' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'crs-2677: stop of 'ora.ctssd' on 'rac-two' succeededcrs-2677: stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.cssd' on 'rac-two'crs-2677: stop of 'ora.drivers.acfs' on 'rac-two' succeededcrs-2677: stop of 'ora.cssd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.gipcd' on 'rac-two'crs-2677: stop of 'ora.gipcd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.gpnpd' on 'rac-two'crs-2677: stop of 'ora.gpnpd' on 'rac-two' succeededcrs-2793: shutdown of oracle high availability services-managed resources on 'rac-two' has completedcrs-4133: oracle high availability services has been stopped.
启动到独占模式且不启动ora.crsd:
[root@rac-two bin]# ./crsctl start crs -excl -nocrscrs-4123: oracle high availability services has been started.crs-2672: attempting to start 'ora.mdnsd' on 'rac-two'crs-2676: start of 'ora.mdnsd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.gpnpd' on 'rac-two'crs-2676: start of 'ora.gpnpd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.cssdmonitor' on 'rac-two'crs-2672: attempting to start 'ora.gipcd' on 'rac-two'crs-2676: start of 'ora.cssdmonitor' on 'rac-two' succeededcrs-2676: start of 'ora.gipcd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.cssd' on 'rac-two'crs-2672: attempting to start 'ora.diskmon' on 'rac-two'crs-2676: start of 'ora.diskmon' on 'rac-two' succeededcrs-2676: start of 'ora.cssd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.drivers.acfs' on 'rac-two'crs-2679: attempting to clean 'ora.cluster_interconnect.haip' on 'rac-two'crs-2672: attempting to start 'ora.ctssd' on 'rac-two'crs-2681: clean of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2672: attempting to start 'ora.cluster_interconnect.haip' on 'rac-two'crs-2676: start of 'ora.drivers.acfs' on 'rac-two' succeededcrs-2676: start of 'ora.ctssd' on 'rac-two' succeededcrs-2676: start of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2672: attempting to start 'ora.asm' on 'rac-two'crs-2676: start of 'ora.asm' on 'rac-two' succeeded
创建磁盘组：
[root@rac-two bin]# su - grid[grid@rac-two ~]$ sqlplus / as sysasmsql*plus: release 11.2.0.4.0 production on fri mar 28 00:01:05 2014copyright (c) 1982, 2013, oracle. all rights reserved.connected to:oracle database 11g enterprise edition release 11.2.0.4.0 - 64bit productionwith the real application clusters and automatic storage management optionssql> create diskgroup gidg normal redundancy 2 disk '/dev/asm-diske','/dev/asm-diskd','/dev/asm-diskc' attribute 'au_size'='1m','compatible.asm'='11.2','compatible.rdbms'='11.2','compatible.advm'='11.2';diskgroup created.sql> select name,state from v$asm_diskgroup;name state------------------------------ -----------datadg dismountedtestdg dismountedgidg mountedsql> exitdisconnected from oracle database 11g enterprise edition release 11.2.0.4.0 - 64bit productionwith the real application clusters and automatic storage management options[grid@rac-two ~]$ exit
首先要恢复的是ocr
[root@rac-two bin]# ./ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-two-cluster/backup00.ocr[root@rac-two bin]# ./ocrcheckstatus of oracle cluster registry is as follows : version : 3 total space (kbytes) : 262120 used space (kbytes) : 3472 available space (kbytes) : 258648 id : 152256349 device/file name : +gidg device/file integrity check succeeded device/file not configured device/file not configured device/file not configured device/file not configured cluster registry integrity check succeeded logical corruption check succeeded
恢复votedisk：
[root@rac-two bin]# ./crsctl query css votedisklocated 0 voting disk(s).[root@rac-two bin]# ./crsctl replace votedisk +gidgcrs-4602: failed 27 to add voting file 58c5df27b4884f70bfb38fc4c93b8f72.crs-4602: failed 27 to add voting file 51caa3f834964f5abf6efcfb90476d85.crs-4602: failed 27 to add voting file 92b70ca3ea5d4f80bf9df66a8e975ba9.failed to replace voting disk group with +gidg.crs-4000: command replace failed, or completed with errors.
提示失败，需要重启crs资源。
[root@rac-two bin]# ./crsctl stop crscrs-2791: starting shutdown of oracle high availability services-managed resources on 'rac-two'crs-2673: attempting to stop 'ora.ctssd' on 'rac-two'crs-2673: attempting to stop 'ora.asm' on 'rac-two'crs-2673: attempting to stop 'ora.drivers.acfs' on 'rac-two'crs-2673: attempting to stop 'ora.mdnsd' on 'rac-two'crs-2677: stop of 'ora.mdnsd' on 'rac-two' succeededcrs-2677: stop of 'ora.asm' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'crs-2677: stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2677: stop of 'ora.drivers.acfs' on 'rac-two' succeededcrs-2677: stop of 'ora.ctssd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.cssd' on 'rac-two'crs-2677: stop of 'ora.cssd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.gipcd' on 'rac-two'crs-2677: stop of 'ora.gipcd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.gpnpd' on 'rac-two'crs-2677: stop of 'ora.gpnpd' on 'rac-two' succeededcrs-2793: shutdown of oracle high availability services-managed resources on 'rac-two' has completedcrs-4133: oracle high availability services has been stopped.[root@rac-two bin]# ./crsctl start crs -excl -nocrscrs-4123: oracle high availability services has been started.crs-2672: attempting to start 'ora.mdnsd' on 'rac-two'crs-2676: start of 'ora.mdnsd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.gpnpd' on 'rac-two'crs-2676: start of 'ora.gpnpd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.cssdmonitor' on 'rac-two'crs-2672: attempting to start 'ora.gipcd' on 'rac-two'crs-2676: start of 'ora.cssdmonitor' on 'rac-two' succeededcrs-2676: start of 'ora.gipcd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.cssd' on 'rac-two'crs-2672: attempting to start 'ora.diskmon' on 'rac-two'crs-2676: start of 'ora.diskmon' on 'rac-two' succeededcrs-2676: start of 'ora.cssd' on 'rac-two' succeededcrs-2672: attempting to start 'ora.drivers.acfs' on 'rac-two'crs-2679: attempting to clean 'ora.cluster_interconnect.haip' on 'rac-two'crs-2672: attempting to start 'ora.ctssd' on 'rac-two'crs-2681: clean of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2672: attempting to start 'ora.cluster_interconnect.haip' on 'rac-two'crs-2676: start of 'ora.drivers.acfs' on 'rac-two' succeededcrs-2676: start of 'ora.ctssd' on 'rac-two' succeededcrs-2676: start of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2672: attempting to start 'ora.asm' on 'rac-two'crs-2676: start of 'ora.asm' on 'rac-two' succeeded[root@rac-two bin]# su - grid[grid@rac-two ~]$ sqlplus / as sysasmsql*plus: release 11.2.0.4.0 production on fri mar 28 00:13:30 2014copyright (c) 1982, 2013, oracle. all rights reserved.connected to:oracle database 11g enterprise edition release 11.2.0.4.0 - 64bit productionwith the real application clusters and automatic storage management optionssql> col name for a30sql> col path for a50sql> set linesize 200sql> select name,path from v$asm_disk;no rows selectedsql> alter system set asm_diskstring='/dev/asm-disk*';system altered.sql> select name,path from v$asm_disk;name path------------------------------ -------------------------------------------------- /dev/asm-diskg /dev/asm-diskd /dev/asm-diskc /dev/asm-diskf /dev/asm-diskh /dev/asm-diskm /dev/asm-diskk /dev/asm-diskj /dev/asm-diski /dev/asm-diskb /dev/asm-diskename path------------------------------ -------------------------------------------------- /dev/asm-diskl12 rows selected.sql> select name,state from v$asm_diskgroup;name state------------------------------ -----------datadg dismountedgidg dismountedtestdg dismountedsql> alter diskgroup gidg mount;diskgroup altered.sql> select name,state from v$asm_diskgroup;name state------------------------------ -----------datadg dismountedgidg mountedtestdg dismountedsql> exitdisconnected from oracle database 11g enterprise edition release 11.2.0.4.0 - 64bit productionwith the real application clusters and automatic storage management options[grid@rac-two ~]$ exitlogout[root@rac-two bin]# ./ocrctl replace votedisk +gidg-bash: ./ocrctl: no such file or directory[root@rac-two bin]# ./crsctl replace votedisk +gidgsuccessful addition of voting disk 4b55dbd068344fb6bfb89250b3586256.successful addition of voting disk 6b157dc173bd4f7ebf1846df01c23d5d.successful addition of voting disk 505d811a77d64f57bfa2797afba6a5f5.successfully replaced voting disk group with +gidg.crs-4266: voting file(s) successfully replaced
至此完成了ocr和votedisk的恢复。
重启crs，查看状况：
[root@rac-two bin]# ./crsctl stop crscrs-2791: starting shutdown of oracle high availability services-managed resources on 'rac-two'crs-2673: attempting to stop 'ora.mdnsd' on 'rac-two'crs-2673: attempting to stop 'ora.drivers.acfs' on 'rac-two'crs-2673: attempting to stop 'ora.ctssd' on 'rac-two'crs-2673: attempting to stop 'ora.asm' on 'rac-two'crs-2677: stop of 'ora.mdnsd' on 'rac-two' succeededcrs-2677: stop of 'ora.asm' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'crs-2677: stop of 'ora.ctssd' on 'rac-two' succeededcrs-2677: stop of 'ora.drivers.acfs' on 'rac-two' succeededcrs-2677: stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.cssd' on 'rac-two'crs-2677: stop of 'ora.cssd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.gipcd' on 'rac-two'crs-2677: stop of 'ora.gipcd' on 'rac-two' succeededcrs-2673: attempting to stop 'ora.gpnpd' on 'rac-two'crs-2677: stop of 'ora.gpnpd' on 'rac-two' succeededcrs-2793: shutdown of oracle high availability services-managed resources on 'rac-two' has completedcrs-4133: oracle high availability services has been stopped.[root@rac-two bin]# ./crsctl start crscrs-4123: oracle high availability services has been started.[grid@rac-two ~]$ crsctl stat res -t--------------------------------------------------------------------------------name target state server state_details --------------------------------------------------------------------------------local resources--------------------------------------------------------------------------------ora.datadg.dg online online rac-two ora.gidg.dg online online rac-two ora.listener.lsnr online online rac-two ora.testdg.dg online offline rac-two ora.asm online online rac-two started ora.gsd offline offline rac-two ora.net1.network online online rac-two ora.ons online online rac-two ora.registry.acfs online online rac-two --------------------------------------------------------------------------------cluster resources--------------------------------------------------------------------------------ora.listener_scan1.lsnr 1 online online rac-two ora.listener_scan2.lsnr 1 online online rac-two ora.listener_scan3.lsnr 1 online online rac-two ora.cvu 1 online online rac-two ora.oc4j 1 online online rac-two ora.rac-one.vip 1 online intermediate rac-two failed over ora.rac-two.vip 1 online online rac-two ora.rac.db 1 online online rac-two open 2 online offline ora.scan1.vip 1 online online rac-two ora.scan2.vip 1 online online rac-two ora.scan3.vip 1 online online rac-two [grid@rac-two ~]$
在节点一重启crs资源，将其加入集群。
[root@rac-one bin]# ./crsctl start crscrs-4123: oracle high availability services has been started.[root@rac-one bin]# ./crsctl stat res -t--------------------------------------------------------------------------------name target state server state_details --------------------------------------------------------------------------------local resources--------------------------------------------------------------------------------ora.datadg.dg online online rac-one online online rac-two ora.gidg.dg online online rac-one online online rac-two ora.listener.lsnr online online rac-one online online rac-two ora.testdg.dg offline offline rac-one online offline rac-two ora.asm online online rac-one started online online rac-two started ora.gsd offline offline rac-one offline offline rac-two ora.net1.network online online rac-one online online rac-two ora.ons online online rac-one online online rac-two ora.registry.acfs online online rac-one online online rac-two --------------------------------------------------------------------------------cluster resources--------------------------------------------------------------------------------ora.listener_scan1.lsnr 1 online online rac-one ora.listener_scan2.lsnr 1 online online rac-two ora.listener_scan3.lsnr 1 online online rac-two ora.cvu 1 online online rac-two ora.oc4j 1 online online rac-two ora.rac-one.vip 1 online online rac-one ora.rac-two.vip 1 online online rac-two ora.rac.db 1 online online rac-two open 2 online online rac-one open ora.scan1.vip 1 online online rac-one ora.scan2.vip 1 online online rac-two ora.scan3.vip 1 online online rac-two [root@rac-one bin]#
至此完成了ocr和votedisk的恢复。
that‘s all！！！！！