1.环境描述
os：aix6.1
oracle ：11.2.0.3.0 rac
2.事故发生
数据库node 2所在小型机发生宕机事故，，本应正常切换至node1，但切换失败，重启系统得以解决。
3.事故分析
事故第二天进行了数据库的alert log分析，从日志中可以看到数据库在实例node2发生宕机后，rac已经做出了实例切换步骤，但在切换的过程中遭遇了ora-00240、ora-29770错误，导致当时数据库没有切换成功。下面是日志的详细分析。
数据库在当时经历了大致以下几个重要步骤：
1. beginning instance recovery of 1 threads
数据库开始在本地恢复宕机的实例
2. started redo application at
thread 2: logseq 4556, block 368380
数据库开始从在线redo日志序号为4556的日志恢复
3. completed instance recovery at
thread 2: logseq 4556, block 376983, scn 3502123313
数据库redo日志4556已经恢复成功。
4. redo thread 2 internally disabled at seq 4557 (smon)
当数据库准备恢复4557的日志时，遭遇失败。
5 ora-00240: control file enqueue held for more than 120 seconds
日志中开始出现ora-00240错误，提示控制文件被持有超过120秒。
6 ora-29770: global enqueue process dia0 (osid 12517556) is hung for more than 300 seconds
incident details in:
紧接着出现ora29770错误，数据库进程dia0 hung住超过5分钟。
在当天实例宕机的20分钟左右时间内，当时系统的监控等脚本均无法执行，分析oracle awr报告得知在当天宕机的时间内，未宕机的node1系统cpu资源几乎耗尽。
经过分析与推测，dia0进程的主要作用是处理数据库死锁、hung住的一个进程，日志中的现象应该是这个进程发现控制文件被持有超过120秒，属于错误行为，进程开始处理这一问题，但当时系统cpu资源耗尽，进程dia0解决故障超过5分钟，导致出现了ora-29770错误。
所以认定这次实例切换失败的主要原因在于ora-00240错误。根据日志中给出的trace文件，查看trace文件中的具体报错原因。
从trace日志中分析得知，当时控制文件被持有超过120秒的主要原因是ksv master wait等待，ksv master wait耗费了2分03秒。
4.事故分析结论
根据以上现象及日志体现，从oracle官方metalink中查找资料得知这是一个bug [bug id 1308282.1]
下面是官方metalink文档中关于这个bug的解释：
high 'ksv master wait' and 'asm file metadata operation' waits in non-exadata 11g
symptoms
high waits for 'ksv master wait' while doing an asm file metadata operation were reported when a data migration utility was running. this wait was also seen for a drop of a tablespace.
the awr showed the top events were cpu (> 100%), with 'asm file metadata operation' (7%).
cause
event 'ksv master wait' indicates the process on the rdbms side is waiting for a reply from a process on the asm side. in 11g, the parameter cell_offload_processing is set to true. although that is a parameter is not applicable for non-exadata databases, it caused asm to try to deliver smart-scan results. the issue was reported in bug 11800170 - asm in ksv wait after application of 11.2.0.2 grid psu.
after applying the workaround for this issue (see solution below), a drop of a tablespace that used to take 13 minutes took 4 seconds.
solution
the following solutions are available for non-exadata databases:
for the quickest solution, use the workaround. the workaround does not negatively impact non-exadata databases. this parameter is to be set on the database instance.
alter system set cell_offload_processing = false;
upgrade to 12.1, when available. or
apply the 11.2.0.3 patch set or
apply one-off patch 11800170, if available for your rdbms and grid homes
note: at the time this note was written (march 2011), neither 12.1 nor 11.2.0.3 were available.
官方文档中给出的最快解决方式是修改oracle中的一个参数cell_offload_processing，修改为false。