前几天某客户的一个核心数据库，大约24TB的rac，asm diskgroup无法mount。经过分析发现是某个disk的一个block坏掉了。其中通过kfed read读取发现确实有问题：
$ kfed read /dev/rdisk/disk392 aun=0 blkn=2 | more
kfbh.endian: 76 ; 0x000: 0x4c
kfbh.hard: 86 ; 0x001: 0x56
kfbh.type: 77 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 82 ; 0x003: 0x52
kfbh.block.blk: 1162031153 ; 0x004: blk=1162031153
kfbh.block.obj: 620095014 ; 0x008: file=386598
kfbh.check: 1426510413 ; 0x00c: 0x5506d24d
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 524288639 ; 0x018: 0x1f40027f
kfbh.spare2: 0 ; 0x01c: 0x00000000
60000000000F3200 4C564D52 45433031 24F5E626 5506D24D [LVMREC01$..&U..M]
60000000000F3210 00000000 00000000 1F40027F 00000000 [.........@......]
我们通过手工构造了一个block，然后进行merge修复之后，我尝试mount diskgroup，发现还是报错，如下：
Fri Oct 28 04:47:56 2016
WARNING: cache read a corrupt block: group=3(DATA) dsk=49 blk=18 disk=49 (DATA_0049) incarn=3636812057 au=0 blk=18 count=1
Errors in file /oracle/ora11g/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_21799.trc:
ORA-15196: invalid ASM block header [kfc.c:26076] [endian_kfbh] [2147483697] [18] [76 != 0]
NOTE: a corrupted block from group DATA was dumped to /oracle/ora11g/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_21799.trc
WARNING: cache read (retry) a corrupt block: group=3(DATA) dsk=49 blk=18 disk=49 (DATA_0049) incarn=3636812057 au=0 blk=18 count=1
Errors in file /oracle/ora11g/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_21799.trc:
ORA-15196: invalid ASM block header [kfc.c:26076] [endian_kfbh] [2147483697] [18] [76 != 0]
ORA-15196: invalid ASM block header [kfc.c:26076] [endian_kfbh] [2147483697] [18] [76 != 0]
ERROR: cache failed to read group=3(DATA) dsk=49 blk=18 from disk(s): 49(DATA_0049)
ORA-15196: invalid ASM block header [kfc.c:26076] [endian_kfbh] [2147483697] [18] [76 != 0]
ORA-15196: invalid ASM block header [kfc.c:26076] [endian_kfbh] [2147483697] [18] [76 != 0]
NOTE: cache initiating offline of disk 49 group DATA
NOTE: process _arb0_+asm1 (21799) initiating offline of disk 49.3636812057 (DATA_0049) with mask 0x7e in group 3
WARNING: Disk 49 (DATA_0049) in group 3 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 3, dsk = 49/0xd8c55919, mask = 0x6a, op = clear
Fri Oct 28 04:47:56 2016
GMON updating disk modes for group 3 at 23 for pid 25, osid 21799
ERROR: Disk 49 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 3)
WARNING: Offline of disk 49 (DATA_0049) in group 3 and mode 0x7f failed on ASM inst 1
Fri Oct 28 04:47:56 2016
NOTE: halting all I/Os to diskgroup 3 (DATA)
Fri Oct 28 04:47:56 2016
NOTE: cache dismounting (not clean) group 3/0x51B5A89F (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 23376, image: oracle@cqracdb1 (B000)
Fri Oct 28 04:47:56 2016
ERROR: ORA-15130 in COD recovery for diskgroup 3/0x51b5a89f (DATA)
ERROR: ORA-15130 thrown in RBAL for group number 3
Errors in file /oracle/ora11g/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_6465.trc:
ORA-15130: diskgroup "DATA" is being dismounted
不难看出，这个错误比较熟悉了。从上述日志来看，第49号disk的第0号AU的第18号block还是有问题，通过kfed读取发现确实也是坏块。跟前面第2号block 一样。这里如法炮制，仍然构造一样的block，然后merge之后，成功将diskgroup mount了。
SQL> alter system set asm_power_limit=0 scope=both;
System altered.
SQL> alter diskgroup data mount;
Diskgroup altered.
将diskgroup mount之后，我检查数据库，发现crs自动将数据库拉起来了，并且已经open了。然而后续进一步检查发现asm的arb进程仍然在报错：
ARB0 relocating file +DATA.278.794162479 (120 entries)
DDE: Problem Key "ORA 600 [kfdAuDealloc2]" was flood controlled (0x2) (incident: 1486148)
ORA-00600: internal error code, arguments: [kfdAuDealloc2],85], [278], [14309 [], [], [], [], [], [], [], [], []
OSM metadata struct dump of kfdatb:
kfdatb.aunum: 7168 ; 0x000: 0x00001c00
kfdatb.shrink: 448 ; 0x004: 0x01c0
kfdatb.ub2pad: 7176 ; 0x006: 0x1c08
kfdatb.auinfo[0].link.next: 8 ; 0x008: 0x0008
kfdatb.auinfo[0].link.prev: 8 ; 0x00a: 0x0008
kfdatb.auinfo[1].link.next: 12 ; 0x00c: 0x000c
kfdatb.auinfo[1].link.prev: 12 ; 0x00e: 0x000c
kfdatb.auinfo[2].link.next: 16 ; 0x010: 0x0010
kfdatb.auinfo[2].link.prev: 16 ; 0x012: 0x0010
kfdatb.auinfo[3].link.next: 20 ; 0x014: 0x0014
kfdatb.auinfo[3].link.prev: 20 ; 0x016: 0x0014
kfdatb.auinfo[4].link.next: 24 ; 0x018: 0x0018
kfdatb.auinfo[4].link.prev: 24 ; 0x01a: 0x0018
kfdatb.auinfo[5].link.next: 28 ; 0x01c: 0x001c
kfdatb.auinfo[5].link.prev: 28 ; 0x01e: 0x001c
kfdatb.auinfo[6].link.next: 32 ; 0x020: 0x0020
kfdatb.auinfo[6].link.prev: 32 ; 0x022: 0x0020
kfdatb.spare: 0 ; 0x024: 0x00000000
Dump of ate#:0
OSM metadata struct dump of kfdate:
kfdate.discriminator: 1 ; 0x000: 0x00000001
kfdate.allo.lo: 0 ; 0x000: XNUM=0x0
kfdate.allo.hi: 8388608 ; 0x004: V=1 I=0 H=0 FNUM=0x0
虽然已经不影响数据库的正常运行，然而由于arb进程的异常，导致reblance操作实际上没有进行完成，客户新加的磁盘基本上没有被使用，导致diskgroup的磁盘使用不均衡。
这个错误看上去很复杂，实际上很简单。根据后面的序号，我们可以判断，本质来讲是因为我们前面手工构造的2个block其实并不完整，这是allocate table，需要将后面的kfdate数据都构造完毕，才能让arb进程正常工作下午。
不过熊爷已经在修改odu代码了，准备odu来搞定这个遗留问题。看来ODU以后将具备修复ASM元数据的功能了。很强悍！
recovery|Recover Case: 一个24TB的rac(asm)恢复案例
http://m.bbyears.com/banzhurengongzuo/114172.html
推荐访问:recovery模式 recovery是什么意思 recovery下载 recovery数据恢复 recovering recover4all