Bugs with rm -rf from within asmcmd

Performing an rm -rf operation is normally a simple operation albeit risky if you are in the wrong folder. However within ASMCMD there are a couple of bugs associated with it.

Here are two examples and workrounds

FRA full - warning in the alert log  - action needs to be taken - this is an 11.2.0.4 database on OEL (5.8)

ORA-19815: WARNING: db_recovery_file_dest_size of 10484711424 bytes is 100.00% used, and has 0 remaining bytes available.

 ARCH: Error 19809 Creating archive log file to '+FRA'

Errors in file /app/oracle/diag/rdbms/tst11204/TST11204/trace/TST11204_ora_13209.trc:

ORA-16038: log 1 sequence# 148 cannot be archived

ORA-19809: limit exceeded for recovery files

ORA-00312: online log 1 thread 1: '+DATA/tst11204/redo01.log'

USER (ospid: 13209): terminating the instance due to error 16038

Let’s look at the disk from asmcmd and we see a lot of old folders that we do not need

ASMCMD [+FRA/TST11204/ARCHIVELOG] > ls

2013_12_08/

2013_12_09/

2013_12_10/

2013_12_11/

2013_12_12/

2013_12_13/

2013_12_14/

2013_12_15/

2013_12_16/

2013_12_17/

2013_12_18/

2013_12_19/

2013_12_20/

2013_12_21/

2013_12_22/

2013_12_23/

2013_12_24/

Lets remove them all

 rm -r 2013*

You may delete multiple files and/or directories.

Are you sure? (y/n) y

ORA-29261: bad argument

ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [KFCHK_ERRRET], [kfa.c], [3025], [], [], [], [], [], [], [], [], []

], [], [], [], [], [], [], [], [], []

ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)

rm -r 2013_12_08

You may delete multiple files and/or directories.

Are you sure? (y/n) y

ORA-29261: bad argument

ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [KFCHK_ERRRET], [kfa.c], [3025], [], [], [], [], [], [], [], [], []

], [], [], [], [], [], [], [], [], []

ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)

We have hit a known bug – only applies to 11.2.0.4 but I am sure I have seen manifestations of this issue on previous version of 11GR2

Bug 16892396  ORA-600 [KFCHK_ERRRET] [kfa.c] from ASMCMD “rm -rf” operation

11.2.0.4

fixed:

Interim patches may be available for earlier versions – click here to check.

Description

This bug is only relevant when using Automatic Storage Management (ASM)

Running the ASMCMD rm -rf command interleaved with the RDBMS file activities can trigger an ORA-600 [KFCHK_ERRRET] [kfa.c] error.Note:  If you run ‘rm -rf’ on a directory not accessed by an RDBMS instance, you  should not hit the issue.

I got round it using probably the best way of deleting  archive logs, through rman

Startup mount

rman target / catalog

 DELETE ARCHIVELOG UNTIL TIME 'sysdate -10';

DELETE EXPIRED ARCHIVELOG ALL;

CROSSCHECK ARCHIVELOG ALL;

The second bug is

Bug 15994107  ORA-600 [kfcDeadlockAvoid01] from “asmcmd rm -rf” if fix for bug 15900104 present

 This note gives a brief overview of bug 15994107. 
  The content was last updated on: 28-JUN-2013
  Click here for details of each of the sections below.

Affects:

Product (Component)	Oracle Server (Rdbms)
Range of versions believed to be affected	Versions BELOW 12.2
Versions confirmed as being affected	(None Specified)
Platforms affected	Generic (all / most platforms affected)

Fixed:

The fix for 15994107 is first included in

11.2.0.4 (Server Patch Set)

Interim patches may be available for earlier versions – click here to check.

Description

This bug is only relevant when using Automatic Storage Management (ASM)ORA 600 [kfcDeadlockAvoid01] can occur during “asmcmd rm -rf” if the fix for bug 15900104 is present. Rediscovery Notes If ORA 600 [kfcDeadlockAvoid01] is encountered during asmcmd rm -rf AND the fix of bug 15900104 is present then you are likely hitting this issue. Workaround None Note: For interim patches please use this fix instead of 15900104 to address both issues.

This fix has been superseded – please see the fixed version information for Bug:15994107 . The box below only shows versions where the code change/s for 15900104 are first included – those versions may not contain the later improved fix.

The fix for 15900104 referred to is actually then superceded by the original bug I started with.  So the moral of the story is not to use rm -rf wherever possible and try and remove files from rman – assuming they are database related files of course

The fix for 15900104 is first included in

12.1.0.1 (Base Release) 11.2.0.4 (Server Patch Set)