1
  • Because of failed backups, the disk used for transaction logs appears to be full.
  • The database is not accepting connections, so I cannot run the (HP Data Protector) backup job manually.
  • The Oracle instance directly mounts disk from a SAN, so there isn't any file system visible to Windows.

The database service is running, however. Is there a simple way to recover the database, maybe to the point that the backup job can be re-run and restore service?

Assume no Oracle knowledge beyond tnsping ;-)

From the alert_mydatabase.log file, (<orahome>\admin\mydatabase\bdump\), 

ORA-15041: diskgroup space exhausted 
...
ARCH: Archival stopped, error occurred. Will continue retrying
...
U. Windl
  • 366
  • 3
  • 17
nray
  • 1,540
  • 17
  • 23

3 Answers3

1

Some random thoughts:

  1. Assuming you've got an Oracle Support contract, I'd get them involved. Not necessarily because this is an unusual or particularly difficult problem; but as much because of your unfamiliarity with Oracle. Get something wrong and you could well screw up your db.

  2. Why are you thinking a restore is necessary? When the disk gets full, writes to the db are "frozen" (as you're seeing), but there should be no data loss/corruption.

  3. Do you have your db sys password?

  4. Is this a live/production system?

  5. Are there any other ways of accessing the SAN filesystem?

Repond to these questions and we can take it from there.

cagcowboy
  • 1,072
  • 1
  • 14
  • 21
1

ORA-15041: diskgroup space exhausted

Seems like you're using ASM to store your archivelogs/backups and you have probably filled it up (probably because old backups are not getting purged)

As cagcowboy has told you you're database is probably hung because it cannot archive and switch its current logfile but does not seem you have to recover/restore or something. Once you are able to free up some space the instance will continue its activity.

As already suggested, I should try to get in touch with Oracle support to get proper advise.

Regards.

scetoaux
  • 1,289
  • 2
  • 12
  • 26
  • You're all correct, but you get the rep 'cause you have the least to start with ;-) – nray Jul 14 '09 at 16:52
1

Here are more random thoughts:

1) Where is the dba?

2) Since the database volumes are not viewable in windows, I assume the database is either using raw partitions or ASM. Do you know which one? And if not asm or raw, then what filesystem is the underlying data disk?

3) when you backup the database, how do you handle removal/archiving of the archive logs?

4) if not using raw/asm and using filesystem, then see if the san volume for the archive log can be extended/grow in size to allow for archive to proceed. Then immediately get do a backup and remove/archive the older archive logs.

5) You can also try creating a new volume for archive log, get into the database and alter the db to archive to the new archive log destination.

As far as your archive log, it appears the db has ground to halt waiting for space but has not crashed. If you make space it'll continue. Your best bet is to get a dba and a san administrator together and determine what can be done to make space. Call oracle support for help if you get stuck.

Michael_N
  • 81
  • 1
  • The filesystem is ASM, I believe. The DBA was able to remove older log files, free space and then start the database (although it should have started itsself, once space was available). Then the backup job could run, removing the logs and solving the problem. – nray Jul 14 '09 at 16:51
  • What I really wanted to know was, in the absence of a DBA, is there any way to view the "ASM" filesystem, determine if there are files which can be moved/removed and then remove them? – nray Jul 14 '09 at 16:57
  • The asm instance can be access like any other oracle instance. There are a number of asm views and commands that can be used to administer the instance and its disks. http://www.oracle-base.com/articles/10g/AutomaticStorageManagement10g.php has a good preview. Oracle manuals on asm can be downloaded from OTN for more reading. I would suggest getting very familiar with ASM before getting into the asm instance and making changes. Borking the ASM instance will result in the corruption and destruction of the database it houses. – Michael_N Jul 16 '09 at 19:49