4

The basic question:

How long should fsck take to fix a 100GB (17 million block) file with multiply-claimed blocks?

The long version of the question:

After a UPS failure, I am faced with an Ubuntu 10.04 server which dropped into fsck on initial boot. This is normal, buy usually about half an hour of fixing the various problems by agreeing to the prompts is enough to get the server back.

Not today, though. Today, I got a huge list of numbers scroll past the console matrix-style for a good few minutes. It was basically line after line of:

Multiply-claimed blocks in inode xxxxxxxxx

Anyway, after a few minutes of those scrolling past, it finally settled down and I got:

Pass 1C: Scanning directories for inodes with multiply-claimed blocks

followed by...

Pass 1D: Reconciling multiply-claimed blocks

..and..

(There are 32 inodes containing multiply-claimed blocks.)

That didn't sound so bad, but then it started going through some files as so:

File /path/to/a/file

has 1 multiply-claimed block(s) shared with 1 file(s):

/path/to/another/file

Clone multiply-claimed blocks? yes

This question was answered for me and the process continued. However, it took a very very long time. Hours and hours even though it was only a 2MB file.

After that, a similar dialogue appeared but this time for a virtual machine image file which is 100GB and reported as being over 17 Million multiple-claimed blocks, shared with 0 file(s).

That was 2 days ago and it's still running now.

So, back to my original question, how long should this take? Is it a lost cause and are there any alternative ways to deal with this? What I really don't understand is why the 100GB file is reported as being shared with 0 files which is a contradiction if I understand the meaning of multiply-claimed blocks correctly.

Joseph
  • 277
  • 1
  • 2
  • 11

5 Answers5

5

It seems as thought the cause for the large amount of time and also the mystery of the multiply-claimed blocks being shared by zero files was the result of a degraded RAID array.

As soon as I removed the faulty drive, the fsck went much faster. There were still a few multiply-claimed blocks but they were fixed very quickly.

I have experienced degraded RAID arrays in Ubuntu before and usually there is a warning just after the grub phase, but this did not happen in this case.

Joseph
  • 277
  • 1
  • 2
  • 11
  • It's been a while, but can you elaborate? A functioning or degraded array should be transparent to the layers above. Having the drive still in should not cause errors. – Halfgaar Sep 11 '15 at 07:38
4

This happens to me on RAID array of 6 disks, 4.5TB ext4 filesystem. Linux 3.3.7-1-ARCH #1 SMP PREEMPT i686

I use rsync to sync over entire servers onto the ext4 and these are the files I mostly get these multiply-claimed-blocks and duplicate inode messages on.

EXT4 Mount Options

A couple things I did that seemed to help was to make sure that the ext4 was being mounted with barrier and data=ordered support. 

/dev/md5 /md5  ext4  defaults,noatime,nouser_xattr,stripe=1536,data=ordered  0 2

RAID Bitmap

The other step I took was to enable bitmap on the RAID. 

mdadm --grow /dev/md5 --bitmap=internal

or

mdadm --grow /dev/md5 --bitmap=/external/md5.bitmap

Having both the raid bitmap and ext4 journal located on an external device seems to work the best.

USB Autosuspend

In the past I had experienced this issue when my drives would go into autosuspend mode. Writing to them (or attempting to) while they were trying to wake up from a suspended state seemed to cause these issues big time. What I did was disable autosuspend compeletely on usb devices with: 

usbcore.autosuspend=-1

EXT4 Data Modes

From: http://kernel.org/doc/Documentation/filesystems/ext4.txt

There are 3 different data modes:

  • writeback mode In data=writeback mode, ext4 does not journal data at all. This mode provides a similar level of journaling as that of XFS, JFS, and ReiserFS in its default mode - metadata journaling. A crash+recovery can cause incorrect data to appear in files which were written shortly before the crash. This mode will typically provide the best ext4 performance.

  • ordered mode In data=ordered mode, ext4 only officially journals metadata, but it logically groups metadata information related to data changes with the data blocks into a single unit called a transaction. When it's time to write the new metadata out to disk, the associated data blocks are written first. In general, this mode performs slightly slower than writeback but significantly faster than journal > mode.

  • journal mode data=journal mode provides full data and metadata journaling. All new data is written to the journal first, and then to its final location. In the event of a crash, the journal can be replayed, bringing both data and metadata into a consistent state. This mode is the slowest except when data needs to be read from and written to disk at the same time where it outperforms all others modes. Currently ext4 does not have delayed allocation support if this data journalling mode is selected.

Fix with Debugfs

This has great examples to fix: http://www.redhat.com/archives/ext3-users/2009-February/msg00021.html

AskApache Htaccess
  • 401
  • 4
  • 6
3

How long it takes would depend on disk subsystem performance, the damage being repaired, etc.

It sounds like there's some decent filesystem corruption. How big is the actual filesystem? You said it's a 100 GB file and later that it's a VM image? Is this a VM server? Or are you talking about virtualbox?

Personally if it took over a day and the damage was definitely to one file, I'd restore the file from backup and if there were any indications of continuing issues, reformat and restore from backup, assuming the drive isn't coincidentally failing. I have trust issues with filesystems that start to go bad. If the drive itself isn't failing, the filesystem may have a pervasive problem until it's started fresh.

But that's me.

Bart Silverstrim
  • 31,172
  • 9
  • 67
  • 87
  • To answer your questions, it's a VM server running KVM. The filesystem is 2TB but is mostly empty. There's just the OS and the 100GB VM image. – Joseph Mar 07 '12 at 01:40
  • I'd probably work on the reformat/restore idea myself, because it shouldn't have taken this long. But it might recover...hrm...I'd still be very wary of it. – Bart Silverstrim Mar 07 '12 at 02:09
  • With VM's you run the *chance* of having multiple filesystems fudged; the host filesystem, the VM's filesystem, and the host's filesystem's error "bleeding" into the VM. I wouldn't be comfortable with it. – Bart Silverstrim Mar 07 '12 at 02:10
1

I think, I had a similar problam. I have 2 HDD in a RAID0 array. Once, I did a fsck manually, after I unmounted the device. To my pain, I didn't realize, that a VM was still running and accessing the device during I fsck'ed. The result was lots of multiply claimed blocks and because I rebooted my server during the progress, I think it broke the superblock. So I couldn't even mount the RAID anymore.

I fixed the issue by restoring the superblock, once again by running fsck and repairing all issues which didn't have anything to do with "multiply claimed blocks". This took me some time and I needed to attend the process, to tell fsck not to repair the "multiply claimed blocks".

After that, the superblock was fixed and I could mount the device once again. Now, I ran fsck several times and checked, which files were affected by "multiply claimed blocks", stopped the process by hitting ctrl^c and simply copied the affected files and deleted the original once.

Sounds unconventional, but it fixed my problems in a quick way and my HDDs seem to be clean (according to e2fsck).

If there was a better/faster way to fix these to problems, I am glad to hear about them.

longneck
  • 23,082
  • 4
  • 52
  • 86
T_Torture
  • 11
  • 1
0

Are you using ext2 or ext4 without a journal? You shouldn't ever see this kind of error with a journal.

Yes, it makes no sense to have multiply claimed blocks that are shared by zero files. You should report this bug on the linux-ext4@vger.kernel.org mailing list.

psusi
  • 3,347
  • 1
  • 17
  • 9
  • I'm using ext4 exactly as the default installation of Ubuntu Server 10.04 selected. Thanks for confirming that the zero files thing makes no sense. But is it possible that the error could just be caused by faulty hardware rather than a software bug? – Joseph Mar 07 '12 at 01:41
  • @Joseph, could be. These errors certainly should not be happening when using the journal. You aren't hibernating the system and then booting another system that may be trying to mount the disk while the main system is still in hibernation are you? – psusi Mar 07 '12 at 04:04
  • no, definitely not doing anything like that – Joseph Mar 07 '12 at 08:46