Cannot mount XFS - bad superblock

Question

Please bare in mind I am very noob and have not much idea what I am doing. We have a Dell PowerEdge 420 with a 716iS iSCSI raid. The network connect dropped in the middle of disk usage and we are no longer able to mount. I tried xfs_check /dev/sdf

Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
.....<> .....
found candidate secondary superblock
unable to verify superblock continuing...
.....<> .....

But it is never able to find a valid superblock. Similarly:

xfs_repair -n /dev/sdf
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...

This does not find a valid superblock.

What can I do to recover this file system?

edit: The util fdisk doesn't support GPT. Use GNU Parted. Here is the parted: parted -l /dev/sdf

  Model: DELL PERC H310 (scsi)
  Disk /dev/sda: 500GB
  Sector size (logical/physical): 512B/512B
  Partition Table: msdos

  Number  Start   End    Size   Type     File system  Flags
   1      1049kB  525MB  524MB  primary  ext4         boot
   2      525MB   500GB  499GB  primary               lvm


  Model: JetStor JetStor -VOL1 - (scsi)
  Disk /dev/sdb: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  39.0TB  39.0TB               primary


  Model: JetStor JetStor -VOL2 (scsi)
  Disk /dev/sdc: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: JetStor JetStor -VOL1 - (scsi)
  Disk /dev/sdd: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  39.0TB  39.0TB               primary


  Model: JetStor JetStor -VOL2 (scsi)
  Disk /dev/sde: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: AC&Ncorp JetStor (scsi)
  Disk /dev/sdf: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Error: /dev/mapper/mpathcp1: unrecognised disk label

  Model: Linux device-mapper (multipath) (dm)
  Disk /dev/mapper/mpathc: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_cfile-lv_var: 105GB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End    Size   File system  Flags
   1      0.00B  105GB  105GB  ext4


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_jet716s_vol2-lv_jet716sV2a: 37.7TB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system  Flags
   1      0.00B  37.7TB  37.7TB  xfs


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_jet716s-lv_jet716s_jet: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system  Flags
   1      0.00B  39.0TB  39.0TB  xfs


  Error: /dev/mapper/mpathbp1: unrecognised disk label

  Error: /dev/mapper/mpathdp1: unrecognised disk label

  Model: Linux device-mapper (multipath) (dm)
  Disk /dev/mapper/mpathb: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  39.0TB  39.0TB               primary


  Model: Linux device-mapper (multipath) (dm)
  Disk /dev/mapper/mpathd: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_cfile-lv_swap: 33.8GB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system     Flags
   1      0.00B  33.8GB  33.8GB  linux-swap(v1)


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_cfile-lv_root: 53.7GB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system  Flags
   1      0.00B  53.7GB  53.7GB  ext4

edit2: Here is the file -s.

file -s /dev/sdf{,1,2,3,4,5,6}
    /dev/sdf:  x86 boot sector; partition 1: ID=0xee, starthead 0, startsector 1, 4294967295 sectors, extended partition table (last)\011, code offset 0x0
    /dev/sdf1: cannot open `/dev/sdf1' (No such file or directory)
    /dev/sdf2: cannot open `/dev/sdf2' (No such file or directory)
    /dev/sdf3: cannot open `/dev/sdf3' (No such file or directory)
    /dev/sdf4: cannot open `/dev/sdf4' (No such file or directory)
    /dev/sdf5: cannot open `/dev/sdf5' (No such file or directory)
    /dev/sdf6: cannot open `/dev/sdf6' (No such file or directory)

edit 3: Here's the output of multipath -l

      mpathd (35001b4d0189b4801) dm-2 JetStor,JetStor -VOL2
      size=47T features='0' hwhandler='0' wp=rw
      `-+- policy='round-robin 0' prio=0 status=active
        |- 7:0:0:1 sdc 8:32 active undef running
        `- 7:0:1:1 sde 8:64 active undef running
      mpathc (32029001378917600) dm-9 AC&Ncorp,JetStor
      size=47T features='0' hwhandler='0' wp=rw
      `-+- policy='round-robin 0' prio=0 status=active
        `- 8:0:0:0 sdf 8:80 active undef running
      mpathb (35001b4d0189b4800) dm-3 JetStor,JetStor -VOL1 -
      size=35T features='0' hwhandler='0' wp=rw
      `-+- policy='round-robin 0' prio=0 status=active
        |- 7:0:0:0 sdb 8:16 active undef running
        `- 7:0:1:0 sdd 8:48 active undef running

And the content of /etc/multipath.conf

      # This is a basic configuration file with some examples, for device mapper
      # multipath.
      # For a complete list of the default configuration values, see
      # /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
      # For a list of configuration options with descriptions, see
      # /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
      #
      # REMEMBER: After updating multipath.conf, you must run
      #
      # service multipathd reload
      #
      # for the changes to take effect in multipathd

      ## By default, devices with vendor = "IBM" and product = "S/390.*" are
      ## blacklisted. To enable mulitpathing on these devies, uncomment the
      ## following lines.
      #blacklist_exceptions {
      #       device {
      #               vendor  "IBM"
      #               product "S/390.*"
      #       }
      #}

      ## Use user friendly names, instead of using WWIDs as names.
      #defaults {
      #       user_friendly_names yes
      #}

      ##
      ## Here is an example of how to configure some standard options.
      ##
      # 5/21/14: uncommented this
      defaults {
              udev_dir                /dev
              polling_interval        10
              path_selector           "round-robin 0"
              path_grouping_policy    multibus
              getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
              prio                    alua
              path_checker            readsector0
              rr_min_io               100
              max_fds                 8192
              rr_weight               priorities
              failback                immediate
              no_path_retry           fail
              user_friendly_names     yes
      }
      ##
      ## The wwid line in the following blacklist section is shown as an example
      ## of how to blacklist devices by wwid.  The 2 devnode lines are the
      ## compiled in default blacklist. If you want to blacklist entire types
      ## of devices, such as all scsi devices, you should use a devnode line.
      ## However, if you want to blacklist specific devices, you should use
      ## a wwid line.  Since there is no guarantee that a specific device will
      ## not change names on reboot (from /dev/sda to /dev/sdb for example)
      ## devnode lines are not recommended for blacklisting specific devices.
      ##
      #blacklist {
      #       wwid 26353900f02796769
      #       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
      #       devnode "^hd[a-z]"
      #}
      #multipaths {
      #       multipath {
      #               wwid                    3600508b4000156d700012000000b0000
      #               alias                   yellow
      #               path_grouping_policy    multibus
      #               path_checker            readsector0
      #               path_selector           "round-robin 0"
      #               failback                manual
      #               rr_weight               priorities
      #               no_path_retry           5
      #       }
      #       multipath {
      #               wwid                    1DEC_____321816758474
      #               alias                   red
      #       }
      #}
      #devices {
      #       device {
      #               vendor                  "COMPAQ  "
      #               product                 "HSV110 (C)COMPAQ"
      #               path_grouping_policy    multibus
      #               getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
      #               path_checker            readsector0
      #               path_selector           "round-robin 0"
      #               hardware_handler        "0"
      #               failback                15
      #               rr_weight               priorities
      #               no_path_retry           queue
      #       }
      #       device {
      #               vendor                  "COMPAQ  "
      #               product                 "MSA1000         "
      #               path_grouping_policy    multibus
      #       }
      #}

      blacklist {
      }

There is another SCSI array on this, that may be why it changes devices.

You're running `xfs_check` on the whole disk. filesystems like `xfs` reside in partitions. something like `file -s /dev/sdf{1,5,6}` might reveal which partition has `xfs` filesystem. — Nizam Mohamed, Sep 09 '15 at 16:09
I have added the file -s info to my original post. It doesn't seem very informative. What else will can I use to get some info about this file system? — user1666203, Sep 09 '15 at 17:31

Jesse K · Answer 1 · 2015-09-09T18:30:41.423

3

It's hard to say without more information, but there are a couple of possibilities. I would comment if I could.

Worst case: You're hosed. Restore from backup. You have backups, right?

Best case: /dev/sdf is no longer the device name. Check the iSCSI LUN IDs and see if it got reassigned to another device name. Are you using a hardware iSCSI adapter or software? It might also be helpful to try fdisk -l /dev/sdf and post the output here.

EDIT: Thanks for that. I see several issues in your latest output:

Model: AC&Ncorp JetStor (scsi) Disk /dev/sdf: 52.0TB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1049kB 52.0TB 52.0TB primary Error: /dev/mapper/mpathcp1: unrecognised disk label

First of all, it's weird that the array changes model across devices. Is /dev/sdf provided by a different storage array?

Second, and probably more important, it looks like you have multipathing configured. Can you run multipath -l and paste the output? Also, can you paste the contents of your /etc/multipath.conf?

edited Sep 09 '15 at 18:30

answered Sep 09 '15 at 14:43

Jesse K

186
5

There are back ups but they are a little out of date so some data will be lost if we are force to use them. Here is the I'll post the fdisk -l /dev/sdf – user1666203 Sep 09 '15 at 16:41
Start by running `xfs_check /dev/sdf1`. This probably won't fix it, but let's see what kind of output that gives you. – Jesse K Sep 09 '15 at 17:50
xfs_check was the first thing I tried. It has a bad primary superblock - bad magic number and is not able to find a secondary superblock to use. Basically it give no information beyond that. – user1666203 Sep 09 '15 at 18:15
My mistake, focusing on the wrong detail. See above for edits. – Jesse K Sep 09 '15 at 20:39
I have added the multipath -l and /etc/multipath.conf to my original post. – user1666203 Sep 11 '15 at 18:24

score 2 · Answer 2 · answered Oct 19 '15 at 20:48

OK, post the result from this command. Change the device as necessary: sudo xfsdump -f /dev/sdb1 | grep -i superblock

I want to note my personal experience, that Ubuntu hasn’t had good XFS support for a while, and you might find it very helpful to download an old copy of warty warthog, and use a livecd with that.

You might get a response that looks like:

Couldn't find valid filesystem superblock.

if so, don't give up yet. You can try sudo mkfs.xfs -N /dev/sdb1 which will list the details about the filesystem, without writeing any changes. Hopefully you will see something at the end of that list like: ... superblock backups: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

That is a list (examples: yours may vary) of alternative superblocks. you can then use in a trial and error type method, the list of superblocks, to reconstruct your XFS filesystem using mkfs, fsck, and mknode.

Before I go writing a long answer, post back details from xfsdump, and mkfs.xfs

What do you mean by "reconstructing the XFS" using the superblock numbers. How would one do that? Thanks — ChrisR, Feb 25 '17 at 00:42

Cannot mount XFS - bad superblock

2 Answers2