1

I'm looking for some advice on setting-up multi-host I/O access on our SAN:

I have a blade enclosure (PowerEdge1000e) containing an Equallogic PS-M4110 storage blade with a single RAID6 volume currently formatted as ext4.

This is connected via iSCSI to one of other blades (all running ubuntu server 14.04) and mounted there as a standard drive.

Now I am trying to connect another of the blades in the enclosure to the SAN in a way that allows multi-host I/O.

Preferably trying to avoid the obvious solution of NFS because some of the slightly questionably coded tools we use have a habit of crashing and burning when doing high I/O to NFS. This is particularly problematic as these tools take weeks to run and don't have many opportunities to checkpoint (have you guessed this is an academic environment yet?).

However, everything plays nicely with the current iSCSI set-up. So I was leaning towards a cluster-aware or distributed file-system + iSCSI as the best option but I'm worried of split-brain issues etc as we only have 1 node.

1) Is any of the above remotely sane?

2) Do you have any recommendations of which fs to use (FOSS and linux compatible preferable)?

Eric S Nyberg
  • 11
  • 1

2 Answers2

0

iSCSI is a block level protocol that reads/writes to raw disk over a network. It does not have the concept of file locking. Since it is also a network protocol, it does allow multiple connections to a single target. You have to be able to guarantee that the file system level above iSCSI will have adequate file locking to prevent simultaneous writes.

I have never tried it (too scary). I have heard of examples where folk have allowed one machine read/write access and others read-only. Obviously a lot depends on what you want on those volumes. For example, the disk based contents of a SQL server would obviously require in depth knowledge of the locking mechanism used by that server and perhaps a read-only table feature.

ericx
  • 416
  • 1
  • 4
  • 10
  • Aye, there even seems to be inconsistency issues doing the read-only approach with ext4 as the [log is replayed even in read-only mounts](http://unix.stackexchange.com/questions/68790/can-the-same-ext4-disk-be-mounted-from-two-hosts-one-readonly). So to handle multihost it really needs a proper clustered or distributed fs but also one that can tolerate a single node without having issues. While I was looking for recommendations on them I'm starting to resign myself (after several people's advice) to using NFS and trying to sort the issues we have been having with certain applications. – Eric S Nyberg Aug 11 '14 at 16:41
  • 1
    Concurrent access to block devices is bad juju. Don't do it. Even with clustering software - they only allow access one at a time for a good reason. – Sobrique Aug 12 '14 at 14:39
0

iSCSI as has been mentioned is a block access protocol - it just happens to travel across ethernet. It is a terrible plan to try and do multi-host IO to a single block device. Most clusters do not support this for a reason. You need to worry about caching at various layers through the OS, and the atomicity of your I/O operations.

Even with one 'writable' and one 'read only' - the 'read only' host is effectively always reading a 'dirty' filesystem. I've written backup solutions that - effectively - do this by taking clone copies, consistency check the clones and mount them for backup. This still only JUST works, because I'm explicitly flushing data/host buffers, and ensuring I separately capture transaction logs so I can 'replay' and fix the dirty filesystem.

NFS exists for a reason, and it does handle multi-host access to Unix file systems in a pretty clean way. It's not perfect - you can end up tripping up with filesystem locking, but at least you're not corrupting your filesystem every time you use it.

Figure out why your tools are bombing out on NFS for starters - you may find it's down to soft vs. hard mounting, or timeouts on the NFS. Or try switching from TCP to UDP or vice versa.

If you really must use iSCSI, then I'd suggest what you want to do is - as I've outlined above - take a clone-instance approach, rather than a shared concurrent IO.

Sobrique
  • 3,747
  • 2
  • 15
  • 36