7

Does anyone know how shared access (multiple Initiators) to a single iSCSI target is supposed to work.

I have got it semi-working with Linux (CentOS-7) target and Win 2012 server / Win Hyper-V 2012 R2 Initiators. Problem is each instance seems to get it's own version of the NTFS partition.

The reason for doing this is so that I can have my virtual machine files hosted on the remote iSCSI target and migrate the actual running virtual machines between servers. Without sharing the iSCSI target I cannot see any way of doing this.

NickC
  • 2,373
  • 13
  • 41
  • 55

2 Answers2

17

To have multiple initiators share a single target, whether over ISCSI, Fibre Channel or other SAN solution, you need a cluster-aware filesystem. VMWare ESXi does this with VMFS. Veritas offers one in Veritas Cluster Suite. Sun offered one back in the day that would cause no end of troubles. Oracle did this with RAC, until they got wise and changed to transaction replication instead of shared block-storage. Microsoft offers NTFS Cluster Shared Volumes, which is a Windows clustered filesystem. I believe Red Hat has one, though I have never played with it.

Clustered filesystems have a lot more nuanced work that stand-alone filesystems, since they have to understand quorum-counting, fencing, dead-peer detection and propagation. If they cannot do these things, a split-brain scenario (each node thinks he has the sole access to the shared resource, and cannot check with the other nodes) would enable you to corrupt all your data, or at least get into an inconsistent (and hard to recover from) state.

Another thing to be aware of: most of these are OS-specific. Obviously VMFS is only ESXi (and I think only if it is connected to VCenter). NTFS CSV is only Windows 2012. Oracle RAC is for database clusters running the Oracle database. I think Veritas is the only application-independant/cross-operating-system solution in the mix.

Veritas Cluster Suite is complex and not cheap. Also, you need to understand service groups, order of bring-up/tear-down and clustering/fencing for it to work well. If you miss the section on distributed locking, contention and forcible lock-breaking (I got pulled into a meeting with other architects just as we were beginning this module), you will regret.

Veritas Cluster Suite creates very tightly-coupled clusters, in which one node can induce symathy-sickness into another node. Symantec (who bought out Veritas recently) has a pair of classes that you should probably attend before taking a cluster based on Vx Cluster Suite into production, since it is both powerful and dangerous (like a car or a chainsaw or a side-arm).

Giacomo1968
  • 3,542
  • 27
  • 38
DTK
  • 1,718
  • 10
  • 15
  • Ceph, GlusterFS are a couple others. – EEAA Oct 04 '14 at 15:31
  • Thank you for those. I had never heard of Ceph. My understanding of Gluster (I last looked at it before Red Hat acquired) was that it allowed replication and horizontal scaling across nodes, allowing essentially a RAID 1+0 or RAID 5+0, but for nodes, rather than LUNs or disks. Thanks, I need to read up on both of these. – DTK Oct 04 '14 at 15:34
  • 1
    Ceph and GlusterFS are distributed storage solutions, but they do not provide the same thing as typical cluster aware file systems. Ceph/GlusterFS handles replication of files across multiple systems and access to that data, whereas products like VxFS, OCFS2, and VMFS work with shared, direct access storage. – Gene Oct 04 '14 at 15:40
  • Dumb question, but are Ceph and Gluster file-level replication or block-level replication? I thought I remembered Gluster being block-level replication, sort of network-based asynchronous RAID, but it has been a while (at least two years, maybe a little longer). – DTK Oct 04 '14 at 16:07
  • Oracle RAC still uses shared block storage. "Transaction replication" would be DataGuard variants. – Mat Oct 04 '14 at 19:29
  • Dataguard and Golden Gate I believe are the two solutions for asynchronous transaction replication. I think DB2 offers something similar, using MQ Series. In short, tightly coupled clusters can be powerful, but incredibly dangerous (just ask an MVS guy who has watched a cascading failure across an IMS-plex in a SYSPLEX that uses the CF / XCF for shared structures and shared clock). – DTK Oct 04 '14 at 19:32
  • RAC uses OCFS2? – Nils Jun 20 '15 at 20:56
  • I recently got clarification from a DBA buddy of mine. RAC is block replication over the storage plane, using a clustered file system. DataGuard is block replication over the data plane, using a local file system. Golden Gate is transaction replication over the data plane, using per-table or per-table-space transaction-log-sending. Each of these solutions is more loosely coupled than the one before it; you gain partition-resilience and availability at the cost of a delay to consistency. – DTK Jun 20 '15 at 23:41
2

DTK is completely correct, you'll need a clustering solution in order to manage shared access to the same storage, otherwise data corruption can and will ensue.

Since you specifically mentioned Hyper-V virtual machines, I'll offer some more specific documentation on how this should be done on Windows systems (including Hyper-V hosts):

https://technet.microsoft.com/en-us/library/jj612868.aspx

Of course, you'll need to build a complete Windows failover cluster in order to use CSVs, so make sure to read all the surrounding sections about failover clustering, of which CSVs are only a component.

Massimo
  • 70,200
  • 57
  • 200
  • 323