1

i have vms that run nginx and php fpm on separate machines, for now i use a very stupid method of synchronizing directories by hand since cluster is not that big. However i have another server laying around that i would like to utilize as a shared storage device.

So far i understood that i can provide storage via iscsi with ocfs2 filestytem. What bothers me is that initial setup requires to specify each node in advance and then to add a node i would need to shutdown the o2cb to apply changes.

The goal is to provide shared storage to nginx and php-fpm nodes via iscsi (so i dont have to do replication), however number of nodes can grow depending on the load of the cluster.

Idea #1: Maybe i somehow can provide storage to my vms via host, so only host is dealing with ocfs2 directly ? that way the node is known.

Anton Stafeyev
  • 340
  • 2
  • 3
  • 13

2 Answers2

2

OCFS2 is a clustered filesystem that assumes each block device it uses for replicas is identical. It's designed for a very different use-case than something like nginx, where locking and ordering of data is extremely cohesive between participating nodes.

This takes a good deal of overhead, but can be very beneficial for running multiple workers that make modifications to the same dataset. This is a pattern we like to try to avoid in storage, though becomes useful sometimes. This is not that time.

This implementation would benefit from a central NFS or SMB share, not clustered FS backed by iSCSI. That way, each Nginx worker has access to the same directories. It would be best that they not try to write to the same file at the same time, but if you do need to do that you can make sure you're using NFS >v4.1 or SMB >v3.x. Both handle locking better than previous versions.

Spooler
  • 7,046
  • 18
  • 29
  • Thanks Spooler, alright if it is not nginx, can u please give an example of where i would benefit from ocfs2 and iscsi set up. and maybe a resource that goes more in depth about storage. i am still new. – Anton Stafeyev Aug 26 '20 at 16:55
  • An application that expects its filesystem to behave as if it's running locally on that machine while still being aware of locks and in-flight data from other machines will benefit from OCFS2. This is not common. In a solution like this, it would generally benefit the cluster to use multiple segregated communication paths for things like inter-node communication and storage backend communication/replication, and a solution of this design would have very low-latency data access. Thankfully, Nginx benefits from caching, so such a low-latency option at such a high price would not impart benefits. – Spooler Aug 26 '20 at 17:10
  • And somewhat important to note, OCFS2 *CAN* impart low-latency writes to clustered data *IF* the storage architecture is designed correctly - and that's pretty expensive. NFS clients can buffer and cache well enough that you'll probably notice it will be faster overall - throughput will certainly be better without all of the context switching and overhead associated with a clustered FS, and the simplicity of design is a huge benefit itself. Web frontend workers almost always don't require the cohesion offered by OCFS2, and if they do it's typically a requirement of bad design. – Spooler Aug 26 '20 at 17:14
  • It's also worth noting that the reason these OCFS2 or GFS2 clustered filesystems are less common now is because this kind of replication and cohesion is now more aptly handled at the structured data layer, rather than via backend storage. For example, application-level database replication is in a much better place to establish cohesive multi-node writes than using some unaware application to yeet out data to storage that it hopes is handling cohesion between nodes properly before hopefully getting that same data back from the next node. Clustered filesystems also REQUIRE fencing. That's hard. – Spooler Aug 26 '20 at 17:27
1

You could simply set up your third server as an NFS server, that would make more sense than using OCFS2 in that use case. OCFS2 is a better either on top of fast SAN storage, or storage replicated in real-time such as DRBD.

wazoox
  • 6,918
  • 4
  • 31
  • 63