4

We have a couple dozen Win2008 and 2008R2 Enterprise clusters that are used for SQL Server 2008 and 2008 R2 Enterprise/Datacenter. In the past we have had many issues with random failovers and "Network is Partitioned" errors on several servers on the other side of the globe. This was mostly resolved in updating NIC drivers and uninstalling Forefront Endpoint Protection (not sure how that played into everything but it helped).

Fast forward six months to November and we are getting constant alerts from SCOM and in the Event Log that the clusters (two in particular) are failing with "Network is Partitioned" errors several times a week, but no failure actually occurred. SQL Server is still up and running, no interruption in service is noticed on the web front ends. The errors seem to originate from the 'Passive' node and replicate through the network (we receive the first alerts from Passive, then active, then web front end) but all nodes/network adapters/disks/applications/ip's/websites remain functional. We cannot find the reason for these errors continually popping up when nothing appears to be wrong with the cluster, network or anything at all. Any ideas about the cause or possible direction we could go to investigate would be great.

Ddono25
  • 165
  • 3
  • 13
  • have you gotten your networking guys involved to make sure there are no issues between the servers on your network? Only thing that springs to mind right now. – MattB Dec 03 '10 at 16:51
  • Yes, everything else on the network is up and working as it should be. They even went and made sure the cables were snug in the socket and not loose. That was our first concern with the limited errors and randomness of it all. – Ddono25 Dec 15 '10 at 19:13

1 Answers1

1

When you get a network is partitioned error, it means that the server that is currently running your cluster applications is isolated in some fashion from the other nodes. It is entirely possible (and likely), that you services will continue to run, assuming there are no other faults. The warning is telling you that if a failover were needed, it will likely fail (usually due to the node not having a path to hand off the disks/CSVs).

Be sure to carefully check the network topology and your cluster network settings between the servers in question. We had a nasty experience with this where the failover cluster was using multipath NICs for inter-node comms that were different than the ones SQL server were using (i.e. separate VLANs). Because both the primary and backup cluster connections were pathing in such a way that quorum could be lost if only one switch went down, the SQL server would still show as being online, but the cluster showed partitioned, meaning that if the server (or a switch) were to fail, it would bring the cluster down hard.

newmanth
  • 3,943
  • 4
  • 26
  • 47
  • 1
    We currently have nearly the exact same setup on our NIC's. That is a scary thought, I will check on our logs and config docs again to ensure this isn't the case. Thanks! – Ddono25 Dec 15 '10 at 19:15
  • I seem to get this same error few random times a week. The network guys checked and all seems OK. The VMs on the cluster also do not seem to be affected by this. Has anyone found the real issue /fix for this ? – whizkid Mar 30 '13 at 06:38