If you use the least-connection load balancing algorithm, IPVS will send a new connection to the backend with the least number of active connections, which will mean that you'll only get a "too many connections" error when all the servers are full. There are currently no mechanisms in IPVS (or ldirectord) to specify a limit to the number of simultaneous connections a backend will accept before it is unable to handle any more; it wouldn't be too hard to implement, actually, but the question is what do you do with connections when you're full? RST the connection attempt? Whatever you do, it's going to be an error that the client will have to handle, and "too many connections" in MySQL-protocol-speak is easier to diagnose, in my mind, that "Connection refused", because there's a lot more and varied reasons for the latter error than the former.
If it were my cluster, I'd switch to least-connection balancing and add a lot more monitoring around workload; if you're hitting limits and you didn't know you were getting close to them well in advance, you've got a monitoring fail right there.