0

I am having some trouble with what should be a very basic configuration on an Ubuntu 16.04 server to use haproxy to load balance incoming metrics across two carbon-relay processes. Haproxy is only creating a TCP connection to the first server in the list of servers in the "listen" section. Also, when I tcpdump and watch for both carbon-relay ports, only the one connected receives any metrics from haproxy, but if I enable the "check" parameter in the server entries, the health checks do show up in the tcpdump output.

If I comment out the first server, or reverse their order, it does connect to the other server, and other testing doesn't indicate any problems with the carbon-relay processes, so I'm stumped as to why it will only connect to one server.

I have tried roundrobin, static-rr, and leastconn for the balance option, as well.

Here's a gist with the relevant sections of haproxy.cfg and carbon-relay.conf, plus the netstat output to show the listening ports and tcp connection to the one server.

https://gist.github.com/Calygos/b1442e0936adafb27fc80a76477235e9

  • How many clients do you have? – GregL Oct 20 '16 at 18:17
  • None at the moment other than the collectd process running on the same server, and that uses write_graphite to send to the haproxy port. –  Oct 20 '16 at 18:19

1 Answers1

1

Given that you only have a single client (local collectd process), HAProxy is working exactly as it should.

It's receiving the connection from collectd, and picking one of the two backend servers. If you had more clients, then it would start to balance the traffic to both.

If you wanted to test this, set your balance algorithm to roundrobin and restart collectd. Yuu should see this new connection hit the other backend server.

Alternatively, you could start a second connection (with collectd, or even telnet) and it should find it's way to the second server.

GregL
  • 9,370
  • 2
  • 25
  • 36
  • Interesting, that's exactly what happened. Actually that may be a problem for me. This server's collectd process is intended to be a forwarder for all of the other servers' collectd processes, so they'd send to it rather than directly to haproxy. In that case, I wouldn't be taking advantage of having multiple carbon-relays in an attempt to improve performance since they're single-threaded. –  Oct 20 '16 at 18:28
  • Why have the various collectd's send to a central collectd instead of to HAProxy directly? – GregL Oct 20 '16 at 18:32
  • Mostly because it's simpler (fewer chances for mistakes, only requires one place to configure the destination(s) for the metrics, etc.) to have every collectd client define a single Server value in its network plugin rather than having to define all of the necessary parameters for the write_graphite plugin. –  Oct 20 '16 at 18:44
  • Personally, I don't think that simplicity is a big deal, since I doubt you're going to be modifying those config options after the initial round of tweaking. If you've got that many agents, then the relay is going to be handy for certain, but you could always use one of the multi-threaded relay alternatives, like [carbon-c-relay](https://github.com/grobian/carbon-c-relay). – GregL Oct 20 '16 at 19:12