How to fix Twisted python Connection to the other side was lost in a non-clean fashion?

Question

I am using buildbot as my CI. Its is built using twisted python.

My build worker is getting lost approximately after 10 minutes.

My network is fine, but wanted to know why my worker is getting lost.

I want to know in what circumstances we will see this kind of error in twisted python.

Error:

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.

Which Buildbot version do you use? – rutsky Mar 01 '17 at 22:54 — rutsky, Mar 01 '17 at 22:54
buildbot version 0.9.4. twisted is 17. – RMK Mar 02 '17 at 11:07 — RMK, Mar 02 '17 at 11:07

score 2 · Accepted Answer · answered Mar 02 '17 at 15:36

If you see deconnections, and there is no clue in both twisted.log of the master and the worker, then your problem is the networking.

You may have some firewall in between which close long running tcp connections or something.

I would advise to run tcpdump at both sides, and see who is sending the RST packet

score 1 · Answer 2 · answered Mar 02 '17 at 16:40

As @tardyp noticed probably this is your network configuration which resets long TCP connections without traffic.

Try setting smaller keepalive value for master-worker communication as described here:

c['workers'] = [
    worker.Worker('bot-linux', 'linuxpasswd',
                  keepalive_interval=60)
]

Default interval for sending keep-alive packets is one hour.

How to fix Twisted python Connection to the other side was lost in a non-clean fashion?

2 Answers2