How to restart a Java service without disruption using firewall/iptables?

Question

If enable this rule

iptables -t nat -A PREROUTING -p tcp --dport 80  -j REDIRECT --to 8080

Then connections coming to the server port 80 are redirected to localhost:8080. If I want to restart the service, can I just start the service on another port? Say port 8081 and re route the firewalling to

iptables -t nat -A PREROUTING -p tcp --dport 80  -j REDIRECT --to 8081 # Apparently -A won't work. I have to replace the rule, not add it. But I don't know how to do it yet

However, what about the established TCP socket connections on the NAT-ed port 8080? Will they be dropped immediately after firewall changes? Alternatively, will they keep working until a normal TCP socket close?

If so, then this works a restart of the application with no disruption by having the old instance on port 8080 on a graceful shutdown and the new one on port 8081 with the new features.

Is this reasoning correct?

Answer 1

You asked: "Is this reasoning correct?"

My answer is: "In order to be partially correct, you have to fix several details among your assumptions and actions. Not being 100% correct, this also means that... it's basically a wrong approach!". And I say such a statement due to following factors (in more or less random order):

1. IMHO, you're going to solve a problem (restarting a TCP server/service without disrupting existing established TCP connections) using the wrong approach. What looks to me as a "natural" approach is to solve the issue at "application level". As you're talking about a TCP/Server application, chances are very high it's "multithreaded" or "multiprocess". In such an environment, nothing prevent you to "restart" unused threads/processes, while waiting for "busy" threads/process to complete their activities. Sure, it's more complex than putting down a couple of iptables rule but.... it's what looks "natural" (to me). BTW: this is exactly the approach that "apache" developers implemented in their "graceful restart" for the httpd TCP daemon

Having said the above, as for the iptables approach that you are interested into, here are my comments:

2. as already mentioned by @iain, you cannot use -A but need to replace existing rule. Give a look to the iptables man-page and search for the -R or --if you have problems identifying the to-be-replaced rule-- the couple -D + -A;

3. iptables is really powerful so it should be possible to match only new connections. In other words, you can rewrite your NAT-rule to match only new connections (give a look to connection states and conntrack match). With such an approach, existing connection will stick to "old" PORT, while new connections will be redirected to "new" PORT. Unfortunately the "new" connection will became "established" pretty soon and... will be recognized by the first rule (redirected to the OLD port, as it deals with "established" connections). Anyway, you should be able to address such a problem with a mixture of 1) mangling the inbound/oubound packets; 2) proper port-range selection; 3) user-space management of connection-tracking; 4) other-stuff-that-I-surely-ignore . Having said that, please don't ask me for details, as I'm not a netfilter expert (but I bet it can be done);

4. you asked: "what about the established TCP socket connections on the NAT-ed port 8080? Will they be dropped immediately after firewall changes? Alternatively, will they keep working until a normal TCP socket close?"

   4.1 "will they be dropped immediately": here you're misusing the word "DROP" that, in iptables/netfilter context, is a key-concept, on pair with the "REJECT" term. See here as a start for personal investigation. In your particular case (where you change the underlying NAT rule while the connection is established) I think that as soon as the first TCP packet of the established connection ("established" towards the "old" TCP/Server) will be received by the "new" TCP/server due to the updated DNAT-rule, the "new" TCP/server will simply "terminate" the connection (with a "RESET" flag, I guess), as it won't recognized such a packets. I don't think this can be classified as a "DROP", as a RST will be sent (I guess).

   4.2 "will they keep working until a normal TCP socket close": I'd bet they won't! If you apply some "violence" to random TCP packets of an established TCP-connection, not only by removing packets, but shipping them to a different TCP/server.... you can be sure that something strange will happen. And, at the very least, the words "they keep working" cannot apply! BTW: this is exactly the reason for a software like conntrackd to exists, so to ensure the alignement of connection-tracking data among a two-node HA iptables firewall cluster.

Answer 2

As you are using -A you are adding a rule so the earlier rules are unaffected.

With iptables, the first rule that matches wins so in effect with the rules as stated, any connection that reaches port 80 will be forwarded to port 8080.

If however you decide to delete the rule that forwards port 80 to port 8080 then packets destined for port 80 will be forwarded to port 8081. This will break the existing connections as the server on port 8081 won't have an idea about them.