UPDATE
It turned out that the problem is unrelated to NAT and UDP hole punching. So I have created a new Q.
I've implemented UDP hole punching using the following approach:
A known server (no NAT), at a known port, is handling incoming UDP.
A client behind NAT sends a datagram to the server, after which the server responds by sending the address of a peer to this client.
The client now starts communicating with the peer using UDP over the same socket (but to different address of course.)
If after a while, the client stops communication with peer, and sends another datagram to the server again, this datagram does not arrive (unless I keep channels open by sending keepalive datagrams from server to client.)
My question: How come it is OK to switch from datagrams between server<->client to datagrams between peer<->client, but as soon as I switch back and the client sends to the server again, they don't arrive?