So long as the protocol used over UDP is sanely designed, this won't be a problem. If the IP address of one endpoint changes, as soon as the first datagram is sent by that end, the other end should update its destination IP to the new one.
This is only a problem for protocols that create their own "connection" on top of UDP and erroneously assume that the other end's IP address is fixed. So long as they use a sensible connection identifier inside each UDP datagram and update the IP address of the other end if it changes, there is no problem.
TCP, of course, does guarantee that its connections can be uniquely and persistently identified by the 4-tuple of source port, source IP, destination port, and destination IP. Every device that handles TCP is designed not to break this assumption. But if you want to implement connections over UDP, you have to create your own connection identifier. You can't use the 4-tuple of source port, source IP, destination port, and destination IP because devices that handle UDP are under no obligation not to break this assumption.
At least, you can't if you expect your "connections" to remain stable. If you don't mind if your "connections" break and have to be re-formed, then don't worry about it.
There are three possibilities:
The protocol layered over UDP does not need to support IP address changes, in which case it doesn't matter.
The protocol layered over UDP does need to support IP address changes and therefore does so, in which case there's no problem.
The protocol layered over UDP does need to support IP address changes but doesn't do so, in which case the protocol is broken and should be fixed.