How to connect to an IPsec server through a Nginx UDP stream reverse proxy?

Question

Here's the thing, I just want to use a jump server to speed up the connections to my IPsec server. So I configured Nginx to pass UDP packets as follows.

stream {
    # ...
    server {
        listen 500 udp;
        proxy_pass example.com:500;
        # ...
    }

    server {
        listen 4500 udp;
        proxy_pass example.com:4500;
        # ...
    }
    # ...
}

But I can't connect to my IPsec server through my jump server. So, I'd like to know, is this stuff not allowed now? If so, is it caused by the IPsec protocol or just the limitation of the hwdsl2/setup-ipsec-vpn that I used to configure my IPsec server?