I am currently connected to an ISP with unknown limitations.
I have setup WireGuard and it works, since it works via UDP and sometimes this ISP cuts the UDP completely, I needed to setup OpenVPN on TCP which I did, wireguard was unaffected and still works but open-vpn doesn't connect and logs aren't really helpful.
One important thing is that if I first turn on the WireGuard (Talking about client not server) then the OpenVpn it just works, no idea why but it just connects without any issues but it doesn't without WireGuard being on.
I have tried SSH tunnel just to see if that also makes it connect but it doesn't.
Firewall port is okay.
Here's the log when it fails. (With WireGuard off).
[Sep 29, 2022, 21:33:48] OpenVPN core 3.git::d3f8b18b mac x86_64 64-bit built on May 18 2022 07:15:04
⏎[Sep 29, 2022, 21:33:48] Frame=512/2048/512 mssfix-ctrl=1250
⏎[Sep 29, 2022, 21:33:48] UNUSED OPTIONS
4 [resolv-retry] [infinite]
5 [nobind]
6 [persist-key]
7 [persist-tun]
11 [auth-nocache]
13 [tls-client]
16 [ignore-unknown-option] [block-outside-dns]
17 [block-outside-dns]
18 [verb] [3]
⏎[Sep 29, 2022, 21:33:48] EVENT: RESOLVE ⏎[Sep 29, 2022, 21:33:48] EVENT: WAIT ⏎[Sep 29, 2022, 21:33:48] UnixCommandAgent: transmitting bypass route to /var/run/agent_ovpnconnect.sock
{
"host" : "6.8.198.252",
"ipv6" : false,
"pid" : 17606
}
⏎[Sep 29, 2022, 21:33:48] Connecting to [6.8.198.252]:1196 (6.8.198.252) via TCPv4
⏎[Sep 29, 2022, 21:33:57] Server poll timeout, trying next remote entry...
⏎[Sep 29, 2022, 21:33:57] EVENT: RECONNECTING ⏎[Sep 29, 2022, 21:33:57] EVENT: RESOLVE ⏎[Sep 29, 2022, 21:33:57] EVENT: WAIT ⏎[Sep 29, 2022, 21:33:57] UnixCommandAgent: transmitting bypass route to /var/run/agent_ovpnconnect.sock
{
"host" : "6.8.198.252",
"ipv6" : false,
"pid" : 17606
}
⏎[Sep 29, 2022, 21:33:58] Connecting to [6.8.198.252]:1196 (6.8.198.252) via TCPv4
⏎[Sep 29, 2022, 21:34:07] Server poll timeout, trying next remote entry...
⏎[Sep 29, 2022, 21:34:07] EVENT: RECONNECTING ⏎[Sep 29, 2022, 21:34:07] EVENT: RESOLVE ⏎[Sep 29, 2022, 21:34:07] EVENT: WAIT ⏎[Sep 29, 2022, 21:34:07] UnixCommandAgent: transmitting bypass route to /var/run/agent_ovpnconnect.sock
{
"host" : "6.8.198.252",
"ipv6" : false,
"pid" : 17606
}
⏎[Sep 29, 2022, 21:34:08] Connecting to [6.8.198.252]:1196 (6.8.198.252) via TCPv4
⏎[Sep 29, 2022, 21:34:08] EVENT: CONNECTING ⏎[Sep 29, 2022, 21:34:08] Tunnel Options:V4,dev-type tun,link-mtu 1523,tun-mtu 1500,proto TCPv4_CLIENT,cipher AES-128-GCM,auth [null-digest],keysize 128,key-method 2,tls-client
⏎[Sep 29, 2022, 21:34:08] Creds: UsernameEmpty/PasswordEmpty
⏎[Sep 29, 2022, 21:34:08] Peer Info:
IV_VER=3.git::d3f8b18b
IV_PLAT=mac
IV_NCP=2
IV_TCPNL=1
IV_PROTO=30
IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305
IV_AUTO_SESS=1
IV_GUI_VER=OCmacOS_3.3.6-4368
IV_SSO=webauth,openurl,crtext
⏎[Sep 29, 2022, 21:34:39] TCP recv error: Operation timed out
⏎[Sep 29, 2022, 21:34:39] Transport Error: Transport error on '6.8.198.252: NETWORK_RECV_ERROR
⏎[Sep 29, 2022, 21:34:39] EVENT: TRANSPORT_ERROR Transport error on '6.8.198.252: NETWORK_RECV_ERROR⏎[Sep 29, 2022, 21:34:39] Client terminated, restarting in 5000 ms...
⏎[Sep 29, 2022, 21:34:44] EVENT: RECONNECTING ⏎[Sep 29, 2022, 21:34:44] EVENT: RESOLVE ⏎[Sep 29, 2022, 21:34:44] EVENT: WAIT ⏎[Sep 29, 2022, 21:34:44] UnixCommandAgent: transmitting bypass route to /var/run/agent_ovpnconnect.sock
{
"host" : "6.8.198.252",
"ipv6" : false,
"pid" : 17606
}
⏎[Sep 29, 2022, 21:34:45] Connecting to [6.8.198.252]:1196 (6.8.198.252) via TCPv4
⏎[Sep 29, 2022, 21:34:45] EVENT: CONNECTING ⏎[Sep 29, 2022, 21:34:45] Tunnel Options:V4,dev-type tun,link-mtu 1523,tun-mtu 1500,proto TCPv4_CLIENT,cipher AES-128-GCM,auth [null-digest],keysize 128,key-method 2,tls-client
⏎[Sep 29, 2022, 21:34:45] Creds: UsernameEmpty/PasswordEmpty
⏎[Sep 29, 2022, 21:34:45] Peer Info:
IV_VER=3.git::d3f8b18b
IV_PLAT=mac
IV_NCP=2
IV_TCPNL=1
IV_PROTO=30
IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305
IV_AUTO_SESS=1
IV_GUI_VER=OCmacOS_3.3.6-4368
IV_SSO=webauth,openurl,crtext
⏎[Sep 29, 2022, 21:34:47] EVENT: CONNECTION_TIMEOUT BYTES_IN : 248
BYTES_OUT : 954
PACKETS_IN : 4
PACKETS_OUT : 8
NETWORK_RECV_ERROR : 1
TRANSPORT_ERROR : 1
CONNECTION_TIMEOUT : 1
N_RECONNECT : 3
⏎[Sep 29, 2022, 21:34:47] EVENT: DISCONNECTED ⏎[Sep 29, 2022, 21:35:01] Raw stats on disconnect:
BYTES_IN : 248
BYTES_OUT : 954
PACKETS_IN : 4
PACKETS_OUT : 8
NETWORK_RECV_ERROR : 1
TRANSPORT_ERROR : 1
CONNECTION_TIMEOUT : 1
N_RECONNECT : 3
⏎[Sep 29, 2022, 21:35:01] Performance stats on disconnect:
CPU usage (microseconds): 3421812
Network bytes per CPU second: 351
Tunnel bytes per CPU second: 0
⏎
In GUI it keep loading for a minute.
Then it gives:
And here's the log when it is successful (With WireGuard On).
[Sep 28, 2022, 01:45:24] OpenVPN core 3.git::d3f8b18b mac x86_64 64-bit built on May 18 2022 07:15:04
⏎[Sep 28, 2022, 01:45:24] Frame=512/2048/512 mssfix-ctrl=1250
⏎[Sep 28, 2022, 01:45:24] UNUSED OPTIONS
4 [resolv-retry] [infinite]
5 [nobind]
6 [persist-key]
7 [persist-tun]
11 [auth-nocache]
13 [tls-client]
16 [ignore-unknown-option] [block-outside-dns]
17 [block-outside-dns]
18 [verb] [6]
⏎[Sep 28, 2022, 01:45:24] EVENT: RESOLVE ⏎[Sep 28, 2022, 01:45:24] EVENT: WAIT ⏎[Sep 28, 2022, 01:45:24] UnixCommandAgent: transmitting bypass route to /var/run/agent_ovpnconnect.sock
{
"host" : "6.8.198.252",
"ipv6" : false,
"pid" : 37687
}
⏎[Sep 28, 2022, 01:45:24] Connecting to [6.8.198.252]:1196 (6.8.198.252) via TCPv4
⏎[Sep 28, 2022, 01:45:24] EVENT: CONNECTING ⏎[Sep 28, 2022, 01:45:24] Tunnel Options:V4,dev-type tun,link-mtu 1523,tun-mtu 1500,proto TCPv4_CLIENT,cipher AES-128-GCM,auth [null-digest],keysize 128,key-method 2,tls-client
⏎[Sep 28, 2022, 01:45:24] Creds: UsernameEmpty/PasswordEmpty
⏎[Sep 28, 2022, 01:45:24] Peer Info:
IV_VER=3.git::d3f8b18b
IV_PLAT=mac
IV_NCP=2
IV_TCPNL=1
IV_PROTO=30
IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305
IV_AUTO_SESS=1
IV_GUI_VER=OCmacOS_3.3.6-4368
IV_SSO=webauth,openurl,crtext
⏎[Sep 28, 2022, 01:45:25] SSL Handshake: peer certificate: CN=server_erZOtk0JAjlstcg2, 256 bit EC, curve:prime256v1, cipher: ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD
⏎[Sep 28, 2022, 01:45:25] Session is ACTIVE
⏎[Sep 28, 2022, 01:45:25] EVENT: GET_CONFIG ⏎[Sep 28, 2022, 01:45:25] Sending PUSH_REQUEST to server...
⏎[Sep 28, 2022, 01:45:25] OPTIONS:
0 [dhcp-option] [DNS] [1.0.0.1]
1 [dhcp-option] [DNS] [1.1.1.1]
2 [redirect-gateway] [def1] [bypass-dhcp]
3 [route-gateway] [10.8.0.1]
4 [topology] [subnet]
5 [ping] [10]
6 [ping-restart] [120]
7 [ifconfig] [10.8.0.2] [255.255.255.0]
8 [peer-id] [0]
9 [cipher] [AES-128-GCM]
⏎[Sep 28, 2022, 01:45:25] PROTOCOL OPTIONS:
cipher: AES-128-GCM
digest: NONE
key-derivation: OpenVPN PRF
compress: NONE
peer ID: 0
control channel: tls-crypt enabled
⏎[Sep 28, 2022, 01:45:25] TunPersist: short-term connection scope
⏎[Sep 28, 2022, 01:45:25] TunPersist: new tun context
⏎[Sep 28, 2022, 01:45:25] EVENT: ASSIGN_IP ⏎[Sep 28, 2022, 01:45:25] CAPTURED OPTIONS:
Session Name: 6.8.198.252
Layer: OSI_LAYER_3
MTU: 1500
Remote Address: 6.8.198.252
Tunnel Addresses:
10.8.0.2/24 -> 10.8.0.1
Reroute Gateway: IPv4=1 IPv6=0 flags=[ ENABLE REROUTE_GW DEF1 BYPASS_DHCP IPv4 ]
Block IPv6: no
Add Routes:
Exclude Routes:
DNS Servers:
1.0.0.1
1.1.1.1
Search Domains:
⏎[Sep 28, 2022, 01:45:25] SetupClient: transmitting tun setup list to /var/run/agent_ovpnconnect.sock
{
"config" :
{
"iface_name" : "",
"layer" : "OSI_LAYER_3",
"tun_prefix" : false
},
"pid" : 37687,
"tun" :
{
"adapter_domain_suffix" : "",
"block_ipv6" : false,
"dns_servers" :
[
{
"address" : "1.0.0.1",
"ipv6" : false
},
{
"address" : "1.1.1.1",
"ipv6" : false
}
],
"layer" : 3,
"mtu" : 1500,
"remote_address" :
{
"address" : "6.8.198.252",
"ipv6" : false
},
"reroute_gw" :
{
"flags" : 307,
"ipv4" : true,
"ipv6" : false
},
"route_metric_default" : -1,
"session_name" : "6.8.198.252",
"tunnel_address_index_ipv4" : 0,
"tunnel_address_index_ipv6" : -1,
"tunnel_addresses" :
[
{
"address" : "10.8.0.2",
"gateway" : "10.8.0.1",
"ipv6" : false,
"metric" : -1,
"net30" : false,
"prefix_length" : 24
}
]
}
}
POST unix://[/var/run/agent_ovpnconnect.sock]/tun-setup : 200 OK
{
"iface_name" : "utun3",
"layer" : "OSI_LAYER_3",
"tun_prefix" : true
}
/sbin/ifconfig utun3 down
/sbin/ifconfig utun3 10.8.0.2 10.8.0.1 netmask 255.255.255.0 mtu 1500 up
/sbin/route add -net 10.8.0.0 -netmask 255.255.255.0 10.8.0.2
add net 10.8.0.0: gateway 10.8.0.2
/sbin/route add -net 6.8.198.252 -netmask 255.255.255.255 10.66.66.19
add net 6.8.198.252: gateway 10.66.66.19
/sbin/route add -net 0.0.0.0 -netmask 128.0.0.0 10.8.0.1
add net 0.0.0.0: gateway 10.8.0.1
/sbin/route add -net 128.0.0.0 -netmask 128.0.0.0 10.8.0.1
add net 128.0.0.0: gateway 10.8.0.1
MacDNSAction: FLAGS=F RD=1 SO=5000 DNS=1.0.0.1,1.1.1.1 DOM= ADS=
open utun3 SUCCEEDED
⏎[Sep 28, 2022, 01:45:25] Connected via utun3
⏎[Sep 28, 2022, 01:45:25] EVENT: CONNECTED 6.8.198.252:1196 (6.8.198.252) via /TCPv4 on utun3/10.8.0.2/ gw=[10.8.0.1/]⏎[Sep 28, 2022, 01:45:25] MacLifeCycle NET_IFACE utun2
⏎
in the GUI it connects in less than 2 seconds.
One Important point
1.1.1.1 and 1.0.0.1 are both blocked as DNS (which are setup on open-vpn the config is a little further down keep reading), when wire-guard is on, these two work, otherwise they don't I have no idea if this could be the issue or not, Ideally I want open-vpn to make 1.1.1.1 and 1.0.0.1 work just like wireguard does.
Here's the open-vpn's server.conf:
port 1196
proto tcp
dev tun
user nobody
group nobody
persist-key
persist-tun
keepalive 10 120
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 1.0.0.1"
push "dhcp-option DNS 1.1.1.1"
push "redirect-gateway def1 bypass-dhcp"
dh none
ecdh-curve prime256v1
tls-crypt tls-crypt.key
crl-verify crl.pem
ca ca.crt
cert server_erZOtk0JAjlstcg2.crt
key server_erZOtk0JAjlstcg2.key
auth SHA256
cipher AES-128-GCM
ncp-ciphers AES-128-GCM
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
client-config-dir /etc/openvpn/ccd
status /var/log/openvpn/status.log
verb 3
Here's the /etc/csf/csfpost.sh since I use csf on the server.
#!/bin/bash
#/usr/sbin/iptables -t nat -A POSTROUTING -o ens192 -s 10.66.66.1/24 -j MASQUER$
# Used common for both wireguard and open-vpn
/usr/sbin/iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE
# This section is for Wireguard
/usr/sbin/iptables -A INPUT -i wg0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -i wg0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -o wg0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -i wg0 -o ens192 -m state --state RELATED,ESTABLI$
/usr/sbin/iptables -A FORWARD -i ens192 -o wg0 -m state --state RELATED,ESTABLI$
# This section is for open-vpn
#/usr/sbin/iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE
/usr/sbin/iptables -A INPUT -i tun+ -j ACCEPT
/usr/sbin/iptables -A FORWARD -i tun+ -j ACCEPT
/usr/sbin/iptables -A FORWARD -o tun0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -i tun+ -o ens192 -m state --state RELATED,ESTABL$
/usr/sbin/iptables -A FORWARD -i ens192 -o tun+ -m state --state RELATED,ESTABL$
And here's the client's config:
client
proto tcp-client
remote 6.8.198.252 1196
dev tun
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
verify-x509-name server_erZOtk0JAjlstcg2 name
auth SHA256
auth-nocache
cipher AES-128-GCM
tls-client
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
ignore-unknown-option block-outside-dns
setenv opt block-outside-dns # Prevent Windows 10 DNS leak
verb 6
<ca>
(redacted)
\</ca>
<cert>
(redacted)
</cert>
<key>
(redacted)
</key>
<tls-crypt>
(redacted)
</tls-crypt>
Update right now 1.1.1.1 works for me without WireGuard (Inconsistent ISP) but still OpenVPN is failing to connect so I don't think this is a DNS issue, but what is it?
I have tested the port (with nc) and I can reach it without WireGuard from my Computer this isn't a connection issue I have no idea what it is.
Update more info about the routes (iptables-save, ip route, ip rule, netstat -4npl):
(Since it was too long I had to put it somewhere else.)
https://gist.github.com/Stevemoretz/be556e4b5f4c19981436034419bc7835
Tested with new firewall rules as @gapsf suggested.
#!/bin/bash
#/usr/sbin/iptables -t nat -A POSTROUTING -o ens192 -s 10.66.66.1/24 -j MASQUERADE
/usr/sbin/iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE
/usr/sbin/iptables -I INPUT -j ACCEPT
/usr/sbin/iptables -I OUTPUT -j ACCEPT
/usr/sbin/iptables -I FORWARD -j ACCEPT
/usr/sbin/iptables -A INPUT -i wg0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -i wg0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -o wg0 -j ACCEPT
/usr/sbin/iptables -A FORWARD -i wg0 -o ens192 -m state --state RELATED,ESTABLISHED -j ACCEPT
/usr/sbin/iptables -A FORWARD -i ens192 -o wg0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#/usr/sbin/iptables -t nat -A POSTROUTING -o ens192 -j MASQUERADE
/usr/sbin/iptables -A INPUT -i tun+ -j ACCEPT
/usr/sbin/iptables -A FORWARD -i tun+ -j ACCEPT
/usr/sbin/iptables -A FORWARD -o tun0 -j ACCEPT
Still the same thing happens, same logs same everything.
new iptables-save output
https://gist.github.com/Stevemoretz/6a47895ae06ef69fe2c902d089fce847
Update traceroute with and without wg.
without:
traceroute 6.8.198.252
traceroute to 6.8.198.252 (6.8.198.252), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 114.463 ms 2.728 ms 1.359 ms
2 151-238-217-1.shatel.ir (151.238.217.1) 14.854 ms 14.380 ms 15.212 ms
3 10.51.242.73 (10.51.242.73) 16.762 ms 19.333 ms 17.473 ms
4 * 10.51.242.74 (10.51.242.74) 18.630 ms *
5 * * *
6 10.202.7.18 (10.202.7.18) 17.361 ms 17.189 ms 16.990 ms
7 10.201.47.186 (10.201.47.186) 20.253 ms 20.973 ms 22.286 ms
8 10.21.21.10 (10.21.21.10) 29.191 ms * *
9 * * *
10 core23.fsn1.hetzner.com (213.239.224.249) 100.352 ms 113.237 ms
core24.fsn1.hetzner.com (213.239.224.253) 106.071 ms
11 ex9k3.dc8.fsn1.hetzner.com (213.239.229.138) 102.469 ms 106.643 ms
ex9k3.dc8.fsn1.hetzner.com (213.239.229.142) 104.700 ms
12 static.181.clients.you. (88.99.2.181) 102.573 ms 101.464 ms 100.964 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
31 * * *
32 * * *
with:
traceroute 6.8.198.252
traceroute to 6.8.198.252 (6.8.198.252), 64 hops max, 52 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
31 * * *
32 * * *
Nslookup test:
Without wireguard:
nslookup google.com
Server: 1.1.1.1
Address: 1.1.1.1#53
Non-authoritative answer:
Name: google.com
Address: 216.239.38.120
With wireguard:
nslookup google.com
Server: 1.1.1.1
Address: 1.1.1.1#53
Non-authoritative answer:
Name: google.com
Address: 142.251.39.14
