0

There's already tons of help and guide on how to do this. But for some reason I can't get it working and am not sure how to troubleshoot it.
I've got an RDS postgres instance with the private IP 10.0.122.220. I also have a bastion host with a (yes) private IP 10.0.94.67. I'm able to connect to ports on the bastion host, but not the RDS. So I'm trying to forward port 5432 of the bastion host to port 5432 of the RDS instance.
This is the status of the bastion host:

bastion$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    inet 10.0.94.67/19 brd 10.0.95.255 scope global dynamic eth0
...

bastion$ ip route show | grep default
default via 10.0.64.1 dev eth0

bastion$ cat /proc/sys/net/ipv4/ip_forward
1

Then I added two NAT rules:

bastion# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 5432 -j DNAT --to-destination 10.0.122.220

bastion# iptables -t nat -A POSTROUTING -o eth0 -p tcp --dport 5432 -d 10.0.122.220 -j SNAT --to-source 10.0.94.67

bastion# iptables -v -t nat -L -n
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5432 to:10.0.122.220

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 443 packets, 32660 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 443 packets, 32660 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       tcp  --  *      eth0    0.0.0.0/0            10.0.122.220         tcp dpt:5432 to:10.0.94.67

But I still can't connect to the RDS instance with the help of SSH tunneling:

my-machine$ ssh -v -NL 5432:10.0.94.67:5432 -i my-key ec2-user@10.0.94.67
debug1: Connection to port 5432 forwarding to 10.0.94.67 port 5432 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 5432 for 10.0.94.67 port 5432, connect from 127.0.0.1 port 57447 to 127.0.0.1 port 5432, nchannels 3
debug1: Connection to port 5432 forwarding to 10.0.94.67 port 5432 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: connect failed: Connection refused
debug1: channel 2: free: direct-tcpip: listening port 5432 for 10.0.94.67 port 5432, connect from 127.0.0.1 port 57448 to 127.0.0.1 port 5432, nchannels 3
... keeps repeating the above

What I can confirm is that, RDS is up, running, and responding, and bastion host has access to it, since with the following SSH tunnle I can connect to the database:

my-machine$ ssh -v -NL 5432:10.0.122.220:5432 -i my-key ec2-user@10.0.94.67

What did I miss? How can I troubleshoot it? Thanks.

Rad
  • 195
  • 3
  • 10
  • Could you double check the netmasks on both server to not have a too large one ? Add a '-v' option to your "iptables -L -n' command, will show you counters. If it stay to 0, the rule is not applied. Add logs at the end of your rules to debug the packets – Dom Aug 20 '20 at 07:29
  • @Dom updated the post. – Rad Aug 20 '20 at 07:33
  • Sorry : iptables -L -nv -t nat (else it display the filter rules, which are empty) – Dom Aug 20 '20 at 07:35
  • @Dom NP. That was actually a question for me (why it doesn't show anything without `-t nat`). Updated the question. – Rad Aug 20 '20 at 07:39
  • I'm not an expert, can you please guide how to do the logging (any page to read or something)? – Rad Aug 20 '20 at 07:44
  • 1
    To add logging : iptables -t nat -A PREROUTING -j LOG --log-prefix 'PREROUTING: ', then check in your /var/log/kern.log file. The new rule will be added at the end of the PREROUTING chain. You may want to do that too in POSTROUTING (if needed) – Dom Aug 20 '20 at 08:46

1 Answers1

0

I have a feeling that you are overcomplicating the things. As you are using SSH to forward the connection to the database, forget about iptables and NAT and just use SSH to forward DIRECTLY to the database server:

Use:

my-machine$ ssh -v -NL 5432:10.0.122.220:5432 -i my-key ec2-user@10.0.94.67

instead of:

my-machine$ ssh -v -NL 5432:10.0.94.67:5432 -i my-key ec2-user@10.0.94.67

Explanation why your solution DOESN'T work: The locally generated traffic does NOT go through PREROUTING chain of NAT table so it is NOT DNATted. Use OUTPUT table to DNAT locally generated traffic:

bastion# iptables -t nat -A OUTPUT -p tcp --dport 5432 -j DNAT --to-destination 10.0.122.220

But as I said - it is overcomplicating things. And you may also want to match on destination address in the above rule if you choose to use it.

Tomek
  • 3,390
  • 1
  • 16
  • 10
  • Thanks for the answer. The ultimate solution is not based on ssh (but AWS SSM). At the moment I'm using it for troubleshooting. Regarding your explanation, good point. So you're saying that because I'm using the same host to make a tunnel, it's a local traffic and won't go through PREROUTING. I used a different instance in the same network as a jump server and yes, it worked. But the number of PREROUTING packets (in `iptables -v -t nat -L`) is still 0. Can you please comment on that? – Rad Aug 20 '20 at 08:18
  • As said, not an iptable expert, can you please elaborate on matching on destination comment? thanks. – Rad Aug 20 '20 at 08:27
  • How many interfaces do you have? Are you sure the traffic comes in through eth0? – Tomek Aug 20 '20 at 08:29
  • Lookback + eth0 (the output of `ip a` shows only two entries). – Rad Aug 20 '20 at 08:29
  • Regarding the packets number, I think I looked at the wrong number (`Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)`. In the entry in the column `pkts` it shows some number. – Rad Aug 20 '20 at 08:35