0

I am having a bit of a strange issue. I have a VMWare ESXi server with two virtual machines running on it. They are running just fine and can communicate on the network without a problem.

I am now trying to add a third. I am installing Ubuntu 8.04 Server. I assign it a static IP address and it's a fresh installation. Once installed I can ping the gateway but I cannot ping the DNS server. It's on the same network with the other two VMs which are communicating just fine. I have tried to reinstall the operating system but it still fails to connect.

Here is /etc/network/interfaces

auto eth0
iface eth0 inet static
    address 192.168.1.23
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    dns-nameservers 208.67.222.222 #opendns
    dns-search mydomain.com

Here is route

Destination | Gateway     | Genmask       | Flags | Metric | Ref | Use | Iface
localnet    | *           | 255.255.255.0 | U     | 0      | 0   | 0   | eth0
default     | 192.168.1.1 | 0.0.0.0       | UG    | 100    | 0   | 0   | eth0

Since I'm running this behind a FortiGate this is what the sniff command gives me when I try to ping 208.67.222.222

arp who-has 192.168.1.1 tell 192.168.1.23
arp reply 192.168.1.1 is-at MAC
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request
192.168.1.23 -> 208.67.222.222: icmp: echo request

As you can see it looks like I never get a response. One interesting thing I notice is the arp reply's MAC doesn't look right. I have cleared the FortiGate's ARP cache though and checked the entry and it seems correct. The MAC it lists is the one for the router. However if I ping from a different virtual machine that is also Ubuntu 8.04 with a nearly identical configuration I get this.

192.168.1.22 -> 208.67.222.222: icmp: echo request
208.67.222.222 -> 192.168.1.22: icmp: echo reply
192.168.1.22 -> 208.67.222.222: icmp: echo request
208.67.222.222 -> 192.168.1.22: icmp: echo reply
192.168.1.22 -> 208.67.222.222: icmp: echo request
208.67.222.222 -> 192.168.1.22: icmp: echo reply

So, what could I be missing?

Thanks.

Nathan Palmer
  • 205
  • 3
  • 7

2 Answers2

0
arp who-has 192.168.1.1 tell 192.168.1.23
arp reply 192.168.1.1 is-at MAC

the arp reply's MAC doesn't look right. ... The MAC it lists is the one for the router

As it should. The VM asked for the MAC address of the router.

I'm guessing something on the FortiGate is setup wrong.

Can you swap the addresses on 192.168.1.22 and 192.168.1.23? one of two things will happen:

  1. The problem follows .22
  2. .22 works on the other machine and the new machine does not work with .23 either

also, can 192.168.1.22 ping 192.168.1.23? can 192.168.1.23 ping 192.168.1.22? I'm sure they can, but every bit of information helps.

one more thing, does that sniff command on the Fortigate let you specify the interface? Can you do a capture on the external interface?

you should see something like

your.ip.address. -> 208.67.222.222: icmp: echo request
208.67.222.222 -> your.ip.address: icmp: echo reply
Justin
  • 3,856
  • 18
  • 21
  • Watching the external interface I see nothing pinging from .23. Pinging from .22 I see the expected result. – Nathan Palmer Jan 11 '10 at 03:05
  • You were right. If I swap IPs it works. Is swapped .20 with .23 (.22 is live can I can't change it right now.) The previously unworking machine started working and the working machine stopped working. What does this tell me though other than a router problem? – Nathan Palmer Jan 11 '10 at 03:09
  • Have you checked the ARP cache on the switch, router, and firewall? Can you ping from the router or firewall to .23? Is there a rule on the firewall that's blocking outbound traffic from .23? – joeqwerty Jan 11 '10 at 03:12
  • The ARP cache looks ok. I have even tried to clear it. I cannot ping from the router to .23. The rules are the same for both .20 and .23 and there is one rule that is switch -> wan1 for external traffic. I don't have a wan1 -> wan1 rule however. – Nathan Palmer Jan 11 '10 at 03:16
  • Ok. This is strange. I swapped the machines back so .20 had it's original IP back. But then I gave the other machine .24 and it works. Then I realized. All the VMs that are working have even IPs. .10, .20, .22 and .24. Any ideas? – Nathan Palmer Jan 11 '10 at 03:20
  • Is it a managed switch? If so, are there any ACL's on the switch that may be causing the problem? Whether the ip address is even or odd shouldn't make any difference. Since you can't ping .23 from the router try tracing the route from the router to .23 to see what path it's trying to take. – joeqwerty Jan 11 '10 at 04:10
  • Odd/even issues like this may point to a problem with NIC Teaming on the ESXi host - do you have multiple nics on the VM Port Group\vSwitch? – Helvick Jan 11 '10 at 09:36
  • 1
    I don't think the problem is with NIC Teaming in this case. From the sniff on the Fortigate we know that: the icmp packet is making it from the VM to the Fortigate. The icmp packet is NOT making it from the internal interface on the Fortigate to the external Interface. I believe this points to a problem with the rules in the Fortigate. – Justin Jan 11 '10 at 12:28
  • It is using 1 port on the NIC. I am currently not Teaming. There is 1 rule that goes from the internal network to the outside and it's a NAT'd all->all. How does this describe the even/odd issue? – Nathan Palmer Jan 12 '10 at 03:07
0

I found the problem. The issue was how the SSL VPN was setup. I had configured the VPN to use the same IP range. So for example I had configured the regular local network to be

192.168.1.1

Then I configured the SSL to have the range of

192.168.1.[100-105]

When that happened it also put in a default gateway of 0.0.0.0 for the ssl vpn. So in order to fix my problem I assigned the SSL VPN to use this

192.168.2.[100-105]

Then I changed the default gateway for that to be 192.168.2.1. Everything worked fine after that.

Nathan Palmer
  • 205
  • 3
  • 7