Weird 302 response from apache server

Question

I have this situation that utterly baffles me:

The setup

a newly setup server (centos 7, apache, mysql, nothing fancy) that hosts a simple php app that I need to interact with from my main app on another server. This service is setup to run on service-name.domain.tld while the main app is on domain.tld (just mentioning in case it make any difference).

The problem

For some reason when I try to access the service app from the main server I get an endless loop of 302 redirects.

If I do a curl -D - http://service-name.domain.tld from the main server I get:

HTTP/1.1 302 Found
Date: Wed, 03 Aug 2016 12:30:26 GMT
Server: Apache
Location: http://service-name.domain.tld
Content-Length: 218
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://email-blasts.upgradesale.co">here</a>.</p>
</body></html>

If I do the same command from my computer, the result is

HTTP/1.1 200 OK
Date: Wed, 03 Aug 2016 12:31:39 GMT
Server: Apache/2.4.6 (CentOS) PHP/5.6.24
X-Powered-By: PHP/5.6.24
Cache-Control: no-cache
Content-Length: 30
Content-Type: text/html; charset=UTF-8
Welcome.

As it should be. Same happens for any request I make (even static files). Also I should specify same situation happens for wget and get_file_contents in php

I'm really lost right now, not knowing where to go from here. So any kind of direction is greatly appreciated.

UPDATE #1:

I should mention the 2 servers are not on the same network. The main app is hosted on LiquidWeb and the new server is on Linode.

Also to point the subdomain to the Linode server I created an A record in CPanel's DNS editor.

Running ping from the domain.tld server yields:

64 bytes from li1014-180.members.linode.com (xx.xx.xx.xxx): icmp_seq=1 ttl=64 time=0.022 ms

Same command from my local computer:

64 bytes from (xx.xx.xx.xxx): icmp_seq=0 ttl=52 time=114.982 ms

Following Damiano's instructions running tcpdump I found out that requests from the main server don't reach the new server, instead seems like the main server also responds. The quest continues to figure out why.

Also the main server is running CPanel so that might have something to do with this...

UPDATE 2

While running sudo tcpdump -n -i eth0 icmp on the service server, if I ping service-name.domain.tld from my local machine, I can see the traffic coming in and out on the server, but same command run from the main server gives me nothing on the service server (same goes if I filter for http traffic with tcpdump, and making curl requests).

UPDATE 3

I gave up and cloned the Linode, got a new IP and now it works fine... ¯_(ツ)_/¯

Maybe an (incorrect) entry in the hosts file versus the actual DNS record? — HBruijn, Aug 03 '16 at 12:59
"_I found out that requests from the main server don't reach the new server, instead seems like the main server also responds..."_: please, elaborate on this as it's not really clear, to me, what you mean. Which "tcpdump" have you run (on the server) and which command (on the client) have you launched to generate traffic? What does it mean "_the main server also respond_"? — Damiano Verzulli, Aug 03 '16 at 19:22

score 2 · Answer 1 · answered Aug 03 '16 at 14:47

You asked for "directions" so... here are mine.

In your two transcripts I saw, in the first one, this:

[...]
Server: Apache
[...]

and in the second one, this:

[...]
Server: Apache/2.4.6 (CentOS) PHP/5.6.24
[...]

Even tough those are two different HTTP response (302 Found the first; 200 OK the second), I bet that those two responses are coming from different web servers. So let's try to investigate such an hypothesis...

As a first step, as already mentioned in the @HBruijn comments, we need to ensure that both HTTP clients are opening the HTTP connection towards the same HTTP server. On each of them, please, run:

ping service-name.domain.tld

and ensure that in the output you're seeing the same IP address. If not, let's stop here and... is up to you to step forward on a different issue :-)

If both ping resolve to the same IP address, than the troubleshooting process is really different, based on the "relative" location of the three hosts: are "Client A" and "Client B" connected to the same Ethernet LAN/segment? is "Server C" connected to the same Ethernet LAN/segment?

If the three machines are connected to the same LAN, than something to check could be the MAC address that both "Client A" and "Client B" gets from the ARP resolution of the "Server C" IP address. Right after the PING (on "Client A" and "Client B" versus "Server C"), runs (on both "Client A" and "Client B") an arp -an <ip.of.server.c>. You'll get something like:

verzulli@iMac-Chiara:~$ ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
[...]
verzulli@iMac-Chiara:~$ arp -an 192.168.2.1
? (192.168.2.1) associato a c0:c1:c0:e8:0f:12 [ether] su wlan0
verzulli@iMac-Chiara:~$

Here above you see that the MAC address of my "server" (192.168.2.1) is c0:c1:c0:e8:0f:12. In your case, ensure it's the same, for both "Client A" and "Client B". Again, please note that this is valid only if Client A and Client B are in the same ethernet segment of Server C.

If "Client A" and "Client B" are on different networks and/or "Server C" is far from them, then our only chance is to check what's coming to "Server C" when both "Client A" and "Client B" try to reach it. Unfortunately, here also we have a different approach depending on the relative location of the three hosts. Let's suppose the worst-case scenario: "Client A" and "Client B" connected to two different LANs (maybe two remote sites of a big-company) and "Server C" is connected to a third site (big-company headquarter). Both "Client A" and "Client B" are NATted, when reaching "Server C".

In such a case, we need to discover the "public" IP address assigned to "Client A" and "Client B" when reaching "Server C". There are plenty of ways to get this info. As a quick (...and very dirty) way to get result, you can launch:

verzulli@iMac-Chiara:~$ curl -s http://ipinfo.io | grep '"ip":'
  "ip": "2.239.77.181",

where you can see that my NAT-address (assigned by my provider to my host when leaving its network) is 2.239.77.181.

So, now, I can finally ask "Server C" to show me the traffic coming from my IP and.... just watch if something is really coming into.

On the server:

[root@srv-01 ~]# tcpdump -n -i eth0 host 2.239.77.181 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

it will.... start waiting, for incoming icmp (ping) packets coming from 2.239.77.181. As soon as I, from the client, will PING, such packets will be shown:

[root@srv-01 ~]# tcpdump -n -i eth0 host 2.239.77.181 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:28:21.031355 IP 2.239.77.181 > 78.47.127.152: ICMP echo request, id 60237, seq 1, length 64
16:28:21.031392 IP 78.47.127.152 > 2.239.77.181: ICMP echo reply, id 60237, seq 1, length 64

where you can see the PING (icmp echo request) coming from my client and, righafter, the reply from the server (icmp echo reply).

Should you prefer to check for every kind of traffic (and not only ICMP), you could try (on "Server C") with a:

tcpdump -n -i eth0 host 2.239.77.181

of, if you only want to check for HTTP traffic:

tcpdump -n -i eth0 host 2.239.77.181 and port 80

That's all.

Based on the above, you should be able to check if both "Client A" and "Client B" are reaching your "Server C" (and not two different web servers). Once got this info, get back here confirming this and... I could update this answer to step over.

Thank you very much for your time. I followed you instructions and it appears my request from the main server reaches back to the same server. — Dan F., Aug 03 '16 at 16:07