1

We have a really strange issue with DNS on our EC2 instances. An application we're running on some of our instances is Java-based service that provides a back-end for an Android application. As part of it's functionality, it sends push notifications to phones through a Google API. To do this, it has to make an SSL request to android.apis.google.com. Unfortunately, when running this on our EC2 instances, we get a certificate error because the hostname doesn't match the certificate name:

hostname in certificate didn't match: <android.apis.google.com> != <.gstatic.com> OR <gstatic.com> OR <.gstatic.com>

We have tracked the problem down to a difference in DNS results. When we query DNS for android.apis.google.com from our office (where everything works), we get back the following:

android.apis.google.com. 300    IN      CNAME   clients.l.google.com.
clients.l.google.com.   160     IN      A       74.125.226.230
clients.l.google.com.   160     IN      A       74.125.226.231
clients.l.google.com.   160     IN      A       74.125.226.232
clients.l.google.com.   160     IN      A       74.125.226.233
clients.l.google.com.   160     IN      A       74.125.226.238
clients.l.google.com.   160     IN      A       74.125.226.224
clients.l.google.com.   160     IN      A       74.125.226.225
clients.l.google.com.   160     IN      A       74.125.226.226
clients.l.google.com.   160     IN      A       74.125.226.227
clients.l.google.com.   160     IN      A       74.125.226.228
clients.l.google.com.   160     IN      A       74.125.226.229

When we perform the same query from an EC2 server, we get back a different set of DNS results:

android.apis.google.com. 300    IN      CNAME   clients.l.google.com.
clients.l.google.com.   300     IN      A       72.14.204.138
clients.l.google.com.   300     IN      A       72.14.204.100
clients.l.google.com.   300     IN      A       72.14.204.101
clients.l.google.com.   300     IN      A       72.14.204.102
clients.l.google.com.   300     IN      A       72.14.204.113

Any ideas why the DNS results would be so dramatically different out at EC2? And, more importantly, how we can fix this?

We did try using a custom hostname validator. According to our developers, that allowed the connection to proceed, but the problem is that it's connected to the wrong server, so the request still fails.

organicveggie
  • 1,071
  • 3
  • 15
  • 27
  • DNS results are different, as it is served from a CDN. All the different IPs are different edge nodes. – Shyam Sundar C S Mar 15 '12 at 19:37
  • Actually, it's not a CDN. And while your fundamental point is true, it totally misses the boat. The DNS results retrieved by servers at EC2 are pointing at servers don't support the CD2M service in question. Regardless of whether the edge nodes from Google are there for load balancing or geographic distribution, the endpoints don't work from EC2 instances. – organicveggie Mar 15 '12 at 20:12
  • Why would Google send invalid IPs for `android.apis.google.com`/`clients.l.google.com` to EC2 and my (Time Warner RoadRunner) ISP? You sound a little like you're rushing to an unsupported conclusion. – ceejayoz Mar 15 '12 at 20:53
  • Incidentally, `curl -d "test=test" https://android.apis.google.com/c2dm/send` appears to correctly get the `401 Unauthorized` the documentation says I should. This is against the IPs you claim are non-functional. `curl -d "test=test" https://android.apis.google.com/c2dm/test` returns 404, so something's definitely there. – ceejayoz Mar 15 '12 at 20:58
  • I'm not saying there is nothing there. I'm just following up on a developer complaint who indicated that despite using a custom hostname validator, the request still failed. And the exact same code works perfectly from our office. – organicveggie Mar 17 '12 at 21:44

3 Answers3

2

Google has lots and lots of IPs, and they'll serve you the IPs that are geographically closest to where they think you're located.

When I dig against Google's own public DNS at 8.8.8.8, I get the same list you get from EC2:

[] csternal@~: dig @8.8.8.8 android.apis.google.com

; <<>> DiG 9.6-ESV-R4-P3 <<>> @8.8.8.8 android.apis.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38592
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;android.apis.google.com.   IN  A

;; ANSWER SECTION:
android.apis.google.com. 300    IN  CNAME   clients.l.google.com.
clients.l.google.com.   300 IN  A   72.14.204.102
clients.l.google.com.   300 IN  A   72.14.204.113
clients.l.google.com.   300 IN  A   72.14.204.100
clients.l.google.com.   300 IN  A   72.14.204.101
clients.l.google.com.   300 IN  A   72.14.204.138

;; Query time: 83 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Mar 15 11:38:13 2012
;; MSG SIZE  rcvd: 145

The problem you're encountering appears to be this one: https://stackoverflow.com/questions/6296547/c2dm-ioexception-when-sending-message

The solution most commonly given to resolve the error message you are receiving is to define a custom Hostname validation. The main problem that you are facing is that the domain name returned by Google's Android URL is .google.com. Unfortunately, this causes some issues as the Android SDK is at android.apis.google.com. The JVM will not validate this combination by default (.sdk.google.com would be acceptable).

Here is an example of how you can create your own hostname validator:

URL url = new URL("https://android.apis.google.com/c2dm/send");

HostnameVerifier hVerifier = new HostnameVerifier() {
    public boolean verify(String hostname, SSLSession
            session) {
        return true;
    }
};

HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setHostnameVerifier(hVerifier);
Community
  • 1
ceejayoz
  • 32,910
  • 7
  • 82
  • 106
  • My developers tell me that yes, that allows the connection to proceed. Unfortunately, those are still the wrong servers, so the request still fails. – organicveggie Mar 15 '12 at 20:08
  • How are you determining that they're the wrong servers? The DNS records are provided directly by Google's public DNS. Every single one of those IPs goes correctly to Google.com when accessed directly. – ceejayoz Mar 15 '12 at 20:48
  • Our developer indicated that despite using a custom hostname validator, the request still failed. And the exact same code works perfectly from our office. – organicveggie Mar 17 '12 at 21:44
  • Your developer should do some further investigation - like what error's being returned - because this doesn't appear to be a DNS issue at all. – ceejayoz Mar 18 '12 at 00:24
0

Strange. Everything started working again - like magic.

organicveggie
  • 1,071
  • 3
  • 15
  • 27
-1

It may be a problem coming from Amazon EC2 DNS platform (refresh problem) or a caching results issue coming from both, your "Virtual plateform" or Amazon platform.

Dr I
  • 955
  • 17
  • 33