5

I am getting Could not resolve host metadata.google.internal or Could not resolve host metadata error when I try to access Metadata Server.

Although I use my own VPC with custom firewall, I don't think this is the issue of firewall because according to Google Metadata Server traffic never leaves the Instance. Besides, there's an entry of it in my host file which proves it:

# Google Compute Engine metadata server 169.254.169.254 metadata.google.internal metadata

I tried pinging 169.254.169.254, but it returns General Failure (but, it proves nothing.. I am not sure if Metadata Server allows ICMP).

My Stackdriver Logging Agents are also failing because of this. Here's a line from its log:

Failed to access metadata service: error_class=Errno::ENETUNREACH error="Failed to open TCP connection to 169.254.169.254:80 (A socket operation was attempted to an unreachable network. - connect(2) for \"169.254.169.254\" port 80)"

In case if it matters, the internal traffic in my VPC network is limited to icmp and tcp:22 (ssh) only.

What's happening and how to fix this issue?

user71823
  • 161
  • 1
  • 1
  • 9

4 Answers4

4

Although I use my own VPC with custom firewall, I don't think this is the issue of firewall because according to Google Metadata Server traffic never leaves the Instance.

GCP metadata traffic never leaves the physical host running the instance. Such requests go out the guest's interfaces, but never get forwarded.

A host level firewall in the instance can prevent traffic from reaching the metadata server. As it drops packets in the instance's network stack before they leave the guest.

Normally the metadata service responds to http, DNS, and ICMP echo, and all of them are not working for you. Only one hop away so not like it can be routed incorrectly. Sure sounds like a firewall.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
3

This can be caused by disabling the service account for the instance. You need authorization to access metadata.

In the Google Cloud Console:

  • Go to Compute Engine -> VM instances.
  • Stop the instance.
  • Click edit and scroll down to "Service account".
  • Choose the correct service account, usually "Compute Engine default service account".
  • Under Access scopes, choose your desired configuration.

I recommend either "Allow default access" or "Allow full access to all Cloud APIs". The real access control is determined by the roles assigned to the service account. The "Access scopes" selection can only limit these roles and never increase them.

Next, get rid of your custom entry for metadata in your hosts file.

John Hanley
  • 4,754
  • 1
  • 11
  • 21
1

You need to set the metadata flavor in the curl header,

curl -v -w "\n" -H "Metadata-Flavor: Google" \ 
  http://169.254.169.254/computeMetadata/v1/instance/machine-type
Flannon
  • 11
  • 1
1

To fix the issue, you can try to add manually the internal DNS, you can do this by editing the file resolv.conf that you will find in the path: /etc/resolv.conf

The line should look like: nameserver 169.254.169.254,

For more details, check this documentation

I can see from your post, that you have already added the host:

Google Compute Engine metadata server

169.254.169.254    metadata.google.internal metadata

Additionally, I suggest you to verify that you have installed the guest environment, you can read more about this and how to do it in this documentation.

Jorge P.
  • 21
  • 3
  • What DNS has to do if Stackdriver agents can't connect to Metadata Server even with IP? – user71823 Sep 09 '19 at 19:39
  • 1
    Do not modify /etc/resolv.conf for Google Compute Engine VM instances. This file is managed by Google DHCP. Any changes you make will be overwritten on the next DHCP refresh. – John Hanley Sep 16 '19 at 18:39