1

Yesterday I spun up an Azure Kubernetes Service cluster running a few simple apps. Three of them have exposed public IPs that were reachable yesterday.

As of this morning I can't get the dashboard tunnel to work or the LoadBalancer IPs themselves.

I was asked by the Azure twitter account to solicit help here.

I don't know how to troubleshoot this apparent network issue - only az seems to be able to touch my cluster.

dashboard error log

❯❯❯ make dashboard ~/c/azure-k8s (master) az aks browse --resource-group=akc-rg-cf --name=akc-237 Merged "akc-237" as current context in /var/folders/9r/wx8xx8ls43l8w8b14f6fns8w0000gn/T/tmppst_atlw Proxy running on http://127.0.0.1:8001/ Press CTRL+C to close the tunnel... error: error upgrading connection: error dialing backend: dial tcp 10.240.0.4:10250: getsockopt: connection timed out

service+pod listing

❯❯❯ kubectl get services,pods ~/c/azure-k8s (master) NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE azure-vote-back ClusterIP 10.0.125.49 <none> 6379/TCP 16h azure-vote-front LoadBalancer 10.0.185.4 40.71.248.106 80:31211/TCP 16h hubot LoadBalancer 10.0.20.218 40.121.215.233 80:31445/TCP 26m kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 19h mti411-web LoadBalancer 10.0.162.209 52.168.123.30 80:30874/TCP 26m

NAME READY STATUS RESTARTS AGE azure-vote-back-7556ff9578-sjjn5 1/1 Running 0 2h azure-vote-front-5b8878fdcd-9lpzx 1/1 Running 0 16h hubot-74f659b6b8-wctdz 1/1 Running 0 9s mti411-web-6cc87d46c-g255d 1/1 Running 0 26m mti411-web-6cc87d46c-lhjzp 1/1 Running 0 26m

http failures

❯❯❯ curl --connect-timeout 2 -I http://40.121.215.233 ~/c/azure-k8s (master) curl: (28) Connection timed out after 2005 milliseconds ❯❯❯ curl --connect-timeout 2 -I http://52.168.123.30 ~/c/azure-k8s (master) curl: (28) Connection timed out after 2001 milliseconds

Daniel J. Pritchett
  • 1,061
  • 11
  • 18
  • 1
    We have had similar problems that were solved by rebooting the nodes from the Azure portal. I don't know if it helps and it hides the root cause, but it may get your cluster back up. – ewramner Mar 29 '18 at 16:35
  • That worked, thanks! I'm not sure what to think about that but it did help. – Daniel J. Pritchett Mar 29 '18 at 16:38

2 Answers2

2

If you are getting getsockopt: connection timed out while trying to access to your AKS Dashboard, I think deleting tunnelfront pod will help as once you delete the tunnelfront pod, this will trigger creation of new tunnelfront by Master. Its something I have tried and worked for me.

Arjun Adhikari - MSFT
  • 81
  • 5
0

@daniel Did rebooting the agent VM's solve your issue or are you still seeing issues?

sauryadas_
  • 56
  • 1
  • 1
    The issue hasn't come back but I am a bit nervous about it - I need my k8s cluster to have externally visible IPs if I'm going to be able to use it for client work! – Daniel J. Pritchett Apr 04 '18 at 12:07