2

I'm using python kubernetes 3.0.0 library and kubernetes 1.6.6 on AWS.

I have pods that can disappear quickly. Sometimes when I try to exec to them I get ApiException Handshake status 500 error status.

This is happening with in cluster configuration as well as kube config.

When pod/container doesn't exist I get 404 error which is reasonable but 500 is Internal Server Error. I don't get any 500 errors in kube-apiserver.log where I do find 404 ones.

What does it mean and can someone point me in the right direction.

3h4x
  • 936
  • 8
  • 14

3 Answers3

6

I know that this question is a little old, but I thought I would share what I found when trying to use python/kubernetes attach/exec for several debugging cases (since this isn't documented anywhere I can find).

As far as I can tell, it's all about making the keyword arguments match the actual container configuration as opposed to what you want the container to do.

When creating pods using kubectl run, if you don't use -i --tty flags (indicating interactive/TTY allocation), and then attempt to set either the tty or stdin flags to True in your function, then you'll get a mysterious 500 error with no other debug info. If you need to use stdin and tty and you are using a configuration file (as opposed to run), then make sure you set the stdin and tty flags to true in spec.containers.

While running resp.readline_stdout(), if you get a OverflowError: timestamp too large to convert to C _PyTime_t, set the keyword argument timeout=<any integer>. By default, the timeout variable defaults to None, which is an invalid value in that function.

If you run the attach/exec command and get an APIException and a status code of 0, the error Reason: hostname 'X.X.X.X' doesn't match either of..., note that there appears to be an incompatibility with Python 2. Works in Python 3. Should be patched eventually.

I can confirm 404 code is thrown via an ApiException when the pod doesn't exist.

If you are getting a mysterious error saying upgrade request required, note that you need to use the kubernetes.stream.stream function to wrap the call to attach/exec. You can see this issue on GitHub and this example code to help you get past that part.

Here's my example: resp = kubernetes.stream.stream(k8s.connect_get_namespaced_pod_attach, name='alpine-python-2', namespace="default", stderr=True, stdin=True, stdout=True, tty=True, _preload_content=False)

Note that the _preload_content=False is essential in the attach command or else the call will block indefinitely.

I know that was probably more information than you wanted, but hopefully at least some of it will help you.

TopherGopher
  • 655
  • 11
  • 21
1

For me the reason was, I had two pods, with same label attached, 1 pod was in Evicted state and other was running , i deleted that pod, which was Evicted and issue was fixed

HERAwais
  • 357
  • 5
  • 21
0

For me, The reason for 500 was basically pod unable to pull the image from GCR

JD-V
  • 3,336
  • 1
  • 17
  • 20