Kubernetes prefect UI cannot connect to api

Question

So I am having an issue with prefect-server UI connecting to my API within my kubernetes environment. I am using a nginx as my ingress.

I have my ingress configuration for the prefect-server service as so:

https://myhost.com/prefect/ -> http://localhost:3000/api

I am able to access the UI, but it saying it cannot connect to the api:

deployment.yml for prefect-server:

...
      - image: <my registry>
        imagePullPolicy: Always
        name: prefect-server
        command: ["prefect", "server", "start", "--host", "0.0.0.0", "--log-level", "WARNING", "--port", "3000"]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_UI_API_URL 
            value:  "http://localhost:3000/api"

deployment.yml for agent:

...
      - image: <my registry>
        imagePullPolicy: Always
        name: prefect-agent
        command: ["prefect", "agent", "start", "--work-queue", "dev"]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_API_URL 
            value:  "http://prefect-server.prefect.svc.cluster.local:3000/api" #WORKS

Here is the output of kubectl describe service -n prefect

Name:              prefect-server
Namespace:         prefect
Labels:            app.kubernetes.io/managed-by=Helm
                   run=prefect-server
Annotations:       meta.helm.sh/release-name: prefect
                   meta.helm.sh/release-namespace: prefect
Selector:          run=prefect-server
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                11.11.11.11 <redacted>
IPs:               11.11.11.11 <redacted>
Port:              <unset>  80/TCP
TargetPort:        3000/TCP
Endpoints:         11.22.22.22:3000 <redacted>
Session Affinity:  None
Events:            <none>

The agent is able to connect to the service container with no issues. For the prefect-server PREFECT_UI_API_URL, I have tried:

"http://prefect-server.prefect.svc.cluster.local:3000/api"
"https://myhost.com/prefect/api"
"http://127.0.0.1:3000/api"

UPDATE: Thank you VonC for your help, I think I am heading in the right direction. I made the changes to the PREFECT_UI_API_URL and ingress. Also had to change the agent url to http://prefect-server.prefect.svc.cluster.local:3000/prefect/api. However, the "cannot to to Server API.." error goes away but the page is still 404 when I visit htts://myhost.com/prefect/flow-runs endpoint.

Output for kubectl describe ingress -n prefect

Name:             prefect
Namespace:        prefect
Address:          X.X.X.50
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  /mysecret terminates 
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  myhost.com  
                                  /prefect       prefect-server:80 (X.X.X.131:3000)
                                  /prefect/api   prefect-server:3000 (X.X.X.131:3000)
Annotations:                      field.cattle.io/publicEndpoints:
                                    [{"addresses":["X.X.X.50"],"port":80,"protocol":"HTTP","serviceName":"prefect:prefect-server","ingressName":"prefect:prefect","hostname"...
                                  ingress.kubernetes.io/secure-backends: true
                                  ingress.kubernetes.io/ssl-redirect: true
                                  kubernetes.io/ingress.class: f5
                                  meta.helm.sh/release-name: prefect
                                  meta.helm.sh/release-namespace: prefect
                                  virtual-server.f5.com/balance: round-robin
                                  
                                  virtual-server.f5.com/http-port: 443
                                  virtual-server.f5.com/ip: X.X.X.50
                                  virtual-server.f5.com/partition: kubernetes

My ingress.yml:

{{- if .Values.ingress.enabled }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prefect
  annotations:
{{- if eq .Values.ingress.class "f5" }}
    {{- if .Values.ingress.tls }}
    ingress.kubernetes.io/secure-backends: "true"
    {{- end }}
    {{- $f5healthpath := printf "%s%s" .Values.ingress.hostname .Values.ingress.path }}
    {{- $f5path := trimPrefix "/" .Values.ingress.path }}
    {{- $f5partition := .Values.ingress.f5partition }}
    {{- $f5httpport := .Values.ingress.f5httpport}}
    {{- $f5ip := .Values.ingress.f5ip }}
    {{- $f5ssl := .Values.ingress.f5ssl }}
    {{- $f5class:= .Values.ingress.class}}
    {{- range $key, $value := .Values.ingress.annotations }}
    {{ $key }}: {{ $value | replace "F5HEALTHPATH" $f5healthpath | replace "F5PATITION" $f5partition | replace "F5HTTPPORT" $f5httpport | replace "F5IP" $f5ip | replace "F5SSL" $f5ssl | replace "F5CLASS" $f5class | replace "F5PATH" $f5path | quote }}
    {{- end }}
{{- else }}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
{{- end }}
spec:
  tls:
  - secretName: /mysecret
  rules:
  - host: "myhost.com"
    http:
      paths:
      - path: "/prefect"
        pathType: Prefix
        backend:
          serviceName: prefect-server
          servicePort: 80
      - path: "/prefect/api"
        pathType: Prefix
        backend:
          serviceName: prefect-server
          servicePort: 3000       
{{- end }}

I believe the issue may be within my ingress, I would like all the paths to the prefect-server have the prefix path /prefect/

VonC · Accepted Answer · 2023-07-10T19:41:40.967

Double-check the PREFECT_UI_API_URL in your prefect-server deployment.
This variable is used by the Prefect UI to understand where to find the API.

However, using "localhost" or "127.0.0.1" or "prefect-server.prefect.svc.cluster.local" would only be applicable if the UI (browser) is running on the same machine where the server is running, which is generally not the case.

When using "localhost" or "127.0.0.1", the browser will try to connect to the local machine, not the server.
In the context of Kubernetes, "localhost" or "127.0.0.1" will refer to the client's machine (the machine where the browser is running), not the machine where your Kubernetes cluster is running.
Similarly, "prefect-server.prefect.svc.cluster.local" is only reachable within the Kubernetes cluster network. This internal network is not exposed to the internet and therefore not accessible from a client machine outside of that network.

So, when you are setting the PREFECT_UI_API_URL environment variable, you need to provide an external, publicly accessible URL that the browser can reach over the internet. This URL should point to the external endpoint that's been created for your Prefect API service, typically using an Ingress controller like nginx.
Assuming you have set up ingress to route https://myhost.com/prefect/api to your Prefect API, you should use this URL as the value of PREFECT_UI_API_URL.

Try updating your prefect-server deployment like so:

...
      - image: <my registry>
        imagePullPolicy: Always
        name: prefect-server
        command: ["prefect", "server", "start", "--host", "0.0.0.0", "--log-level", "WARNING", "--port", "3000"]
        workingDir: /opt/prefect
        env:
          - name: PREFECT_UI_API_URL 
            value:  "https://myhost.com/prefect/api"

And you need to ensure that your ingress routes https://myhost.com/prefect/api to the Prefect API service in your Kubernetes cluster properly. Double-check your nginx ingress configuration, it should be something like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prefect-ingress
spec:
  rules:
  - host: "myhost.com"
    http:
      paths:
      - pathType: Prefix
        path: "/prefect/api"
        backend:
          service:
            name: prefect-server
            port:
              number: 3000

Do replace myhost.com with your actual domain name and check if the service name and port number match your cluster configuration.

From the updated information you have provided, it seems you have made good progress. However, you have correctly identified that there may still be an issue with your Ingress configuration. It seems like the nginx rewrite target annotation is missing, which is important when using the ingress controller to rewrite the requested URLs.

The annotation nginx.ingress.kubernetes.io/rewrite-target: /$2 will replace the URL path from the incoming request to match the target service in the Kubernetes cluster. That annotation is important for a scenario where you want to add a prefix path to your service (in your case, "/prefect/").

Consider this part of your ingress.yaml:

{{- else }}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
{{- end }}

That block seems to be intended for an nginx ingress, and the rewrite target is set correctly here for an nginx ingress. However, if you are using an F5 ingress as indicated by your configuration (kubernetes.io/ingress.class: f5), this block will not be applied, and the nginx-specific rewrite rules will not be effective.

If you are using the F5 ingress controller, it may not recognize the nginx.ingress.kubernetes.io/rewrite-target: /$2 annotation, since it's specific to the nginx ingress controller. Therefore, the F5 ingress controller may not have the ability to rewrite the URL path as expected.

You may need to consult the F5 documentation or support for guidance on how to perform URL rewrites or equivalent configurations to achieve your goal. I would check if F5 provides any URL rewriting capabilities similar to nginx's rewrite-target: F5 may have a different mechanism or annotation for doing this, which would need to be configured accordingly to match your desired /prefect/ path prefix.

If the F5 controller doesn't provide this functionality or if you find it easier to configure with nginx, then switching to the nginx ingress controller might be an option to consider.

thank you, I think I am on the right path. I am not longer getting the `cannot to to Server API..` error but instead just getting 404 when I try to go to /prefect/ or /prefect/flow-runs, ect... I updated my original post with my ingress code and ingress description — cozzymotto, Jul 10 '23 at 18:18
@cozzymotto OK, I have updated the answer to address your comment. — VonC, Jul 10 '23 at 19:41
Thank you for providing this information, I will look into the documentation. The bounty is yours. — cozzymotto, Jul 10 '23 at 20:03

Kubernetes prefect UI cannot connect to api

1 Answers1