2

Here's the situation: I have a bunch of applications that I am deploying to my k8s cluster:

  • a hasura backend (exposed with an nginx ingress)
  • some serverless functions hosted on an openfaas instance
  • an authentication service
  • a vuejs frontend application served on an nginx docker image (exposed with an nginx ingress)

My kubernetes cluster is hosted on Jelastic. I have enabled SSL with a nginx load-balancer in front of my cluster:

enter image description here

Inside of my cluster, I have the usual nginx-ingress-controller.

Now, I am busy trying to implement authentication properly. Following this advice, here's what I'm doing:

  • I have an auth server (fusionauth) deployed inside the cluster with service type ClusterIP
  • I have an openfaas function that performs user login: it accepts a username, a password, and an application id; it returns a JWT and issues a session cookie; the function talks to fusionauth and gets the JWT and refresh token
  • I have a hasura action login that calls the login openfaas function
  • I have a frontend application calling the hasura action login

Basically, my frontend served by an nginx docker image on the k8s cluster talks to the hasura api which talks to the internal openfaas login function which talks to the internal fusionauth. Both the frontend and the hasura api have an ingress defined:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: admin-ui
  namespace: services-108-staging
  labels:
    app.kubernetes.io/instance: admin-ui
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: admin-ui
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: admin-ui-0.0.0
  annotations:
    meta.helm.sh/release-name: admin-ui
    meta.helm.sh/release-namespace: services-108-staging
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Access-Control-Allow-Origin: $http_origin";
    nginx.ingress.kubernetes.io/cors-allow-credentials: 'true'
    nginx.ingress.kubernetes.io/cors-allow-methods: 'PUT, GET, POST, OPTIONS'
    nginx.ingress.kubernetes.io/enable-cors: 'true'
spec:
  rules:
    - host: admin-staging.my-env.the-jelastic-host.com
      http:
        paths:
          - path: /
            backend:
              serviceName: admin-ui
              servicePort: 4000
---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: api
  namespace: services-108-staging
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: services-108-staging
spec:
  rules:
    - host: api-staging.my-env.the-jelastic-host.com
      http:
        paths:
          - path: /
            backend:
              serviceName: api
              servicePort: 8080

Only the frontal nginx load-balancer can access both the admin-staging.my-env.the-jelastic-host.com and the api-staging.my-env.the-jelastic-host.com URLs. The two applications are served through the frontal load-balancer on other domains, like admin-staging.my-host.com and api-staging.my-host.com respectively. The frontal load-balancer re-routes its incoming requests like this (e.g. for the admin-staging url):

server {
  server_name admin-staging.my-host.com;

  listen 443 ssl;

  [...]
  
  location / {
    [...]
    proxy_pass http://admin-staging.my-env.the-jelastic-host.com;
  }
}

My goal is to get the session cookie stored in my frontend application. Currently, while I get the Set-Cookie header upon calling the login graphql mutation:

enter image description here

I don't get it in my frontend application's cookies:

enter image description here

I don't know exactly what I should set where to make that session cookie appear in my frontend's storage. Everything is served over https: the hasura api and the frontend application. Should I set a particular domain in my cookie? The cookie is issued from inside of the k8s cluster, from my serverless function hosted on

auth.services-108-staging.svc.cluster.local

(that's the result of the .Net Core call to Request.Host.ToUriComponent()). Should I set a particular domain in my cookie? Should I setup session affinity? On what ingress should I define session affinity? Is that enough to define session affinity in my cluster? What about the nginx load-balancer in front of my cluster? In essence, that load-balancer routes requests to the k8s cluster. Do I have to setup something special there?

Laurent Michel
  • 1,069
  • 3
  • 14
  • 29

2 Answers2

0

Looks like session affinity is a good option to try first.

Could you please try to add the following annotations to all your ingress rules and check if that can help you to solve the issue?

nginx.ingress.kubernetes.io/affinity     cookie
nginx.ingress.kubernetes.io/affinity-mode   persistent
nginx.ingress.kubernetes.io/session-cookie-name hasura-backend
nginx.ingress.kubernetes.io/session-cookie-change-on-failure    true

Alternatively, you can try to eliminate https2http termination that most likely is a reason that your FE application doesn’t see the cookie.

On a screenshot, it is seen that Set-Cookie header has a secure flag set. In order to eliminate SSL termination, you can attach Public IP addresses to your worker nodes of K8s, then activate SSL annotation in ingress and have a direct https connection to your frontend.

Virtuozzo
  • 1,993
  • 1
  • 10
  • 13
  • So, I've tried one first thing: because I'm not sure it's clear in my original post, let me clarify that I have full power on the emission of the cookie. In particular, I can remove the httponly and secure flags, for testing purposes. It might be much easier to start looking if it actually works without some scary https configurations. That first attempt failed with the same result as before. I will soon try to apply the annotations you propose and let you know. Sorry for the delay, I was on holiday last week. – Laurent Michel Jul 06 '20 at 08:42
  • @LaurentMichel, did you try this solution ? – mario Mar 15 '21 at 18:39
  • Ok I was able to completely re-setup my k8s cluster. I've removed my frontal load-balancer and put an IPv4 on each worker node. I've activated the cert-manager. I've tried to put the nginx annotations you suggest on my frontend and hasura ingresses, but I still have no cookie in my frontend application. – Laurent Michel May 05 '21 at 03:27
0

The Set-Cookie comes from a different domain, so the browser does nothing about it. The key phrase for your problem is "cross-origin set cookie".

rollingBalls
  • 1,808
  • 1
  • 14
  • 25