0

Context

In a kubernetes cluster, I have a postrgreSQL pod running, exposed to the cluster via a ClusterIP service. I use the integrated DNS resolution to reach it from an other pod (.NET Core) within the same namespace.

My deployments are done through gitlab-ci.

I would like to run the dotnet database update command to apply DB migrations as a job in the gitlab pipeline rather than run that command from within the web application pod after deploy, because the migration application are prone to fail and if that happened after the pod's deployment I would get an unreachable web application. Thus, running it as a job in the pipeline would prevent the pod's deployment and keep the "safe" version running.

What's not working

The postgres pod is not exposed outside of the cluster, thus the gitlab pipeline can't reach it (even worse, the .net application file expect the k8s DNS resolution to work, so the server url is juste "postgres")

My options

  • I could temporarily expose the pod publicly by changing the service type to node-port, but that's a lot of trouble, the IP might change from one pipeline run to the other, it take a long time to attribute the IP. This seems like a poor, slow and unstable solution overall.
  • I could use the kubectl port-forward service/postgres 5432:5432 command, and that is what I do in localhost. The problem is that it "blocks" the command line until a exit signal is sent, so my dotnet command could not be run while the ports are bound. Despite extensive research I have not found any k8s-native way to detach this command and have not found any way to do this in gitlab-ci either.
  • I could launch an ephemeral pod from the pipeline that runs on a dotnet-sdk image with dotnet-tools installed, clone the repository, run the dotnet-ef command and then destroy the container. This solution seems to be way over-engineered and would make me loose a lot of the advantages of running from the gitlab pipeline(GIT repository auto-cloning, mainly).

Question

Is there any 4th option that would - Be integrated within the gitlab pipeline, so that a failed migration prevents the deployments from being applies - Is fast (not wait for IP adressing to be attribute and avoid any delay longer than a few seconds before the command can be launch) - Is repeatable/reliable (for example in the NodePort case, what if the IP adress fails to be attributed?) - Is simple to implement, understand and maintain

Mathieu VIALES
  • 101
  • 1
  • Note: I am *somewhat* new to gitlab & kubernetes, there might be a differnt aprocha I haven't considered out of ignorance. I am open to anything – Mathieu VIALES Jun 10 '20 at 17:14
  • 1
    Hi Mathieu, welcome to S.F. Two comments: first, it is not `kubectl`'s responsibility to do shell job control, so what you want is `kubectl port-forward ... &` which will set `$!` as the background job's pid for cleanup later; second, have you considered having an in-cluster gitlab-runner, which would help with some of those connectivity issues? – mdaniel Jun 10 '20 at 19:19
  • I ... did not think about it. I'll look into it ! – Mathieu VIALES Jun 10 '20 at 19:23
  • [mdaniel](https://serverfault.com/users/330715/mdaniel) seems plausible. Have you tested it? – Mark Watney Jun 12 '20 at 11:22

0 Answers0