1

I have an API as a Kubernetes service, and I need to run an endpoint in the API every hour. (heavy statistic calculation, takes around 3-5 minutes)

Currently I am using curl to call the end point directly:

containers:
- name: callout
    image: 'buildpack-deps:curl'
    args:
        - /bin/sh
        - '-ec'
        - 'curl http://api-service/v1/Stats/CalculateStats'

The problem is the task is sent to service, and it ends up on one of the pods. Running the calculation keeps the pod busy, and the other requests coming from regular users via front-end slow down.

How can I create a dedicated pod from the same API image with a higher CPU request (so it can run faster) and run the calculation on it, then remove the pod and repeat the process on the next schedule?

Thanks

dvdmn
  • 6,456
  • 7
  • 44
  • 52
  • Instead of running `curl`, can the CronJob launch a pod whose `image:` is the same as your normal application, but with a different `command:` that does the update task? – David Maze Apr 25 '21 at 23:45
  • @DavidMaze I am not sure does .Net Core support it or not. I will check, thanks – dvdmn Apr 26 '21 at 14:02

1 Answers1

2

This is what came to my mind:

  1. Create a separate deployment for your cronjob task. This deployment would be basically identical to the production deployment, but with a higher CPU claim and (important!) a separate tag. Set the replica count to 0 by default. This deployment can be updated whenever the production deployment is updated, so both deployments run the same version of the appplication.
  2. Create a separate service that binds to all pods to the newly created tag in 1.. This service should not be expopsed to the public (i.e. the internet).

In the cronjob:

  • Scale up the deployment created in 1. to 1
  • Wait for the pod of the deployment to become ready
  • Execute the curl-command
  • Scale down the deployment created in 1. back to 0

This will ensure that the additional resources are only consumed when the expensive request is executed. A downside of this approach is that the deployment of the pod may fail if no sufficient resources are available, so we should monitor the cluster and keep some resource "ready" for when the deployment is scaled up. This can be done, e.g., through a cluster autoscaler on AWS. The scaling may take some time, so the cronjob should be able to handle this kind of delay.

By the by: I would also suggest to secure/hide the endpoint with the "expensive operation" from public, such that no user is able to tear down the production deployment, either "by accident" or out of malicious intentions.

Turing85
  • 18,217
  • 7
  • 33
  • 58
  • I found the 'scale up/down deployment' command but is there a way to run kubectl commands in cronjob? kubectl scale --replicas= deployment – dvdmn Apr 27 '21 at 14:51