1

The objective of the below function is:

  1. Scale down an app instance to 0 on K8s cluster.
  2. Wait for the scale down to complete.
  3. Verify if the current scale is reached 0 and no active instances are present for that app.
  4. Scale up the app to its original scale.
  5. Wait for the scale-up to complete.
  6. Verify if the current scale is reached its original scale and active instances are present for that app.

_get_desired_replicas() -> returns the count of currently running app instances
_instances() -> returns the list of active instances with their ids.

The issue

_instances() works on the caching mechanism and doesn't immediately reflect the current system state. There is a noticable lag of 3-4 seconds after which correct state of the system is fetched. This results into while loop being stuck (refer example below) until timeout gets expired because of _instances() not returing the current system state immediately.
One option is to go with or instead of and but don't want to go in that direction.

def _restart_k8sapp_wait(app, timeout=SCALE_UPDOWN_WAIT):
    _sleep_interval = 1
    desired_scale = _get_desired_replicas(app)
    if desired_scale == 0:
        return False
    _scale_app(app, 0)
    barrier_completed = time.time() + timeout
    while time.time() < barrier_completed:
        instances = _instances(app)
        current_scale = _get_desired_replicas(app)
        if current_scale == 0 and not instances:
            _scale_app(app, desired_scale)
            refreshed_instances = _instances(app)
            refreshed_scale = _get_desired_replicas(app)
            if refreshed_scale == desired_scale and refreshed_instances:
                return True
        time.sleep(_sleep_interval)
    return False


Example: while loop iterations after app scale down
- iter 1:
   desired_scale = 1
   instances=[item1], current_scale = 0
   if current_scale == 0 and not instances: evaluated to False
   iter 1 finished ..

- iter 2:
   desired_scale = 1
   instances=[], current_scale = 0
   if current_scale == 0 and not instances: evaluated to True
      instances = [], current_scale = 1
      if current_scale == desired_scale and not instances: evaluated to False
   iter 2 finished ..    

- iter 3:
   desired_scale = 1
   instances=[item1], current_scale = 1
   if current_scale == 0 and not instances: evaluated to False
   iter 3 finished ..
Abhinav Mathur
  • 7,791
  • 3
  • 10
  • 24
saas1990
  • 35
  • 8

0 Answers0