The objective of the below function is:
- Scale down an app instance to 0 on K8s cluster.
- Wait for the scale down to complete.
- Verify if the current scale is reached 0 and no active instances are present for that app.
- Scale up the app to its original scale.
- Wait for the scale-up to complete.
- Verify if the current scale is reached its original scale and active instances are present for that app.
_get_desired_replicas()
-> returns the count of currently running app instances
_instances()
-> returns the list of active instances with their ids.
The issue
_instances()
works on the caching mechanism and doesn't immediately reflect the current system state. There is a noticable lag of 3-4 seconds after which correct state of the system is fetched. This results into while loop being stuck (refer example below) until timeout gets expired because of _instances() not returing the current system state immediately.
One option is to go with or
instead of and
but don't want to go in that direction.
def _restart_k8sapp_wait(app, timeout=SCALE_UPDOWN_WAIT):
_sleep_interval = 1
desired_scale = _get_desired_replicas(app)
if desired_scale == 0:
return False
_scale_app(app, 0)
barrier_completed = time.time() + timeout
while time.time() < barrier_completed:
instances = _instances(app)
current_scale = _get_desired_replicas(app)
if current_scale == 0 and not instances:
_scale_app(app, desired_scale)
refreshed_instances = _instances(app)
refreshed_scale = _get_desired_replicas(app)
if refreshed_scale == desired_scale and refreshed_instances:
return True
time.sleep(_sleep_interval)
return False
Example: while loop iterations after app scale down
- iter 1:
desired_scale = 1
instances=[item1], current_scale = 0
if current_scale == 0 and not instances: evaluated to False
iter 1 finished ..
- iter 2:
desired_scale = 1
instances=[], current_scale = 0
if current_scale == 0 and not instances: evaluated to True
instances = [], current_scale = 1
if current_scale == desired_scale and not instances: evaluated to False
iter 2 finished ..
- iter 3:
desired_scale = 1
instances=[item1], current_scale = 1
if current_scale == 0 and not instances: evaluated to False
iter 3 finished ..