0

Scenario: I have few apps running in PCF and one of them is a SpringBoot listener app.

Problem: I need to scale up or down the instances of listener app which is listening to some External MQ queue based on the queue depth or number of messages.

This is really easy if I use PCF RabbitMQ, I can have Autoscaler rule setup for RabbitMQ queue and this will work fine. Which I am already doing for other apps.

But I need to use some other External MQ (not RabbitMQ) where external systems are publishing message. So I am curious if PCF platform exposes some kind of endpoints for each app ruining in PCF that can be used to scale up/down individual apps.

Avhi
  • 806
  • 2
  • 15
  • 29

1 Answers1

2

The best you could do is to use a custom metric. To do that, your app would need to monitor the queue length and emit a metric that indicates the queue length. You can use the instructions here to emit a custom metric.

https://docs.pivotal.io/pivotalcf/metric-registrar/using.html

Then once the metric is emitted, you can set up an autoscaler rule using a custom metric. Select the metric name that you emitted. Autoscaler will then use the metric you're emitting to scale your app. See this link for more details on using a custom metric.

https://docs.pivotal.io/pivotalcf/appsman-services/autoscaler/using-autoscaler.html#metric

Hope that helps!

Daniel Mikusa
  • 13,716
  • 1
  • 22
  • 28
  • Another solution could potentially to create an app and use the PCF controller to scale applications up and down based of incoming requests/metrics. I believe the autoscaler is being relaced soon(TM) – Darren Forsythe Jul 11 '19 at 23:43
  • @DarrenForsythe That is Pivotal's autoscaler in a nutshell. – Daniel Mikusa Jul 14 '19 at 01:54
  • As mentioned in the [Example](https://github.com/pivotal-cf/metric-registrar-examples/tree/master/java-spring-security), I tried to create an endpoint which emits in Integer value and based on this value I set up the rule in Autoscaler. This is the error: Autoscaler did not receive any metrics for `high_latency` during the scaling window. Scaling down will be deferred until these metrics are available. Followed all the steps mentioned to register the endpoint and I can see one CUPS is added to my app in PCF after registering the metric. Instance Name: `metrics-endpoint-high_latency` – Avhi Jul 15 '19 at 20:31
  • @DanielMikusa Here is the sample code: `@GetMapping("/high_latency") public ResponseEntity highLatency() throws InterruptedException { int queueLength=0; Random random = new Random(); int number = random.nextInt(50); System.out.println("Generate number is : "+number); if(number % 2 == 0) { queueLength=99; } else { queueLength=200; } return new ResponseEntity<>(queueLength, null, HttpStatus.OK); }` Autoscaler Rule: Metric Low High high_latency 100 199 – Avhi Jul 15 '19 at 20:38
  • Install one of these cf cli plugins: `https://github.com/cloudfoundry-community/firehose-plugin` or `https://github.com/cloudfoundry/log-cache-cli` and use that to confirm that your metric is being emitted correctly. If you can't see the metric regularly using one or both of these plugins, then Autoscaler won't be able to see it either. – Daniel Mikusa Jul 16 '19 at 13:23
  • I can see it emitting the metric if I hit the tail command `cf tail --envelope-class=metrics API Retrieving logs for app API in org DEV_System / space dev as *******... 2019-07-16T09:30:05.01-0400 [API/0] GAUGE absolute_entitlement:60474208174050.000000 nanoseconds absolute_usage:455129796685.000000 nanoseconds container_age:60474208174050.000000 nanoseconds 2019-07-16T09:30:19.90-0400 [API/0] GAUGE cpu:0.745884 percentage disk:199491584.000000 bytes disk_quota:1073741824.000000 bytes memory:376762368.000000 bytes memory_quota:1073741824.000000` – Avhi Jul 16 '19 at 13:32
  • Are you sure? I don't see anything called "high_latency". Which of those metrics are you trying to use in your autoscaler rule? – Daniel Mikusa Jul 16 '19 at 13:42
  • Steps I followed. 1. Added code as suggested in [Example](https://github.com/pivotal-cf/metric-registrar-examples/tree/master/java-spring-security). 2. installed `metric-registrar` on my local 3. register endpoint -> `cf register-metrics-endpoint API high_latency` 4. installed `log-cache` on my local 5. build app and deployed to PCF and then Bind Autoscaler & set up custom metric rule. Am I missing something? – Avhi Jul 16 '19 at 15:32
  • I don't think this is an issue with autoscaler and the rule, it's with metrics being emitted. Can you make a new question about how to emit custom metrics? I think we're getting off topic. – Daniel Mikusa Jul 17 '19 at 03:14
  • @DanielMikusa I created new question and added all the information there. https://stackoverflow.com/questions/57079356/how-to-emit-custom-metrics-from-springboot-application-and-use-it-in-pcf-autosca – Avhi Jul 17 '19 at 15:26