I have three namespaces: namespace monitoring with prometheus-operator, namespace rabbitmq with a RabbitMq queue manager and prometheus-adapter, namespace worker with an app that just create inputs for the RabbitMq pod. I want to use an Horizontal Pod Autoscaler (HPA) to scale the worker pod (on worker namespace) with metrics from queue "task_queue" from RabbitMq pod (on rabbitmq namespace). All those metrics are collect by prometheus operator (on monitoring namespace) and they are shown in prometheus front-end:
Query "rabbitmq_queue_messages" at prometheus-url:8080/graph:
rabbitmq_queue_messages{durable="true",endpoint="metrics",instance="x.x.x.x:9419",job="rabbitmq-server",namespace="rabbitmq",pod="rabbitmq-server-0",queue="task_queue",service="rabbitmq-server",vhost="/"}
RabbitMQ, Prometheus-operator and Prometheus-adapter were installed from helm charts
RabbitMQ (values.yaml have password and enable metrics at 9419 for scraping):
helm install --namespace rabbitmq rabbitmq-server stable/rabbitmq \
--set extraPlugins=rabbitmq_prometheus \
-f charts/default/rabbitmq/values.yaml
Prometheus-adaptor:
helm upgrade --install --namespace rabbitmq prometheus-adapter stable/prometheus-adapter \
--set prometheus.url="http://pmt-server-prometheus-oper-prometheus.monitoring.svc" \
--set prometheus.port="9090"
Prometheus-operator:
helm upgrade --install --namespace monitoring pmt-server stable/prometheus-operator \
--set prometheusOperator.createCustomResource=false \
-f charts/default/values.yaml
Prometheus values.yaml:
prometheus:
additionalServiceMonitors:
- name: rabbitmq-svc-monitor
selector:
matchLabels:
app: rabbitmq
namespaceSelector:
matchNames:
- rabbitmq
endpoints:
- port: metrics
interval: 10s
path: /metrics
The custom metrics are ok:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/rabbitmq/services/rabbitmq-server/rabbitmq_queue_messages" | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/rabbitmq/services/rabbitmq-server/rabbitmq_queue_messages"
},
"items": [
{
"describedObject": {
"kind": "Service",
"namespace": "rabbitmq",
"name": "rabbitmq-server",
"apiVersion": "/v1"
},
"metricName": "rabbitmq_queue_messages",
"timestamp": "2020-08-20T12:15:39Z",
"value": "0",
"selector": null
}
]
}
And here is my hpa.yaml:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: rabbitmq-queue-worker-hpa
namespace: worker
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: worker
minReplicas: 1
maxReplicas: 50
metrics:
- type: Object
object:
metric:
name: rabbitmq_queue_messages
describedObject:
apiVersion: "/v1"
kind: Service
name: rabbitmq-server.rabbitmq.svc.cluster.local
target:
type: Value
value: 100
But the hpa don't work as kubectl describe shows:
kubectl describe hpa/rabbitmq-queue-worker-hpa -n worker
Name: rabbitmq-queue-worker-hpa
Namespace: worker
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: rabbitmq-scaling-demo-app
meta.helm.sh/release-namespace: worker
CreationTimestamp: Thu, 20 Aug 2020 08:42:32 -0300
Reference: Deployment/worker
Metrics: ( current / target )
"rabbitmq_queue_messages" on Service/rabbitmq-server.rabbitmq.svc.cluster.local (target value): <unknown> / 100
Min replicas: 1
Max replicas: 50
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetObjectMetric the HPA was unable to compute the replica count: unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: the server could not find the metric rabbitmq_queue_messages for services rabbitmq-server.rabbitmq.svc.cluster.local
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 60m (x12 over 63m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get object metric value: unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
Warning FailedGetObjectMetric 60m (x13 over 63m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
Warning FailedGetObjectMetric 58m (x3 over 58m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: the server is currently unable to handle the request (get services.custom.metrics.k8s.io rabbitmq-server.rabbitmq.svc.cluster.local)
Warning FailedGetObjectMetric 2m59s (x218 over 57m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: the server could not find the metric rabbitmq_queue_messages for services rabbitmq-server.rabbitmq.svc.cluster.local
I belive that HPA is trying to find the RabbitMq service on worker namespace,
Warning FailedGetObjectMetric 60m (x13 over 63m) horizontal-pod-autoscaler unable to get metric rabbitmq_queue_messages: Service on worker rabbitmq-server.rabbitmq.svc.cluster.local/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
but the service is on rabbitmq namespace. I tried with the rabbit's service FQDN (rabbitmq-server.rabbitmq.svc.cluster.local) and just the service's name (rabbitmq-server). What am I missing? Is there a way to make it work? The point here is that I have another project with 10+ namespaces and all of them uses the same rabbit server (on rabbitmq namespace), so let them all together in the same namespace will be a nightmare. Thanks.
Edit 1: My custom metrics config.yaml
prometheus:
url: http://pmt-server-prometheus-oper-prometheus.monitoring.svc
port: 9090
rbac:
create: true
serviceAccount:
create: true
service:
port: 443
logLevel: 6
rules:
custom:
- seriesQuery: 'rabbitmq_queue_messages{namespace!="",service!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
service: {resource: "service"}
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,queue="task_queue"}) by (<<.GroupBy>>)
And the adapter helm install with this file:
helm upgrade --install --namespace rabbitmq prometheus-adapter stable/prometheus-adapter -f config.yaml
And this is the HPA describe if the HPA is created in rabbitmq namespace:
Name: rabbitmq-queue-worker-hpa
Namespace: rabbitmq
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: rabbitmq-scaling-demo-app
meta.helm.sh/release-namespace: worker
CreationTimestamp: Fri, 21 Aug 2020 08:45:25 -0300
Reference: Deployment/worker
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 80%
Min replicas: 1
Max replicas: 50
Deployment pods: 0 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False FailedGetScale the HPA controller was unable to get the target's current scale: deployments/scale.apps "worker" not found
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 9s (x17 over 4m11s) horizontal-pod-autoscaler deployments/scale.apps "worker" not found
