4

I am trying to setup a monitoring environment in aws eks with fargate compute nodes alone. I have seen a blog post here where we need a additional node group to host prometheus and grafana. But I need to get this done with fargate alone.

I tried following blog post to setup but after creating grafana pod, I am not able to get the metrics.

Is there any blog post which can direct me to setup monitoring using Amazon managed prometheus ?

Vamsi
  • 388
  • 2
  • 12

1 Answers1

1

You can follow the recipe which, in a nutshell, uses ADOT (our OpenTelemetry distro) to scrape in-cluster and remote-write into AMP.

Michael Hausenblas
  • 13,162
  • 4
  • 52
  • 66
  • Thanks a lot @Michael for the blog. I tried to reproduce the same setup. I have done it twice but the metrics are not recorded in the AMP, Please find the setup which I have done, the policy is – Vamsi Oct 04 '21 at 06:32
  • ```{"Version":"2012-10-17", "Statement":[{ "Effect":"Allow", "Action":[ "aps:RemoteWrite","aps:QueryMetrics","aps:GetSeries","aps:GetLabels", "aps:GetMetricMetadata", "logs:PutLogEvents","logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:DescribeLogGroups", "xray:PutTraceSegments", "xray:PutTelemetryRecord", "xray:GetSamplingRules", "xray:GetSamplingTargets", "xray:GetSamplingStatisticSummaries","ssm:GetParameters"], "Resource":"*" }]}``` – Vamsi Oct 04 '21 at 06:32
  • Trust Policy is ```{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/"},"Action": "sts:AssumeRoleWithWebIdentity","Condition": {"StringEquals": {"oidc.eks.eu-west-1.amazonaws.com/id/:sub": "system:serviceaccount:adot-col:iamproxy-service-account"}}}]}```. Followed by, I have already created a cluster and mapped the same OIDC accounts in the trust policy. I even have the amp workspace . – Vamsi Oct 04 '21 at 06:38
  • I also modified the prometheus-fargate.yaml in the blog with the corresponding values and changed the service account name to the one in the trust policy. I already have some deployments, so I just changed the existing deployment to have more replicas and still I cannot see any metrics populated in the command `awscurl --service="aps" --region="$REGION" "https://$AMP_ENDPOINT/api/v1/query?query=adot_test_gauge0"` The response is `{"status":"success","data":{"resultType":"vector","result":[]}}` Sorry for multiple comments but there is a character limit for comments, so could not fit in one. – Vamsi Oct 04 '21 at 06:39
  • Thanks for the update. Your `awscurl` output tells me that nothing gets ingested, so check the write path (in IAM console go to the policy and click on the "Policy usage" tab, if the policy is never used then it's an IRSA issue). Interactive debugging on SO is not the best place, BTW ;) – Michael Hausenblas Oct 04 '21 at 08:03
  • 1
    I do not think there is an IRSA issue as I can see role mapped in the policy usage . I am not sure what an SO is. – Vamsi Oct 04 '21 at 09:24
  • SO is the site we're on (StackOverflow). Send me a Twitter DM (find via profile) and we can try to debug interactively via Slack … – Michael Hausenblas Oct 04 '21 at 09:37
  • Did you have this working? – farp332 Sep 15 '22 at 15:31