1

I have a torchserve service running on kubernetes and I am already able to track metrics with it on port 8082. My problem is that from the kubernetes pod I can see it logs hardware metrics like:

[INFO ] pool-3-thread-2 TS_METRICS - CPUUtilization.Percent
[INFO ] pool-3-thread-2 TS_METRICS - DiskAvailable.Gigabytes
[INFO ] pool-3-thread-2 TS_METRICS - GPUMemoryUtilization.Percent

Although, if i check the metrics that I am currently scraping I am only able to see:

# TYPE ts_inference_requests_total counter
ts_inference_requests_total 144.0
ts_inference_requests_total 20.0
# HELP ts_inference_latency_microseconds Cumulative inference duration in microseconds
# TYPE ts_inference_latency_microseconds counter
ts_inference_latency_microseconds 6.051944813839998E8
ts_inference_latency_microseconds 4.7464253726E7
# HELP ts_queue_latency_microseconds Cumulative queue duration in microseconds
# TYPE ts_queue_latency_microseconds counter
ts_queue_latency_microseconds 2633867.5069999998
ts_queue_latency_microseconds 1080.43

Is it possible to also scrape the metrics that are being logged on kubernetes? Thanks for any help!

Prosciutt0
  • 55
  • 8
  • 1
    what does your `metrics.yaml` file look like? – Rick Rackow May 02 '23 at 09:10
  • 1
    **cAdvisor** can expose container metrics in a format compatible with Prometheus, enabling users to scrape and store these metrics using Prometheus for further analysis and visualization. For advanced use the **node_exporter** can be passed an optional list of collectors to filter metrics. The **collect[]** parameter may be used multiple times. In Prometheus configuration you can use this syntax under the [scrape config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). – Veera Nagireddy May 02 '23 at 12:17
  • Hi @Prosciutt0, Feel free to update the status of the question. Let me know the comment above helps to resolve your issue? I am happy to help you if you have any further queries. – Veera Nagireddy May 05 '23 at 05:30

0 Answers0