3

There is an AKS cluster in Azure with a microservices app (NodeJS) deployed.
The app is streaming logs to Application Insights (and to Log Analytics) using application insights.
All settings are by default: maxBatchSize & maxBatchIntervalMs.

When I check the pod logs using kubectl logs {POD_NAME} -n {NAMESPACE_NAME} I see the following output: 

---

2020-05-05T00:22:24.851Z info xxxxxxxxx-a94b-4666-9e83-31e945e1ee15 0oaXXXXXXXXXX PUT /storages/1XXXXXX007475 xxxMiddleware

xxxxMiddleware ended
Postgres pool raised an error. Error: read ETIMEDOUT
---

In Log Analytics using Kusto query I see the same: 

LogEntrySource  stderr
LogEntry    Postgres pool raised an error. Error: read ETIMEDOUT
TimeGenerated   2020-05-05T00:58:09
Computer    aks-agentpool-xxxxx-3
ContainerID 4a5e5e69957917578d8b18b59628ec3f21cd1dd19b5d66ca1xxxxxxxxxx

I'm wondering why the difference in timestamps is ~ 35+ minutes, i.e. 2020-05-05T00:58:09 - 2020-05-05T00:22:24.851Z

Yoni L.
  • 22,627
  • 2
  • 29
  • 48
Sergey
  • 381
  • 6
  • 24

1 Answers1

0

Yes, the difference is about 30 min. I guess this could be the difference in the timezone. kubectl reports logs in the timezone of the pod where as LogAnalytics might be reports it into UTC timezone or in your browser timezone. Please check the timezones of pod and the browser and see if this makes sense.

Hopefully this should solve the problem.

Atul
  • 1,116
  • 1
  • 10
  • 20
  • Could you please point me to the official explanation of these delays? I need some kind of proof. For my config, the pod logs and Log Analytics logs seem to be in UTC whereas I'm located in UTC+3 zone. However, as I showed above, the difference is not 3 hr but ~30 minutes. – Sergey May 07 '20 at 04:41
  • So all the logs are reporting in UTC time but there is a difference of ~30 mins between the timestamps? Can you check the latest logs of the pod and latest logs of lognalytics and see if loganalytics is missing ~30 mins of logs? I can't explain the delays at all but just trying to see what could be wrong here. Initially I was expecting that there is a difference in timezone of both sources and hence the difference. – Atul May 07 '20 at 10:58
  • Logs are streamed to Log Analytics workspace as intended (every 5 minutes). There is a strong difference in timestamps between pod logs and Azure Monitor logs. The reason is a main question of this topic. – Sergey May 15 '20 at 14:35