1

We want to add Azure cloud compute audit logs to Data Lake. For Virtual Machines, these would be Event Logs, and includes one time historical and ongoing. We want to do this for auditing purposes. Is there an option we can enable for this? We are trying to avoid having to feed this data manually extracting and then loading the data ourselves. So if there is an option in place that would be ideal!!

I looked into the possibility of ETW Events but isnt that just useful at the application level and not the VM level?

Ive also looked at this thread and there was a comment referring to an alternative solution called "Log Analytics" from Azure but clicking the link leads to an unknown page

Cataster
  • 3,081
  • 5
  • 32
  • 79

1 Answers1

2

You can use Azure diagnostic extension, an agent in Azure Monitor that collects monitoring data from the guest operating system

Go to Azure Portal, select your VM, under Monitoring section select Diagnostic Settings.

enter image description here

Configure the Event logs you need and sink (adls)

enter image description here

Select Logs

enter image description here

Select your app insights account where you want to route logs

enter image description here

enter image description here

enter image description here

Find logs in your data storage as tables

enter image description here

KarthikBhyresh-MT
  • 4,560
  • 2
  • 5
  • 12
  • That's awesome! So I dont need to worry about writing Kusto queries in Log Analytics to pull this information right? Also, howd you get the logs in the container (assuming datalake?) In the last step? Is enabling the diagnostics feature automatically populating the container?? Because that would be even more ideal! – Cataster Oct 29 '21 at 06:22
  • Correct, [Azure diagnostics extension](https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview#:~:text=The%20Azure%20Diagnostics,Azure%20Storage%20Explorer.) automatically stores logs as log tables or blobs in the storage account. This here lists the different types of data collected from the diagnostics extension and whether they're stored as a table or a blob. https://learn.microsoft.com/en-us/azure/azure-monitor/agents/diagnostics-extension-windows-install#data-storage – KarthikBhyresh-MT Oct 29 '21 at 07:39
  • yes, it is populating the ADLS account which I choose as storage account while configuring the agent as you can see in first and fifth snips – KarthikBhyresh-MT Oct 29 '21 at 07:42
  • After following all these steps, i still dont see logs in the storage account. Now in the Sinks section, i did include the required parameters which is the Source subscription and Instrumentation Key, but not "Application Insights Account" which doesnt appear to have been a required field. In the Overview page, i see this `Agent Diagnostic data is being sent to this storage account: abcdbtestguestdiag` so i suppose diagnostic logging should be enabled, yet when i proceed to the storage account container, i see this: `You don't have any containers yet. Click '+ Container' to get started.` – Cataster Nov 17 '21 at 03:51
  • shouldnt the container be automatically created? or did we have to create a container first? – Cataster Nov 17 '21 at 03:53
  • It would be created automatically. You would have to wait couple of mins for it to load, depending on the type of log level you have chosen, there should some relevant activity for it to log – KarthikBhyresh-MT Nov 17 '21 at 04:20
  • interesting, its been a few days actually since we have configured it and it hasnt been created yet.... – Cataster Nov 17 '21 at 04:21
  • From overview page: `Sinks Diagnostic data is being sent to an Application Insights account with instrumentation key: 66d3137b-XXXXXXXXX....` I suppose that should have triggered it to start collecting/storing logs...we have selected the same logs per your snippets yet nothing stored 4 days later since friday – Cataster Nov 17 '21 at 04:28
  • ah you should be looking at the tables in data storage. You should find a table with name `WADWindowsEventLogsTable`. Please see my second comment above. https://learn.microsoft.com/en-us/azure/azure-monitor/agents/diagnostics-extension-windows-install#data-storage – KarthikBhyresh-MT Nov 17 '21 at 05:14
  • Since you have chosen event logs, it will be in tables under your storage account. I overlooked the mention of container while you have set event logs – KarthikBhyresh-MT Nov 17 '21 at 05:34
  • ohhh i see, so basically only iis logs get logged in a container as blobs, while event logs get logged in the table. This is the natural ADLS configuration correct? Also, whats the point of "Application Insights Account" field when configuring the Sink if its not a required field? Its currently blank for us because i guess we didnt have to choose it so whats the point of the field? – Cataster Nov 17 '21 at 06:00
  • 1
    Correct. Applications insights if you'd want to do live monitoring on applications and got some analytical tools. Sinks is just additional to storage if you would want to send the logs to. If you have disabled app insights option, you can leave it blank. – KarthikBhyresh-MT Nov 17 '21 at 07:57
  • Thank you, really appreciate your help. btw i was checking out how to view those tables, and it looks like storage explorer (preview) is something that would allow visibility of the diagnostic data directly on the portal instead of having to install software to view tables/blobs. is this correct? https://learn.microsoft.com/en-us/azure/azure-monitor/agents/diagnostics-extension-windows-install#tools-to-view-diagnostic-data – Cataster Nov 17 '21 at 15:12
  • That is correct – KarthikBhyresh-MT Nov 17 '21 at 15:28