is it possible to get the ingestion volume of a log analytics workspace using powershell or any programming language?

Question

I am getting a summary of our log analytics workspaces in the company, this includes the tables that are being used in each workspace, as well as other data such as the ingestion volume.

The closest thing to "get" this is to use this command in PowerShell

Get-AzOperationalInsightsWorkspaceUsage -ResourceGroupName "RG_name" -Name "WS_name"

And it shows me this info:

Id            : DataAnalyzed
CurrentValue  : 0
Unit          : Bytes
Limit         : -1
NextResetTime : 7/24/2023 8:00:00 AM
QuotaPeriod   : 1.00:00:00

Which is not enough, I am looking for this:

I searched for anything similar but didn't find anything else. Hope there is a solution that I am missing.

are you planning on using your own account or a service principal to make this query? — Santiago Squarzon, Jul 24 '23 at 20:09
well my answer already shows how you can pull this with your user account, if using a service principal you just need to acquire a token — Santiago Squarzon, Jul 26 '23 at 18:30

score 1 · Answer 1 · answered Jul 24 '23 at 19:05

You can get this using the REST API. The call you want to make is to Workspace Usages, which will show you your usage in bytes.

https://learn.microsoft.com/en-us/rest/api/loganalytics/workspace-usages/list?tabs=HTTP#workspacelistusagesresult

You can call REST API's directly from powershell using Invoke-RestMethod. It is a twostep process. First, you need to make a REST call to authenticate, then you can make your subsequent REST calls using the token you received during the auth call. Steps are fully documented here (the example here shows how you can extract the token from Powershell Context):

https://learn.microsoft.com/en-us/azure/governance/resource-graph/first-query-rest-api

thanks for your help, unfortunately I am getting the same as the command "value": [ { "name": { "value": "DataAnalyzed", "localizedValue": "Data Analyzed" }, "unit": "Bytes", "currentValue": 0.0, "limit": -1.0, "nextResetTime": "2023-07-25T08:00:00Z", "quotaPeriod": "P1D" } ] } @ken-w-zero-networks — samauces, Jul 24 '23 at 22:22

Santiago Squarzon · Accepted Answer · 2023-07-26T20:20:38.023

Assuming you will be using your user account to query the Log Analytics Rest API and you have access to the Az Module plus Reader roles over the target Log Analytics Workspace, this is how you can get the ingestion volume by querying the Usage table.

# connect impersonating user
Connect-AzAccount
# the GUID of the LAW goes here
$workspaceId = 'xxxxx-xxxxx-xxxxx...'
$resource = 'https://api.loganalytics.io'
# get a token with permissions to query the LAW API
$token = Get-AzAccessToken -ResourceUrl $resource

$invokeRestMethodSplat = @{
    Headers     = @{
        Authorization = '{0} {1}' -f $token.Type, $token.Token
    }
    Uri         = '{0}/v1/workspaces/{1}/query' -f $resource, $workspaceId
    ContentType = 'application/json'
    Method      = 'Post'
    Body        = @{
        query = '
        Usage
        | where TimeGenerated > ago(24h)
        | summarize ["TotalIngestionVolume(GB)"] = sum(Quantity) / 1024.0 by DataType
        | order by ["TotalIngestionVolume(GB)"]
        '
    } | ConvertTo-Json
}
$response = Invoke-RestMethod @invokeRestMethodSplat

Up to this point in $response you would have the ingestion volume per table in your Log Analytics Workspace, problem is the response from this API is really bad so you have to enumerate the columns and rows to get objects out of it like so:

$columns = @($response.tables.columns.name)
$result = [ordered]@{}

foreach ($row in $response.tables.rows) {
    for ($i = 0; $i -lt $columns.Count; $i++) {
        $result[$columns[$i]] = $row[$i]
    }
 
    [pscustomobject] $result
    $result.Clear()
}

If using a Service Principal instead of impersonating our user account, the logic is pretty much the same, the only change is the way we acquire the token:

$clientId = 'xxxxx-xxxx-xxxx....'
$tenantId = 'xxxxx-xxxx-xxxx....'
$secret = 'fo0B4rB4z'

$cred = [pscredential]::new(
    $clientId,
    (ConvertTo-SecureString $secret -AsPlainText -Force))

Connect-AzAccount -ServicePrincipal -Tenant $tenantId -Credential $cred

$resource = 'https://api.loganalytics.io'
# get a token with permissions to query the LAW API
$token = Get-AzAccessToken -ResourceUrl $resource

# rest stays the same

you deserve heaven!, thank you very much I was only looking for the management of log analytics and not to query them indeed, now I looked for it on microsoft's docs and I understand more. thank you again this is exactly what I was looking for :) — samauces, Jul 26 '23 at 20:07
@samauces added the example if using a service principal but I guess you already understood that part — Santiago Squarzon, Jul 26 '23 at 20:21
do you know why when using this command 'Get-AzOperationalInsightsTable' I get around 500~ tables but when using your query 'usage .....' I only get a few I am guessing the rest of the tables are empty or are built in not included when using the query rather than the command — samauces, Jul 26 '23 at 20:49
@samauces perhaps its because of the limiting `where TimeGenerated > ago(24h)` if those tables havent been written in the past 24h they wouldnt be visible (but im just guessing here) — Santiago Squarzon, Jul 26 '23 at 20:55

is it possible to get the ingestion volume of a log analytics workspace using powershell or any programming language?

2 Answers2