I have followed and taken code from @Juval's answer , I have tried to reproduce in my environment and got expected results as below:
az login
$x=az graph query -q "Resources| where type == 'microsoft.insights/actiongroups'| project name"
$m=$x | ConvertFrom-json
$v=$m.data.name
$y=az graph query -q "Resources| project alertName = name, location, type,props = properties| where type contains 'microsoft.insights/activitylogalerts' | mvexpand actionGroups = parse_json(props['actions']['actionGroups'])| extend actionGroup = extract(@'([^\/]+$)',1,tostring(actionGroups.actionGroupId))| union (resources| project alertName = name,location,type,props = properties| where type contains 'microsoft.insights/metricalerts'| mvexpand actionGroups = parse_json(props['actions'])| extend actionGroup = extract(@'([^\/]+$)',1,tostring(actionGroups.actionGroupId)))|project actionGroup" | ConvertFrom-Json
$t=$y.data.actionGroup
$diff = Compare-Object $v $t
$result = $diff | Where-Object { $_.SideIndicator -eq "<=" } | Select-Object -ExpandProperty InputObject
$result
Output:

Here $v
contains all the action groups names.
$t
contains the actiongroup names associated with alerts.
$result
contains the names of action groups which are not associated with alerts.