Get-AzureBlobContent throwing error when run from Azure Automation account

Question

I am receiving a failure while trying to download blob (JSON file) from Azure storage account from my Azure Automation account. It looks like an authorization issue.

This works on my local laptop, but does not work on Azure Automation Account. Does not work even if I make the container "public"

I have assigned OWNER privileges for the Automation accounts's service principle on the Resource Group (Automation account + Storage account stay in this RG) and specifically on the Storage Account as well:

Below is the code:

$connectionName = "AzureRunAsConnection"
try
{
    # Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName         

    "Logging in to Azure..."
    Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 
}
catch {
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    } else{
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

$config_file_resource_group_name = "vg-datalake-manjunath"
$config_file_storage_account_name = "datalakelog"
$primary_key = (Get-AzureRmStorageAccountKey -ResourceGroupName $config_file_resource_group_name -AccountName $config_file_storage_account_name).value[0]
 $config_file_context = New-AzureStorageContext -StorageAccountName $config_file_storage_account_name -StorageAccountKey $primary_key

 Get-AzureStorageBlobContent -Blob "mw_services.json" -Container "fwconfigfiles" -Destination "C:\temp\mw_services.json" -Context $config_file_context

 get-content "C:\temp\mw_services.json" | write-output

ERROR:

Get-AzureStorageBlobContent : The remote server returned an error: (403) Forbidden. HTTP Status Code: 403 - HTTP Error 
Message: This request is not authorized to perform this operation.
At line:30 char:2
+  Get-AzureStorageBlobContent -Blob "mw_services.json" -Container "fwc ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Get-AzureStorageBlobContent], StorageException
    + FullyQualifiedErrorId : 
StorageException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureStorageBlobContentCommand

Possible duplicate of [Azure automation, PowerShell to fetch a file in private blob container](https://stackoverflow.com/questions/34903180/azure-automation-powershell-to-fetch-a-file-in-private-blob-container) — hujtomi, May 14 '19 at 12:10
This works on my local laptop, but does not work on Azure Automation Account. Does not work even if I make the container "public" — Manjunath Rao, May 14 '19 at 12:32
Please check if the storage account is part of a virtual network or firewalled. That could be the reason for this error. — Gaurav Mantri, May 14 '19 at 12:44
@GauravMantri yes. That was the issue. If I select "Allow All Networks", it works. But we will not enable that in our environment. We allow access to the storage account via ProxyRules. However, if I enable "Allow Microsoft services to access this storage account", the code still fails. Any idea? — Manjunath Rao, May 14 '19 at 13:28
Looking here: https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security#exceptions, I believe Azure Automation is not part of trusted Azure Services. Not sure why this is not part of trusted services though :). — Gaurav Mantri, May 14 '19 at 13:32
I’m not sure. I’ve asked the question on Twitter though. Let’s hope someone from Azure team responds. Will provide an update once I hear back. — Gaurav Mantri, May 14 '19 at 14:48
Please see this post on MSDN Forums: https://social.technet.microsoft.com/Forums/en-US/26bd07d4-05bc-446f-a4d5-c185f517d8bb/storage-account-firewall-and-azure-automation?forum=windowsazuredata. Looks like you will need to look up the IP address for Azure Automation service and add an exception. — Gaurav Mantri, May 14 '19 at 16:48
Hope Microsoft address your question. However, regarding the MSDN Forum, it is very difficult to update the Azure IP address and maintain it, because they keep updating the XML file. Also, we have a logic (security implementation) that only "Client proxy" address has to be allowed as a firewall for the PaaS services. (like storage account, ADLA, ADLS etc.,) — Manjunath Rao, May 16 '19 at 07:56
One approach would be to use a Hybrid Runbook worker and integrate the Storage Account into the same Vnet as the Hybrid Runbook worker. This way you don't need access on the internet facing side of the Storage Account. Another approach would be the use of a SAS token. — Felix Bodmer, May 17 '19 at 15:07
We are looking on the Hybrid Worker solution. Meanwhile, regarding "whitelisting Azure Datacenter IPs", the WestEurope region has 123 ranges, however, one Storage Account can have maximum of 100 Network Rules. So this solution is not working. Any workaround ? Each storage account supports up to 100 IP network rules, which may be combined with Virtual network rules. https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security#grant-access-from-an-internet-ip-range — Manjunath Rao, May 20 '19 at 08:37

Md Farid Uddin Kiron · Answer 1 · 2019-05-22T09:15:55.887

3

The possible reason is that you may configure to selected networks to access.

If you enable this option, and whether you tick "allow trusted microsoft services to access", you would get this error, since automation is not listed under MS trusted services. see https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security#trusted-microsoft-services.

edited May 22 '19 at 09:15

answered May 20 '19 at 09:43

Md Farid Uddin Kiron

16,817
3
17
43

The cmdlet "Get-AzureStorageBlobContent" does not work even if we click on "Allow trusted Microsoft services", because "Azure Automation" service is not under "trusted service". – Manjunath Rao May 20 '19 at 09:54
I have tested the cmdlet "Get-AzureStorageBlobContent, and it works well. I use the code below. Set-AzureRmCurrentStorageAccount –ResourceGroupname "" –StorageAccountName "" $Container = Get-AzureStorageContainer -Container "" Get-AzureStorageBlobContent -Container $Container.Name -Blob "1.txt" -Destination "C:\1.txt" get-content "C:\1.txt" | write-output – Md Farid Uddin Kiron May 22 '19 at 02:56
1. From the Storage Account's firewall - Select "Selected Networks" 2. Check "Allow trusted Microsoft services to access this storage account" 3. Run your code from an Azure automation runbook. The code still fails with the error ==> Get-AzureStorageContainer : The remote server returned an error: (403) Forbidden. HTTP Status Code: 403 - HTTP Error Message: This request is not authorized to perform this operation. – Manjunath Rao May 22 '19 at 08:42
Sorry, my mistake. The azure automation is not in MS trusted service based on the document here https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security#trusted-microsoft-services. So, to access storage from azure automation, you need to set "allow access from all networks". Let me correct my answer. Thanks. – Md Farid Uddin Kiron May 22 '19 at 09:13

Get-AzureBlobContent throwing error when run from Azure Automation account

1 Answers1

Linked