I can connect my local Azure function to my CosmosDB database, but when I publish the code on Azure, it cannot connect anymore

Question

I have developed a blob-triggered function app that then saves data to a CosmosDB database using the Gremlin API and the Gremlin.Net driver.

I have created a CosmosDB database on my Azure account, and then read the connection parameters from local.settings.json when I run my function locally. This works perfectly fine, and my function manages to store data on the database hosted on Azure. Here's my code for connecting to my database :

        var hostName = Environment.GetEnvironmentVariable($"CosmosDBHostname");

        var portNumber = Int32.Parse(Environment.GetEnvironmentVariable($"CosmosDBPort"));

        var enableSsl = Boolean.Parse(Environment.GetEnvironmentVariable($"CosmosDBEnableSSL"));

        string containerLink = Environment.GetEnvironmentVariable($"CosmosDBContainerLink");

        string password = Environment.GetEnvironmentVariable($"CosmosDBPassword");

        var gremlinServer = new GremlinServer(hostName, portNumber, enableSsl,
            username: containerLink,
            password: password);

        var gremlinClient = new GremlinClient(gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(),
            GremlinClient.GraphSON2MimeType);

I then created a Function App on the same ResourceGroup as the database on my Azure account, and publish my code to this Function App, and copied the parameters of my local.settings.json file to the Azure settings of my function app. But when I do that, my function is not able to connect to the database anymore.

I get the following exception when my function runs and tries to connect to the database :

Error while processing files: Microsoft.WindowsAzure.Storage.StorageException: An attempt was made to access a socket in a way forbidden by its access permissions ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask1.get_Result() at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Threading.Tasks.ValueTask1.get_Result() at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask) at System.Threading.Tasks.ValueTask1.get_Result() at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) --- End of inner exception stack trace --- at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token) at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)

Is there any explanation as to why my function can connect to my database from my local development environment, using local.settings.json, but not when it is published, using the Azure settings ?

For debugging purpose you could consider logging your hostname/port so you now the settings are correct. — Preben Huybrechts, Jul 31 '20 at 10:31
I actually did that, sorry, I should have specified. But the settings are exactly the same ones as in my local.settings.json file. — MarleneHE, Jul 31 '20 at 10:33
do you have some firewall / IP restriction in place on your cosmosdb? — silent, Jul 31 '20 at 10:33
Sorry, I checked and no, there is no IP restriction. This is very confusing. — MarleneHE, Jul 31 '20 at 15:20

Matias Quaranta · Accepted Answer · 2020-07-31T19:03:22.540

0

The error is coming from Azure Storage (Microsoft.WindowsAzure.Storage.StorageException), not from your Gremlin client, based on the stack trace you provided:

Error while processing files: Microsoft.WindowsAzure.Storage.StorageException: An attempt was made to access a socket in a way forbidden by its access permissions

Functions use an Azure Storage account which is also part of the settings. Locally it usually uses the Storage Emulator.

Double check that configuration and check which Storage account is using, and if that Storage account (if it exists) has any Firewall / VPN restrictions.

EDIT: The stack trace also shows that the issue seems related to the access of the file:

Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync

This could also be related to the number of connections you are opening, if this is happen sporadically or after X amount of time running (if you restart the Function App it works temporarily).

I see that your code is creating a GremlinServer instance, if you are doing that on every Function execution, that might lead to connection issues.

See:

Limits on instances https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#service-limits
Connection recommendations https://learn.microsoft.com/en-us/azure/azure-functions/manage-connections#static-clients

edited Jul 31 '20 at 19:03

answered Jul 31 '20 at 15:25

Matias Quaranta

13,907
1
22
47

OMG I can't believe I missed that, thank you so much !! So my Storage account does exist, anytime I push a blob to it, it triggers my function, and there is no Firewall. If my logging is correct, it seems like the exception is thrown when my function tries to parse the file. – MarleneHE Jul 31 '20 at 15:46
Please see my edit, it might be related to connections – Matias Quaranta Jul 31 '20 at 19:03
Very insightful, thank you, I will look into this ! It's anyway pretty bad that an instance of GremlinServer is created everytime my function excecutes. I'm however confused at the fact that my Function App contains two functions, and the other one doesn't have any issue at connecting and reading files from my Storage account. – MarleneHE Aug 03 '20 at 14:32
1

I finally managed to narrow down the line of code that is causing the issue, and it's so obvious I can't believe it took me so long to figure this out. This is the line : `var list = await container.ListBlobsSegmentedAsync(blobContinuationToken);` When my container is on my storage emulator, it works like a charm, but not when the container is on my Azure account. – MarleneHE Aug 03 '20 at 16:08
1

So I finally found what the issue was. I was using `CloudStorageAccount` to retrieve my Azure Storage Account, then my container, and then list the blobs on that container using the line of code I provided above. The issue was I was still retrieveing my development storage account, instead of the Azure one, using `var storageAccount = CloudStorageAccount.DevelopmentStorageAccount;`, which of course my code couldn't (_and shouldn't try to_) access once it was published on my Azure account. – MarleneHE Aug 04 '20 at 13:37

I can connect my local Azure function to my CosmosDB database, but when I publish the code on Azure, it cannot connect anymore

1 Answers1