Connecting to azurite using a hostname fails

Question

UPDATE: I can verify this behavior is fixed in Azure.Storage.Blobs 12.5.1 https://www.nuget.org/packages/Azure.Storage.Blobs https://github.com/Azure/azure-sdk-for-net/issues/9404

How can I connect to azurite using a hostname?

I'm trying to emulate Azure Blob Storage in docker using Azurite for integration tests.

All works well, to the point I have to access Azurite via a hostname (which is AFAIK required for docker networking)

My connection string looks like this (which is the default well-known connection string):

"AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;DefaultEndpointsProtocol=http;BlobEndpoint=http://azurite:10000/devstoreaccount1;"

My docker compose part for azurite looks like this:

services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    hostname: azurite
    command: "azurite-blob --loose --blobHost 0.0.0.0"
    ports:
      - "10000:10000"
    volumes:
      - ./test/azurite:/data
    networks:
      - stillsnet

  images:
    container_name: images
    image: myapp/images
    build:
      context: .
      dockerfile: Dockerfile
    ports:
       - "5000:5000"
       - "5001:5001"
    environment:
      - ASPNETCORE_ENVIRONMENT=Test
      - ASPNETCORE_URLS=http://+:5000
      - imagesStorage__AzureBlobStorage__ConnectionString=AccountName=devstoreaccount1;DefaultEndpointsProtocol=http;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://azurite:10000;
    depends_on:
      - azurite
    links:
      - azurite
    networks:
      - stilssnet

my code looks like this:

private const string ConnectionString ="AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;DefaultEndpointsProtocol=http;BlobEndpoint=http://azurite:10000/devstoreaccount1;";

[Fact]
public async Task UploadFile()
{
   var container = new BlobContainerClient(ConnectionString, "images");
   await using var stream = File.OpenRead(@"C:\temp\output\3ee9bc41-40ea-4d05-b180-e74bd5065622\images\00000000.jpg");
   await container.UploadBlobAsync("test.jpg", stream);
}

this will throw an exception:

System.Xml.XmlException : Root element is missing.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
   at System.Xml.Linq.XDocument.Load(Stream stream, LoadOptions options)
   at Azure.Storage.Blobs.BlobRestClient.Container.CreateAsync_CreateResponse(Response response)
   at Azure.Storage.Blobs.BlobRestClient.Container.CreateAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, PublicAccessType access, Nullable`1 timeout, IDictionary`2 metadata, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, Boolean async, CancellationToken cancellationToken, String operationName)
   at Azure.Storage.Blobs.BlobContainerClient.CreateIfNotExistsInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateIfNotExistsAsync(PublicAccessType publicAccessType, IDictionary`2 metadata, CancellationToken cancellationToken)

If I change the connection string from azurite to 127.0.0.1 it all works fine.

@the_witch_king_of_angmar only with the hack I described below, I've filed a bug at the Azure Storage SDK — Wiebe Tijsma, Feb 10 '20 at 17:00
@the_witch_king_of_angmar this behavior is fixed in Azure.Storage.Blobs 12.5.1 — Wiebe Tijsma, Aug 21 '20 at 05:40

score 6 · Accepted Answer · answered May 12 '20 at 14:08

6

The host's name in that network is azurite, so you can use this connection string UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://azurite.

Works for me and I think is the way to do it.

Also you can use the legacy storage emulator (that includes Tables), instead of azurite you put docker.host.internal

answered May 12 '20 at 14:08

Isaac Ojeda

285
4
11

doesn't seem to work for me unfortunately, I get the same exception – Wiebe Tijsma May 25 '20 at 09:15
Just to add, I am running Azurite in a Docker container and I had to bind to a different port for blobs. You can add the port after the ProxyUri, and it works perfectly :) `UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://azurite:11000` – René Sackers Apr 28 '22 at 20:18

Wiebe Tijsma · Answer 2 · 2020-08-19T15:05:53.277

UPDATE: I can verify this behavior is fixed in Azure.Storage.Blobs 12.5.1 https://www.nuget.org/packages/Azure.Storage.Blobs https://github.com/Azure/azure-sdk-for-net/issues/9404

It looks like the BlobContainerClient itself strips away the account name from the URL if you use anything else than localhost or an IP address, so the client generates this:

PUT /images?restype=container HTTP/1.1" 400 - instead of PUT /devstoreaccount1/images?restype=container HTTP/1.1 201

As an ugly workaround we can include the account name in the container name when testing against azurite: var container = new BlobContainerClient(ConnectionString, "devstoreaccount1/images");

Though await container.CreateIfNotExistsAsync(); doesn't work properly against azurite then (throws a 409 Exception when it already exist...

So we either have:

an ugly hack
dropping azurite in favor of a real blob storage account
dropping the BlobContainerClient which seems to perform too much magic based on the host name.

It doesn't seem to be related to Azurite itself, or ideally it should support root URL's that are compatible with the Azure blob store without having to specify the account name prefix.

score 1 · Answer 3 · answered Aug 14 '20 at 07:12

Root of the problem is how internals of azure-storage-queue library split URL. My problem was fix by usage of newer version of connector. I know that it's c# thread but in java I had the same problem. Please use 12.6.0-beta.1 or higher if version number is the same in c#.

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-queue</artifactId>
    <version>12.6.0-beta.1</version>
</dependency>

version 12.0.1 doesn't seem to work properly.

Yes, looks like it was fixed: https://github.com/Azure/azure-sdk-for-net/issues/9404#issuecomment-572686630 — Wiebe Tijsma, Aug 19 '20 at 14:30

score 0 · Answer 4 · answered Jan 06 '20 at 21:41

0

I ran the azurite with this command (from the docker hub page https://hub.docker.com/_/microsoft-azure-storage-azurite?tab=description)

docker run -p 10000:10000 -p 10001:10001 mcr.microsoft.com/azure-storage/azurite

my config file is just this:

<add name="BlobStorage" connectionString="UseDevelopmentStorage=true" />

in the connectionStrings section (from the page: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite#install-and-run-the-azurite-docker-image)

answered Jan 06 '20 at 21:41

user9705827

1
1

1

Sure that works, however my problem occurs when you want to start using Azurite from another container using docker networking.You need to provide a host name, because it's a different container, so UseDevelopmentStorage=true doesn't work. – Wiebe Tijsma Jan 07 '20 at 08:51

score 0 · Answer 5 · answered Jan 08 '20 at 18:13

I run both Sql server and Azurite. I just use docker compose to start them up. Notice the restart: always. this will start the containers when docker starts . It's just like running the installed azure storage emulator.

version: "2"

services:

  sql-server-db:
    container_name: sql-server-db
    image: microsoft/mssql-server-linux:2017-latest
    restart: always
    ports:
      - "1433:1433"
      - "1434:1434"
    environment:
      SA_PASSWORD: "<EnterYourPasswordHere!!!>"
      ACCEPT_EULA: "Y"

  azure_storage_emulator:
    container_name: azurite
    image: mcr.microsoft.com/azure-storage/azurite
    restart: always
    ports:
      - "10001:10001"
      - "10000:10000"

Hi Mike, thanks! Yes that works lovely indeed. now if you succeed to add another container to docker compose that connects to Azurite and let me know how you did it, you will have answered my question :) — Wiebe Tijsma, Jan 08 '20 at 22:31

Connecting to azurite using a hostname fails

5 Answers5

Linked