4

Our application performs several batches of TableBatchOperation. We ensure that each of these table batch operations has

  • 100 or fewer table operations
  • table operations for one entity partition key only

Along the lines of the following:

        foreach (var batch in batches)
        {
            var operation = new TableBatchOperation();
            operation.AddRange(batch.Select(x => TableOperation.InsertOrReplace(x)));
            await table.ExecuteBatchAsync(operation);
        }
  • When we use emulated storage we 're hitting a Microsoft.WindowsAzure.Storage.StorageException - "Element 99 in the batch returned an unexpected response code."
  • When we use production Azure, everything works fine.

Emulated storage is configured as follows:

<add key="StorageConnectionString" value="UseDevelopmentStorage=true;" />

I'm concerned that although everything is working OK in production (where we use real Azure), the fact that it's blowing up with emulated storage may be symptomatic of us doing something we shouldn't be.

I've run it with a debugger (before it blows up) and verified that (as per API):

  • The entire operation is only only 492093 characters when serialized to JSON (984186 bytes as UTF-16)
  • There are exactly 100 operations
  • All entities have the same partition key

See https://learn.microsoft.com/en-us/dotnet/api/microsoft.windowsazure.storage.table.tablebatchoperation?view=azurestorage-8.1.3

EDIT: It looks like one of the items (#71/100) is causing this to fail. Structurally it is no different to the other items, however it does have some rather long string properties - so perhaps there is an undocumented limitation / bug?

EDIT: The following sequence of Unicode UTF-16 bytes (on a string property) is sufficent to cause the exception:

r     e     n     U+0019         space
114 0 101 0 110 0 25 0 115 0 32 0

(it's the bytes 25 0 115 0 i.e. unicode end-of-medium U+0019 which is causing the exception).

EDIT: Complete example of failing entity:

JSON:

{"SomeProperty":"ren\u0019s ","PartitionKey":"SomePartitionKey","RowKey":"SomeRowKey","Timestamp":"0001-01-01T00:00:00+00:00","ETag":null}

Entity class:

public class TestEntity : TableEntity
{
    public string SomeProperty { get; set; }
}

Entity object construction:

var entity = new TestEntity
{
    SomeProperty = Encoding.Unicode.GetString(new byte[]
        {114, 0, 101, 0, 110, 0, 25, 0, 115, 0, 32, 0}),
    PartitionKey = "SomePartitionKey",
    RowKey = "SomeRowKey"
};
eddiewould
  • 1,555
  • 16
  • 36
  • 1
    Can you share how the last entity's data look like? What happens when you remove that last entity from the batch operation? Does the batch operation succeed? – Gaurav Mantri Jun 26 '17 at 07:13
  • The last entity doesn't look special at all. Just a POCO (deriving from `Microsoft.WindowsAzure.Storage.Table.TableEntity`). It doesn't succeed with the last entity removed (we see "Element 98 in the batch..."). My testing shows that with this particular data it succeeds with 70 items but fails with 71. I've since tried it with just item #71 in isolation and it fails - so the problem must be with that item. – eddiewould Jun 26 '17 at 11:38
  • I would be very curious to see the data for the failed entity. – Gaurav Mantri Jun 26 '17 at 11:41
  • Using Windows Azure Storage Emulator 5.1.0.0 – eddiewould Jun 26 '17 at 13:07
  • I've posted on the MSDN forums here: https://social.msdn.microsoft.com/Forums/azure/en-US/b62dda82-c656-4ce1-ad45-c3936187c4fe/storage-emulator-not-handling-certain-unicode-normal-azure-storage-is-fine?forum=windowsazuredata – eddiewould Jun 26 '17 at 13:07
  • Do you have a try to use the **same data** with Azure table stage? Do you have a try to just remove the item#71? According to you comment it seems that the special item cauases the issue. `does have some rather long string properties - so perhaps there is an undocumented limitation / bug?` According to [official document](https://learn.microsoft.com/en-us/azure/azure-subscription-service-limits#storage-limits),`Max size of a table entity is 1M` – Tom Sun - MSFT Jun 27 '17 at 06:21
  • @Tom Sun - MSFT: As per the question text, the same data stores fine with real Azure. Item #71 causes the issue with emulated storage (unsupported unicode character, nothing to do with length/size). Please read it all again :) – eddiewould Jun 27 '17 at 09:52
  • Do you mind supplying the Item #71 for reproducing that? – Tom Sun - MSFT Jun 27 '17 at 09:58
  • I've added an example of an entity that fails using emulated storage but succeeds using real Azure storage. – eddiewould Jun 27 '17 at 10:18

1 Answers1

0

According to your description, I also can reproduce the issue that you mentioned. After I tested I found that the special Unicode Character 'END OF MEDIUM' (U+0019) seems that not supported by Azure Storage Emulator. If replace to other unicode is possible, please try to use another unicode to instead of it.

we also could give our feedback to Azure storage team.

Tom Sun - MSFT
  • 24,161
  • 3
  • 30
  • 47
  • Why is that? Surely the emulator should support valid C# strings just like real Azure does. Seems like we should raise a bug report - I couldn't find the place to do it, though. – eddiewould Jun 27 '17 at 21:44
  • 1. `Why is that?` Just based on my test. I also find some related info from the [document](https://learn.microsoft.com/en-us/azure/storage/storage-use-emulator) . `If you use a version of the storage services that is not yet supported by the emulator, the storage emulator returns a VersionNotSupportedByEmulator error (HTTP status code 400 - Bad Request)` 2. `I couldn't find the place to raise a bug`. As I mentioned above, we could give our **[feedback](https://feedback.azure.com/forums/217298-storage)** to azure team. – Tom Sun - MSFT Jun 28 '17 at 00:46
  • I've posted an "idea" using that feedback link. Really, I want a mechanism to report a bug. But thanks anyway :) – eddiewould Jun 28 '17 at 03:42