12

I am getting the following error with my storage queue (NOT BLOB - I know others have seen this with the blob) when pushing the message from my c++ app to Azure Storage:

The request body is too large and exceeds the maximum permissible limit.

I'm aware that I probably need to cut the json down, but are there any other suggestions? (as in increase message size somewhere?)

David Makogon
  • 69,407
  • 21
  • 141
  • 189
user1112324
  • 623
  • 1
  • 7
  • 23

2 Answers2

18

As others have stated, the Azure Storage Queue message size limit (64K) is a hard limit.

Aside from encoding, compression (minification), etc: The most common pattern to work around this limit is to not store your payload in the queue message; rather, store it in something like Blob storage, and only store the message type & metadata (if needed), along with a URI pointing to the blob containing your payload to process.

By following this pattern, and using blob storage for your payload, you effectively have potential payload size of 4+ TB. And you also have the ability to persist your payload if needed (whereas queue messages are deleted after processing).

David Makogon
  • 69,407
  • 21
  • 141
  • 189
  • I found an example with storage queue and blob [Dealing with large Service Bus messages using claim check pattern](https://www.serverless360.com/blog/deal-with-large-service-bus-messages-using-claim-check-pattern) – Michael Freidgeim May 17 '19 at 13:29
5

The maximum size for a message in the Azure storage queue is 64KB (48 KB when using Base64 encoding) based on the latest Azure Storage Service Limits documentation as below.

It is non-configurable and at the moment Azure support also will not increase the size upon request.

https://learn.microsoft.com/en-us/azure/azure-subscription-service-limits#storage-limits

I will suggest you to reduce the size of your JSON message e.g. JSON minify

juvchan
  • 6,113
  • 2
  • 22
  • 35
  • 2
    Adding one more thing: Since Azure uses UTF-16 encoding to store the data (i.e. use 2 bytes to store each character so to speak), message size is essentially limited to 32KB. – Gaurav Mantri Aug 15 '17 at 13:59
  • 1
    Hi @GauravMantri, is there any documentation which mention Azure storage queue message is using UTF-16 encoding? Thanks – juvchan Aug 15 '17 at 15:50
  • I remember reading it somewhere but I am not able to find that. However I have tried it myself and for any string/byte array greater than 32KB in size I got this error. – Gaurav Mantri Aug 16 '17 at 02:05
  • Is this the reason when I use the Azure Portal to put a decently large JSON message in a queue, the checkbox “Encode the message body in Base64” gets grayed out? I am not sure how much 32KB is in number of characters including spaces. – Oliver Nilsen Oct 07 '22 at 19:20