0

I am using a REST API below to create a new file on ADLS gen1

https://mydomain.azuredatalakestore.net/webhdfs/mypath/test_50MB.txt?op=CREATE&write=true

I tried to upload a 50MB text file and the response is 413 (large entities) and the file cannot be created on the storage.

Then I tried with another file (28MB) and it worked.

I did search some posts and found some guys got the same issue as me.

And now, I am wondering what exactly maximum file size I can create on the storage using that REST API?

I found the official post of that API at https://learn.microsoft.com/en-us/rest/api/datalakestore/webhdfs-filesystem-apis, but it does not mention the limitation.

NoName
  • 877
  • 12
  • 28
  • Have you not just tried uploading files of varying sizes to see where the limit is? – iamdave Jul 13 '21 at 09:08
  • 1
    Similar to the following question https://stackoverflow.com/questions/54134106/azure-data-lake-hdfs-upload-file-size-limit – Anupam Chand Jul 13 '21 at 10:09
  • @iamdave: Yes, I tried uploading files of varying sizes and found that 28MB is able to be uploaded successfully with the CREATE command. – NoName Jul 15 '21 at 09:16
  • @AnupamChand: No, your post is not the answer to my case. I am talking about the CREATE REST API, not APPEND FILE API – NoName Jul 15 '21 at 09:16
  • Yes, there is no official documentation on the file size limit. If you check https://social.msdn.microsoft.com/Forums/azure/en-US/48e86ce8-79f8-4412-838f-8e2a60b5f387/notfound-error-on-call-to-data-lake-store-create?forum=AzureDataLake , it says the limit (for SDK) is 30000000 bytes or about 28.6MB and check Tim's response in the same link I sent earlier where he suspects the SDK to be using the Rest API under the covers and he is talking of the create file api itself. Looks like this conforms with the limit you got. – Anupam Chand Jul 15 '21 at 09:32

0 Answers0