0

I need to implement a deployment pipeline, and at the end of the pipeline, we are uploading a file, in this case, to Huawei's app store. But for a file with more than 5 megabytes in size, we have to use a chunked API. I'm not familiar of how chunked uploads work. Can someone give me an implementation guideline, preferably in java of how to implement such mechanism? The API parameters are as follow :

enter image description here

Edit : In response in the comment below, let me clarify my question. Looking up for some references of how to do a chunked request, libraries such as httpclient and okhttp simply set the chunk flag to true, and seemed to hide the details from the library's client :

https://www.java-tips.org/other-api-tips-100035/147-httpclient/1359-how-to-use-unbuffered-chunk-encoded-post-request.html

Yet, the Input Parameters of the API seems to expect that I manage the chunk manually, since it expect ChunkSize and a sequence number. I'm thinking that I might need to use the plain java http interface to work with the API, yet I failed to find any good source to get me starting. If there is anyone who could give me a reference or an implementation guidance, that will definitely help.

More updates : I tried to manually chunk my file into several parts, each of 1 megabyte in size. Then I thought I could try calling the API for every chunk, using a multipart/form-data. But the server side always close the connection before writing even begin, causing : Connection reset by peer: socket write error.

It shouldn't be a proxy issue, since I have set it up, and I could get the token, url and auth code without problem.

Mycotina
  • 369
  • 3
  • 11
  • 1
    Yeah, why not ask Huawei? Also, where are your efforts in coding this? – Smutje Jan 30 '20 at 08:38
  • Yeah, I would be glad if I have a direct support from Huawei of how to implement this, which I didn't. But I'm pretty sure that chunked request is not Huawei's proprietary technology, so I'm confident this community can help to at least point me to the right direction. Isn't this the very reason why this forum exist? And why so much judge? I'm still tinkering on it and will definitely share the answer if I could manage to solve it before I got any answer. But if someone with experience can tell me how to solve it with 5 minutes of writing, isn't it the benefit for the comunity as well? – Mycotina Jan 31 '20 at 03:24
  • Your question lacks code and self-initiative. People here are usually glad to help with mistakes, but you are more or less asking for either a tutorial and/or for someone to do the work for you. That is not the purpose of stackoverflow from what I understood. – maio290 Jan 31 '20 at 08:51

1 Answers1

0
  1. File segmentation: a file with more than a few gigabytes is uploaded to the server. If you can only use the simplest upload, receive, process and succeed, I can only say that your server is very good. Even if the server is good enough, this operation is not allowed. So we have to find a way to solve this problem.

First of all, we have to solve the problem of large files. There is no way to cut them into several m bytes and send them to the server many times and save them. Then name these files with MD5 + index of the source file. Of course, some friends use UUID + index to name them. The differences between the two will be described in detail below. When you upload these small files to the server separately, it is better to save these records to the database.

(1) When the first block upload is completed, write the name, type, MD5, upload date, address and unfinished status of the source file to a table, and change the splicing completion status to finished. Temporarily named file table

(2) After each block upload, the record is saved in the database. The MD5 + index name of the source file, the MD5 of the block (this is a key point), the upload time and the file address. Save into database and name it file__ TEM table

  1. Second transmission function: many online disks realize this function. At the beginning of upload, send Ajax request to query the existence of the file to be uploaded. Here, H5 provides a method to obtain the MD5 file, and then use ajax to request whether the MD5 exists in the file and whether the status is completed. If it exists, also verify whether the local file still exists. In the case of simultaneous existence. You can return the presence status to the front desk, and then you can proudly tell the customer, seconds passed. here is the link: https://blog.csdn.net/weixin_42584752/article/details/80873376
Mary
  • 11
  • 1