0

I have few files which i did tar.gz.
As this file can get too big thus I used the Linux split. As this needs to be transferred to a different machine i have used s3 bucket to transfer these files. I used application/octet-stream content-type to upload these files. The files when downloaded shows exactly same size as original size thus no bytes are lost.

now when I do cat downloaded_files_* > tarball.tar.gz the size is exactly as the original file but only the part with _aa gets extracted.

i checked the type of files

file downloaded_files_aa

this is tar zip file(gzip compressed data, from Unix, last modified: Sun May 17 15:00:41 2020) but all other files are data files

I am wondering how can i get the files.

Note: Http upload via API gateway done to upload the files to s3

================================ Just putting my debugging finding with a hope probably it will help someone facing same problem.

As we wanted to use API gateway out upload calls were done http calls. This is something which is not using regular aws sdk.

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html Code Samples: https://docs.aws.amazon.com/AmazonS3/latest/API/samples/AWSS3SigV4JavaSamples.zip

After some debugging, we found this leg was working fine.

As the machine which we wanted to download the files had direct access to s3 we used the aws sdk for downloading the files. This is the URL https://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html

this code does not work well, though it showed the exact file size download as upload the file lost some information. The code also complained about still pending bytes. Some changes were done to get rid of error but it never worked.

the code which I found here is working like magic

InputStream reader = new BufferedInputStream(
object.getObjectContent());
File file = new File("localFilename");      
OutputStream writer = new BufferedOutputStream(new FileOutputStream(file));

int read = -1;

while ( ( read = reader.read() ) != -1 ) {
    writer.write(read);
}

writer.flush();
writer.close();
reader.close();

This code also make the download much faster then our previous approach.

R-JANA
  • 1,138
  • 2
  • 14
  • 30

0 Answers0