How to know if object has fully uploaded to amazon s3?

Question

How can we check if an object has been fully uploaded to Amazon s3? I am uploading images to s3 from android app after which lambda function is invoked. I want to make sure that the object has been uploaded to s3 before I invoke lambda. Currently large file uploads are creating problems.

Answer 1

You can check it by using PutObjectResult The response indicates that the object has been successfully stored. Amazon S3 never stores partial objects: if you receive a successful response, then you can be confident that the entire object was stored. So the main key player is MD5 algorithm, using which you can confirm if the object that you wanted to upload is same that got uploaded, To inorder for you to know the MD5 value of a any document, lets say its an image you can use the below to find so

`    package aws.example.s3;

    import java.io.File;
    import java.io.FileInputStream;
    import java.io.IOException;
    import java.security.MessageDigest;
    import java.security.NoSuchAlgorithmException;

    public class MD5demo {
      public static String MD5check(String path){
        // TODO Auto-generated method stub
        //Create checksum for this file
        File file = new File(path);
    
        //Get the checksum
        String checksum="null";
        try {
            //Use MD5 algorithm
            MessageDigest md5Digest;
            try {
                md5Digest = MessageDigest.getInstance("MD5");
                checksum = getFileChecksum(md5Digest, file);
                //see checksum
                //System.out.println("Checksum: "+checksum);
            } catch (NoSuchAlgorithmException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
         }catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        return checksum;
         
    }
    
    private static String getFileChecksum(MessageDigest digest, File file) 
     throws IOException
    {
        //Get file input stream for reading the file content
        FileInputStream fis = new FileInputStream(file);
         
        //Create byte array to read data in chunks
        byte[] byteArray = new byte[1024];
        int bytesCount = 0;
          
        //Read file data and update in message digest
        while ((bytesCount = fis.read(byteArray)) != -1) {
            digest.update(byteArray, 0, bytesCount);
        };
         
        //close the stream; We don't need it now.
        fis.close();
         
        //Get the hash's bytes
        byte[] bytes = digest.digest();
         
        //This bytes[] has bytes in decimal format;
        //Convert it to hexadecimal format
        StringBuilder sb = new StringBuilder();
        for(int i=0; i< bytes.length ;i++)
        {
            sb.append(Integer.toString((bytes[i] & 0xff) + 0x100, 
      16).substring(1));
        }
         
        //return complete hash
       return sb.toString();
      }
}`

so, you just need to pass the file path in the MD5check(String path) and it will return the MD5 value

Now, in your class where you are writing the code to upload any doc, just use the below code

PutObjectResult putObjectResult = null;
    try  {
           putObjectResult = s3.putObject(new PutObjectRequest(<Enter your Bucket name here>, <KeyName> , <pass your file object here>));
    
           String MD5=MD5check(<pls enter your file path here>);  //you can pass your file also directly, for that please change the method implementation of MD5Demo class accepting a file instead of path
           
           System.out.println("Etag: "+putObjectResult.getETag()); //getting the Etag which is nothing but same as MD5 value of the file that got uploaded in S3
           System.out.println("MD5:  "+MD5);
           
           if(MD5.equals(putObjectResult.getETag())) {
               System.out.println("Congrats!! Document Uploaded Successfully");
           }else {
               System.out.println("Sorry! Could not upload the document");
           }

Hope it helps, please let me know if you have any suggestions :)

Answer 2

Rather than invoking an AWS Lambda function after uploading an object to Amazon S3, simply configure the S3 bucket to trigger lambda itself:

_{(source: amazon.com)}

This way, the function will only be triggered when the object is successfully uploaded to S3, and there is less work for your application.

See: Using AWS Lambda with Amazon S3

If you really want to do it separately, then you could use the ETag (which is an MD5 checksum) to confirm that the file has been uploaded as expected.