accurate method to compare the s3 object and local object

Question

I saved the modified time in ms and file size into the object's metadata in s3. I realized even if I did not change anything for my file if I open it then just save the file without editing. The modified time will be changed and in this case it will update the s3 object. I thought of using size but size wouldn't be as accurate too because there are chances for the size to be the same even after modified. I also used the Binary got back from s3.getObject and locally's file Binary but without any changes. The Binary wont be the same too. What would be a better more accurate way to track changes?

I have something like this in my code which saves the file modified ms and file size

fs.readFile(path, async (err, fileBinary) => {
      if (err) throw err;

      const s3 = new AWS.S3();
      const Key = path.replace(process.env.WATCH_PATH, '');
      const filename = Key.split('/').pop();

      // if filename is within the regex, ignore the file.  Do nothing.
      if (new RegExp(IGNORE_FILES_TO_S3()).test(filename)) return false;

      const getStat = await getFileStat(path);
      // console.log(getStat, 'getstatsssssssssssssss');
      const s3PutParams = {
          Body: fileBinary,
          Bucket: process.env.S3_BUCKET,
          Key,
          Metadata: {  // thought of saving these two as comparison in future usage, which works but really really accurate though
              mtimeMs: String(getStat.mtimeMs),
              size: String(getStat.size)
          }
      };
// rest of the code here just do comparisons and decide if `s3.putOjbect` should be done or not.
});

my getFileStat()

exports.getFileStat = (path) => {
    /*
    SAMPLE: success
    {
    dev: 2097,
    mode: 33204,
    nlink: 1,
    uid: 1000,
    gid: 1000,
    rdev: 0,
    blksize: 4096,
    ino: 5639856,
    size: 2,
    blocks: 8,
    atimeMs: 1545952029779.866,
    mtimeMs: 1545952020431.9802,
    ctimeMs: 1545952020439.98,
    birthtimeMs: 1545952020439.98,
    atime: 2018-12-27T23:07:09.780Z,
    mtime: 2018-12-27T23:07:00.432Z,
    ctime: 2018-12-27T23:07:00.440Z,
    birthtime: 2018-12-27T23:07:00.440Z
    }
    */
    return new Promise((res, rej) => {
        fs.stat(path, (err, stat) => {
            if (err) rej(err);
            res(stat);
        });
    });
};

Thanks in advance for any suggestions and help.

PS. This is not saving anything into the DB so no info will be saved at all in case there's an idea of saving something into DB for comparison purpose

I'm kinda confused about what you want the end result to be and what exactly your question is. — Charlie Fish, Dec 28 '18 at 01:02
@CharlieFish sryz I didn't post the rest of the codes after having `const s3PutParams` the rest is just to use the right comparison to compare the object then decide either to `putObject` or not — Dora, Dec 28 '18 at 03:03

score -1 · Answer 1 · answered Dec 28 '18 at 01:10

-1

To compare the contents of a local file with an Amazon S3 object, use the ETag, which is a checksum on the contents. The ETag is available when retrieving information about an S3 object.

See: All about AWS S3 ETags - Teppen.io

Also, please note that an object uploaded via multi-part upload has a slightly more complex calculation. See: What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?

answered Dec 28 '18 at 01:10

John Rotenstein

241,921
22
380
470

1

John - The etag is no longer reliable for a content hash (md5). https://stackoverflow.com/a/53886736/8016720 – John Hanley Dec 28 '18 at 01:59
@JohnHanley I could be completely wrong. But I think the OP is talking about metadata. `The ETag reflects changes only to the contents of an object, not its metadata`. – Charlie Fish Dec 28 '18 at 03:15

accurate method to compare the s3 object and local object

1 Answers1

Linked