Smart way of uploading files based on changes

Question

My app has to watch for file changes to a directory and upload a copy whenever a change is made. To do this, I've employed the use of a FileObserver. I have it upload on Creation, Move To, Close Write, and Modify. The problem lies with modify. Modify is called every time a change is written to disk which, if a file is large (being copied or a lot changed) isn't atomic so it fires hundreds of Modify events, this makes my app freak out because it's trying to upload so many times and it crashes. My first thought was to remove the modify event so that it'll only upload when Close Write is called. Unfortunately, this isn't always called. So I'm kinda stuck with Modify. So my question is this: is there a best practice when it comes to detecting the end of a file modification? I need to upload this file when it's done being modified, not while. How would you go about doing this? How can I discover when the last Modify event is fired. Do I have to make a complex timer system, or is there an easier way. (If there isn't, could you tell my the best practice of making such a timer system?)

I know I've asked a lot, but I'd appreciate any brainstorming and ideas.

Thanks!

Edit: So I've found something weird. At least on Android 4.2, Open never fires, therefore close never fires. Just an FYI.

score 1 · Answer 1 · answered Jul 24 '13 at 19:06

1

If the files are text based, you could maybe diff the original file and the modified file and only upload the file if the number of different lines are sufficiently large.

answered Jul 24 '13 at 19:06

Lee Avital

542
6
15

Text based files are actually the ones that work fine. They're more often than not atomic operations that fire Modify and Close Write once. – Osmium USA Jul 24 '13 at 19:22
+1 for creativity though, didn't think about this. It would have worked if all of the files were strictly text based. They aren't though. – Osmium USA Jul 24 '13 at 21:09

score 1 · Accepted Answer · answered Jul 24 '13 at 19:17

1

Use a worker thread that test the file for changes every x seconds and if it is changed, then send the updated version to wherever you need. To avoid sending a file that is being written test for File.isOpen or use a boolean value to keep track of occurring changes.

answered Jul 24 '13 at 19:17

type-a1pha

1,891
13
19

`File.isOpen()`? That sounds useful if I could find it. What are you talking about, it this some sort of library? – Osmium USA Jul 24 '13 at 19:24
Use a FileObserver and keep track of the open/close events. You can also use a FileLock http://developer.android.com/reference/java/nio/channels/FileLock.html to ensure mutual access to the file. If your file is accessed sequentially use a boolean variable. – type-a1pha Jul 24 '13 at 19:31
Alright, I did... sort of this. I used a hashmap and and timertask to bottleneck the multiple events into a ten second delay. Thanks! – Osmium USA Jul 24 '13 at 21:10

score 1 · Answer 3 · answered Jul 24 '13 at 19:30

With a bit of luck, one might do something like track the time between modification events, assuming that they're received more-or-less consistently…

 last-notification = now;
 notification-interval = 1 s; /* some default */
 listen for (file closed);
 listen for (file modified);
 alarm at (now + notification-interval);

 on (file modified) =
      cancel alarm;
      notification-interval = max ( notification-interval
                                  | [now - last-notification] );
      alarm at (now + 2 × notification-interval);

 on (file closed) =
      cancel alarm;
      do upload;

 on (alarm) =
       if (file is open?) then alarm at (now + 2 × notification-interval);
       else (signal file closed)

Smart way of uploading files based on changes

3 Answers3