Sync local file with HTTP server location (in Python)

Question

I have an HTTP server which host some large file and have python clients (GUI apps) which download it.
I want the clients to download the file only when needed, but have an up-to-date file on each run.

I thought each client will download the file on each run using the If-Modified-Since HTTP header with the file time of the existing file, if any. Can someone suggest how to do it in python?

Can someone suggest an alternative, easy, way to achieve my goal?

So you have some code already downloading the file in your GUI app? It would help if we could see it. — Rob Cowie, Oct 01 '11 at 23:45
Jochen- rsync don't work against HTTP server, no? Rob- my current code is just urllib.urlopen(url) if the file is not available... — Uri Cohen, Oct 02 '11 at 20:11

score 1 · Answer 1 · answered Oct 02 '11 at 01:09

1

You can add a header called ETag, (hash of your file, md5sum or sha256 etc ), to compare if two files are different instead of last-modified date

answered Oct 02 '11 at 01:09

TheOneTeam

25,806
45
116
158

Can I add this ETag to a standard HTTP server? – Uri Cohen Oct 02 '11 at 20:13

score 0 · Answer 2 · answered Oct 02 '11 at 00:49

I'm assuming some things right now, BUT.. One solution would be to have a separate HTTP file on the server (check.php) which creates a hash/checksum of each files you're hosting. If the files differ from the local files, then the client will download the file. This means that if the content of the file on the server changes, the client will notice the change since the checksum will differ.

do a MD5 hash of the file contents, put it in a database or something and check against it before downloading anything.

Your solution would work to, but it requires the server to actually include the "modified" date in the Header for the GET request (some server softwares does not do this).

I'd say putting up a database that looks something like:

[ID] [File_name] [File_hash]

0001 moo.txt asd124kJKJhj124kjh12j

score 0 · Accepted Answer · edited May 23 '17 at 12:03

0

It seems to me the easiest solution is hosting the file in mercurial and using mercurial api to find the file's hash, downloading the file if the hash has changed. Calculating the hash can be done as the answer to this question; for downloading the file urllib will be enough.

edited May 23 '17 at 12:03

Community

1
1

answered Oct 04 '11 at 22:08

Uri Cohen

3,488
1
29
46

Sync local file with HTTP server location (in Python)

3 Answers3