2

I have two scripts that watch a directory (postgres warm-standby)

one script copies compressed files into the directory and then uncompresses them.

the other script watches for the uncompressed files and then ingests them into postgres.

Im wondering if i could his a case where the ingestor script could read teh output of gunzip while it was writing it?

in short: does gunzip

  1. decompress and then change name
  2. copy into new name as it uncompresses
  3. something else
Arthur Ulfeldt
  • 3,249
  • 9
  • 33
  • 40

2 Answers2

1

It's probably not a good idea to base the correctness of your scripts on assumptions about how a particular utility (such as gunzip) works, since the internal behaviour of a utility can change in subsequent releases. I recommend, instead, that you use the proper locking methods (as provided by your particular OS) to ensure mutual exclusion. Most Linux distros provide flock for use in shell scripts.

Steven Monday
  • 13,599
  • 4
  • 36
  • 45
1

It is non-atomic, as you can see if you do an ls during a large compression:

dfranke@alatar:~$ dd if=/dev/zero of=bigfile bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 39.1106 s, 82.4 MB/s
dfranke@alatar:~$ gzip bigfile &
[1] 19876
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     114688 2010-11-08 17:18 bigfile.gz
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     212992 2010-11-08 17:18 bigfile.gz
dfranke
  • 379
  • 1
  • 7
  • @Arthur: Even more informative is `strace gunzip bigfile.gz`. You'll see an `open` of `bigfile.gz`, then `read`, then `open` of `bigfile`, then `write` to `bigfile`, `close` of `bigfile.gz`, `close` of `bigfile`, then finally `unlink` of `bigfile.gz`. (I used a smaller file, so you'd probably see more `read` and `write` operations for a bigger file.) – Dennis Williamson Nov 08 '10 at 22:43