-3

I am using a combination of find and copy command in my backup script. it is used on a fairly huge amount of data,

  1. first, out of 25 files it needs to find all the files older than 60 mins
  2. then copy these files to a temp directory - each of these files are 1.52GB to 2GB

  3. one of these 25 files will have data being appended continuously.

    I have learnt from googling that Tarring operation will fail if there is an update going on to the file being attempted to tar, is it the same thing with find and copy also?? I am trying something like this,

    /usr/bin/find $logPath -mmin +60 -type f -exec /bin/cp {} $logPath/$bkpDirectoryName \;

  4. after this I have a step where I tar the files copied to the temp directory as mentioned above(&bkpDirectoryName), here I use as mentioned below,

    /bin/tar -czf $bkpDir/$bkpDirectoryName.tgz $logPath/$bkpDirectoryName

    and this also fails.

the same backup script was running from past many days and suddenly it has started failing and causing me headache! can someone please help me on this??

mklement0
  • 382,024
  • 64
  • 607
  • 775
Vasanth Nag K V
  • 4,860
  • 5
  • 24
  • 48
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/72794/discussion-between-vasanth-nag-k-v-and-etan-reisner). – Vasanth Nag K V Mar 11 '15 at 19:54
  • 2
    By definition a file that is still actively growing will **not** be found by `find -mmin +60` because that command returns only files that were last **modified** over one hour ago. So if you truly mean `-mmin +60`, then the issue of the growing file is irrelevant. If you are still having difficulty, please [edit your original post](http://stackoverflow.com/posts/28995618/edit) with a **complete** copy of your code and all the errors you are seeing. Merely stating "it fails" is not nearly enough to help us help you. – dg99 Apr 02 '15 at 17:04
  • can you post the error messages please – Biswajit_86 Apr 03 '15 at 04:28
  • As @dg99 asked, does the list of 25 old files contain the growing one? And like Etan said, active writing will continue in the moved file. But some programs open the logfile for each write action. Will the file be recreated when you move it? – Walter A Apr 04 '15 at 13:37
  • Is it possible that `tar` starts while copying still hasn't finished? – baf Apr 05 '15 at 20:07
  • hi Walter and dg99, the 25 files which i am talking about does not include the continuously changing file. all 25 files are static and nothing is written to it at the time of copying or moving. i am running tar inside an if condition and the condition always fails giving an output 2, thats all the error message i get, and that is what i meant by saying "it fails" – Vasanth Nag K V Apr 07 '15 at 10:26

3 Answers3

0

can you try these steps please

  1. instead of copying files older than 60 min, move them.
  2. run the tar on the moved files

If you do the above, the file which is continuously appended will not be moved.

In case any of your other 24 files might be updated after 60 min, you can do the following

  1. Once you move a file, touch a file with the same name in case there are async updates which are not continuous.
  2. When tarring the file, give a timestamp name to the tar file.This way you have a rolling tar of your logs

If nothing works due to some custom requirement on your side, try doing a rsync and then do the same operations on the rsynced files (i.e find and tar or just tar)

Biswajit_86
  • 3,661
  • 2
  • 22
  • 36
0

try this 

output=`find $logPath -mmin 60 -type f`
if [ "temp$output" != "temp" ];then
  cp -rf $output $other_than_logPath/$bkpDirectoryName/
else 
   echo sorry
fi

I think, you are using +60 instead of 60.

I also want to know, at what interval your script gets called.

Pratrick
  • 7
  • 3
0
#!/bin/bash
for find in `find / -name "*" -mmin 60`
do
 cp $find / ## Choose directory
done

That's basically what you need, just change the directory I guess//

Dersuss
  • 1
  • 4