Bash script that move files from one directory with lots of files to a month folder

Question

I have a directory with thousands of files. They have a specific creation date. Now I want to archive these files on specific times to a specific directory.

Example:

Files created on:

May 15 testmay.txt
Jun 10 testjun.txt
Jul 01 testjul.txt

They should come in those directory's

/2013-05/testmay.txt
/2013-06/testjun.txt
/2013-06/testjul.txt

I already have this to rsync the files from a remote server to a temp month directory.

#!/bin/sh

GAMESERVER=game01
IP=172.1.1.1

JAAR=`date --date='' +%Y`
MAAND=`date --date='' +%m`
DAG=`date --date=''  +%d`
LOGDIR=/opt/archief/$GAMESERVER

if [ ! -e $LOGDIR/$JAAR-$MAAND ]; then
        mkdir $LOGDIR/$JAAR-$MAAND/tmp
        chmod -R 770 $LOGDIR/$JAAR-$MAAND/tmp
fi

rsync -prlt --remove-source-files -e ssh root@$IP:/opt/logs/sessions/ $LOGDIR/$JAAR-$MAAND/tmp

chmod -R 770 $LOGDIR/ -R

How can I complete this script?

Can you be more explicit about what you want to happen next? Just explain in words, step by step what you want to happen. — SethMMorton, Jul 10 '13 at 06:29
As you see in the script I catch lots of files to a tmp directory, now that tmp directory contains thousands of files. I want those files now sorted out: - File created in a specific month should be moved to a directory created for that month. — JoeriBe, Jul 10 '13 at 06:48
It looks like that is already happening... isn't `$LOGDIR/$JAAR-$MAAND/` a directory for a specific month and year? — SethMMorton, Jul 10 '13 at 18:30
Yes, but where I get the files, /opt/logs/sessions/, there are lots of files from different months. So it will copy them but not sort them according to the creation date. — JoeriBe, Jul 11 '13 at 06:59
How will you determine the creation date? Do you search within the file, or will you do it by timestamp? Also, will you be creating even more directories under $LOGDIR/$JAAR-$MAAND/ that are dates? I'm confused why you are copying all of the files into a directory named after a single date, instead of copying them into something like $LOGDIR/tmp. — SethMMorton, Jul 11 '13 at 15:19

score 5 · Answer 1 · answered Oct 29 '13 at 01:42

I've just had need to do something similar and have come up with what I think is a pretty neat way to do it. I have >1 million files in one directory that I need to archive off based on their mtime. I'm using zip to archive the files here because I want them compressed but also easily accessible from Windows systems, but you could easily replace it with just a simple mv or whatever suits.

SRC="/path/to/src"  # Where the originals are found
DST="/path/to/dst"  # Where to put the .zip file archives

FIND="find $SRC -maxdepth 1 -type f \( -name \*.tmp -o -name \*.log \)" # Base find command

BOUND_LOWER=$( date -d "-3 years" +%s ) # 3 years ago (because we need somewhere to start)
BOUND_UPPER=$( date -d "-1 years" +%s ) # 1 year ago (because we need to keep recent files where they are)

# Round down the BOUND_LOWER to 1st of that month at midnight to get 1st RANGE_START
RANGE_START=$( date -d $( date -d @$BOUND_LOWER +%Y-%m )-01 +%s )

# Loop over each month finding & zipping files until we hit BOUND_UPPER
while [ $RANGE_START -lt $BOUND_UPPER ]; do
    ARCHIVE_NAME=$( date -d @$RANGE_START +%Y-%m )
    echo "Searching for files from $ARCHIVE_NAME"
    RANGE_END=$( date -d "$( date -d @$RANGE_START ) +1 month" +%s )
    eval "$FIND -newermt @$RANGE_START \! -newermt @$RANGE_END -print0" |
        xargs -r0 zip -Tjm $DST/$ARCHIVE_NAME -@
    echo
    RANGE_START=$RANGE_END
done

For each file in $SRC that matches the $FIND conditions and is within $BOUND_* variable time range (to the nearest month), this will archive it to the appropriate $DST/YYYY-MM.zip file based on its mtime.

If you're using a version of find older than 4.3.3, see this page for examples of how to use -newer instead of -newermt, which you will just need to do inside the main loop.

score 1 · Answer 2 · answered Jul 16 '13 at 13:30

something like this?

DEBUG=echo
cd ${directory_with_files}
for file in * ; do 
    dest=$(stat -c %y "$file" | head -c 7) 
    mkdir -p $dest
    ${DEBUG} mv -v "$file" $dest/$(echo "$file" | sed -e 's/.* \(.*\)/\1/')
done

DISCLAIMER: test this in a safe copy of your files. I won't be responsible for any lost of data ;-)

score 1 · Answer 3 · answered Oct 17 '13 at 08:15

If you put

for file
do  dir=/`date +%Y-%m -r$file`
    mkdir -p $dir && mv $file $dir
done

into a script file named, let's say, archive, you can execute for example

archive *

to move all files to the desired directories. In case this produces a line too long error, execute

/bin/ls | xargs archive

instead. (If you want to be cautious, you can use the mv -i option.)

Bash script that move files from one directory with lots of files to a month folder

3 Answers3