Create folders automatically and move files

Question

I have a lot of daily files that are sort by hours which comes from a data-logger (waveform). I downloaded inside a USB stick, now I need to save them inside folders named with the first 8 characters of waveform.

Those files have the following pattern:

Year-Month-Day-hourMinute-##.Code_Station_location_Channel

for example, inside the USB I have:

2020-10-01-0000-03.AM_REDDE_00_EHE; 2020-10-01-0100-03.AM_REDDE_00_EHE; 2020-10-02-0300-03.AM_REDDE_00_EHE; 2020-10-20-0000-03.AM_REDDE_00_EHE; 2020-10-20-0100-03.AM_REDDE_00_EHE; 2020-11-15-2000-03.AM_REDDE_00_EHE; 2020-11-15-2100-03.AM_REDDE_00_EHE; 2020-11-19-0400-03.AM_REDDE_00_EHE; 2020-11-19-0900-03.AM_REDDE_00_EHE;

I modified a little a code from @user3360767 (shell script to create folder daily with time-stamp and push time-stamp generated logs) to speed up the procedure of creating a folder and moving the files to them

for filename in 2020-10-01*EHE; do  
    foldername=$(echo "$filename" | awk '{print (201001)}');
    mkdir -p "$foldername"  
    mv "$filename" "$foldername"
    echo "$filename $foldername" ;
done

2020-10-01*EHE

Here I list all hours from 2020-10-01-0000-03.AM_REDDE_00_EHE

foldername=$(echo "$filename" | awk '{print (201001)}');

Here I create the folder that belongs to 2020-10-01 and with the following lines create the folder and then move all files to created folder.

    mkdir -p "$foldername"  
    mv "$filename" "$foldername"
    echo "$filename $foldername" ;

As you may notice, I will always need to modify the line for filename in 2020-10-01*EHE each time the file changes the date. Is there a way to try to create folders with the first 8 number of the file?

Tonino

Your question is quite hard to follow... it just launches into a load of stuff that doesn't work. Maybe you could click `edit` underneath it and update it by starting with what you actually do want to achieve. You want to move some files automatically each morning/night/week/month? The files are created by another program? They need to be moved (or copied?) to a new folder based just on the date (or the date and time?) in their filename (or depending on the system date?)... — Mark Setchell, Dec 02 '20 at 22:06

rohit89 · Accepted Answer · 2020-12-03T05:27:55.400

1

Use date

And since the foldername doesn't change, you don't need to keep creating one inside the loop.

files="$(date +%Y-%m-%d)*EHE"
foldername=$(date +%Y%m%d)
mkdir -p "$foldername"
for filename in $files; do  
    mv "$filename" "$foldername"
    echo "$filename $foldername" 
done

Edit:

If you want to specify the folder each time, you can pass it as an argument and use sed to get the filename pattern

foldername=$1
files=$(echo $1 | sed 's/\(....\)\(..\)\(..\)/\1-\2-\3/')
filepattern="$files*EHE"
mkdir -p "$foldername"
for filename in $filepattern; do
  mv "$filename" "$foldername"
  echo "$filename $foldername"
done

You call it with

./<yourscriptname>.sh 20101001

edited Dec 03 '20 at 05:27

answered Dec 02 '20 at 20:04

rohit89

5,745
2
25
42

the variable `files="$(date +%Y-%m-%d)*EHE"` does not list all files it only shows 2020-10-01.*EHE and then it can not move any file to folder, by the way the folder name will change for each day – tonino Dec 02 '20 at 20:57
@tonino I don't understand. `files` isn't listing all the files that match the pattern? It uses `date` so the pattern for today will be 2020-12-03*EHE or 2020-12-02*EHE depending on your timezone. – rohit89 Dec 02 '20 at 21:07
I did a bad explication, the files are waveform that comes from a data-logger, this equipment saves information each hour in the format mentioned, I downloaded the data from past months. The main idea is to move the files to a folder named with the 8 first characters (20201001), this is the tricky part I am not able to manage – tonino Dec 03 '20 at 01:57
1

@tonino If I understood correctly, you will have to pass it as an argument to the script, check my edit. – rohit89 Dec 03 '20 at 05:29

Mark Setchell · Answer 2 · 2020-12-03T09:41:13.413

I think you want to move all files whose names end in *EHE into subdirectories. The subdirectories will be created as necessary and will be named according to the date at the start of each filename without the dashes/hyphens.

Please test the following on a copy of your files in a temporary directory somewhere.

#!/bin/bash

for filename in *EHE ; do
    # Derive folder by deleting all dashes from filename, then taking first 8 characters
    folder=${filename//-/}
    folder=${folder:0:8}
    echo "Would move $filename to $folder"
    # Uncomment next 2 lines to actually move file
    # mkdir -p "$folder"
    # mv "$filename" "$folder"
done

Sample Output

Would move 2020-10-01-0000-03.AM_REDDE_00_EHE to 20201001
Would move 2020-10-01-0100-03.AM_REDDE_00_EHE to 20201001

Note that the 2 lines:

folder=${filename//-/}
folder=${folder:0:8}

use "bash parameter substitution", which is described here if you want to learn about it, and obviate the need to create whole new processes to run awk, sed or cut to extract the fields.

works fine and very useful the bash parameter substitution – tonino Dec 04 '20 at 14:57 — tonino, Dec 04 '20 at 14:57

Create folders automatically and move files

2 Answers2