Bash script to obtain the newest file X in a folder and create a new variable called X+1

Question

I am trying to create a loop in Bash script for a series of data migrations:

At the beginning of every step, the script should get the name of the newest file in a folder

called "migrationfiles/ and store it in the variable "migbefore" and create a new variable called "migbefore+1":

    Example:  if the "migrationfiles/" folder contains the following files: 

    migration.pickle1            migration.pickle2            migration.pickle3

The variable "migbefore" and migafter should have the following value:

    migbefore=migration.pickle3
    migafter=migration.pickle4

At the end of every step, the function "metl", which is in charge of making the data migration, uses the file "migbefore" to load the data and creates 1 new file called "migafter" and stores it in the "migrationfiles/" folder, so in this case, the new file created will be called:

    "migration.pickle4"

The code I pretend using is the following:

#!/bin/bash

migbefore=0
migafter=0

for y in testappend/*
    for x in migrationfiles/*
    do
         migbefore=migration.pickle(oldest)
         migafter=migbefore+1

    done


do
    metl -m migrationfiles/"${migbefore}" 
         -t migrationfiles/"${migafter}" 
         -s "${y}" 
         config3.yml

done

Does anyone know how I could make the first loop (The one that searches for the newest file in the "migrationfiles/" folder) and then assigns the name of the variable "migafter" as "migbefore+1"?

Is this something that will be run with arbitrary migration files already existing (so at the start the newest file might be `migration.pickle6`) or will it always start from `0`/`1`? — Etan Reisner, Sep 11 '14 at 16:46
Hello Etan, it will only start only once with 0 and 1 and never go back ideally. — Johan Garzon, Sep 11 '14 at 16:51
And the files in `testappend` control how many times `metl` is going to be called? — Etan Reisner, Sep 11 '14 at 16:52
Yes, but for each iteration in the main 'For' metl only runs once with the respective file "${y}" — Johan Garzon, Sep 11 '14 at 16:56
Seem your first `do` have to be moved 6 lines higher: imediately after `for y in testappend/*`. — F. Hauri - Give Up GitHub, Sep 11 '14 at 17:26

score 0 · Answer 1 · answered Sep 11 '14 at 16:49

Copy with Numbered Backups

It's really hard to tell what you're really trying to do here, and why. However, you might be able to make life simpler by using the --backup flag from the cp command. For example:

cp --backup=numbered testappend/migration.pickle migrationfiles/

This will ensure that you have a sequence of migration files like:

migration.pickle
migration.pickle.~1~
migration.pickle.~2~
migration.pickle.~3~

where the older versions have larger ordinal numbers, while the latest version has no ordinal extension. It's a pretty simple system, but works well for a wide variety of use cases. YMMV.

score 0 · Accepted Answer · answered Sep 11 '14 at 17:05

I think this might do what you want.

#!/bin/bash

count=0
prefix=migration.pickle
migbefore=$prefix$((count++))
migafter=$prefix$((count++))

for y in testappend/*; do
    echo metl -m migrationfiles/"${migbefore}" \
         -t migrationfiles/"${migafter}" \
         -s "${y}" \
         config3.yml

    migbefore=$migafter
    migafter=$prefix$((count++))
done

score 0 · Answer 3 · answered Sep 11 '14 at 17:20

0

# configuration:
path=migrationfiles
prefix=migration.pickle

# determine number of last file:
last_number=$( find ${path} -name "${prefix}*" | sed -e "s/.*${prefix}//g" | sort -n | tail -1 )

# put together the file names:
migbefore=${prefix}${last_number}
migafter=${prefix}$(( last_number + 1 ))

# test it:
echo $migbefore $migafter

This should work even if there are no migration files yet. In that case, the value of migbefore is just the prefix and does not point to a real file.

answered Sep 11 '14 at 17:20

Michael Jaros

4,586
1
22
39

Thanks a lot Michael, I implemented this script together with the one provided by Etan and everything worked well. – Johan Garzon Sep 12 '14 at 08:03
Although I just realized and if ever there is a migration.pickle file with an end number of more than one digit (ie 10) then the tail -1 statement will not work right? – Johan Garzon Sep 25 '14 at 16:06
Why do you think so? After removing the prefix (which is everything but the number) from each line, numeric sort (`-n`) is applied, then the last line (the one with the highest number) selected. – Michael Jaros Sep 30 '14 at 20:43
Yes you're right, I am not very familiar with Bash script and I thought that tail -1 actually got the last element from the name's string, thanks a lot! – Johan Garzon Oct 02 '14 at 07:51

Bash script to obtain the newest file X in a folder and create a new variable called X+1

3 Answers3

Copy with Numbered Backups