How can I iterate from a list of source files and locate those files on my disk drive? I'm using FD and RIPGREP

Question

I have a very long list of files stored in a text file (missing-files.txt) that I want to locate on my drive. These files are scattered in different folders in my drive. I want to get whatever closest available that can be found.

missing-files.txt

wp-content/uploads/2019/07/apple.jpg
wp-content/uploads/2019/08/apricots.jpg
wp-content/uploads/2019/10/avocado.jpg
wp-content/uploads/2020/04/banana.jpg
wp-content/uploads/2020/07/blackberries.jpg
wp-content/uploads/2020/08/blackcurrant.jpg
wp-content/uploads/2021/06/blueberries.jpg
wp-content/uploads/2021/01/breadfruit.jpg
wp-content/uploads/2021/02/cantaloupe.jpg
wp-content/uploads/2021/03/carambola.jpg
....

---

Here's my working bash code:

while read p;
do
    file="${p##*/}"
    /usr/local/bin/fd "${file}" | /usr/local/bin/rg "${p}" | /usr/bin/head -n 1 >> collected-results.txt
done <missing-files.txt

---

What's happening in my bash code:

1. I iterate from my list of files
2. I use FD (https://github.com/sharkdp/fd) command to locate those files in my drive
3. I then piped it to RIPGREP (https://github.com/BurntSushi/ripgrep) to filter the results and find the closest match. The match I'm looking for should match the same file and folder structure. I only limit it to one result.
4. Then finally stored it on another text file where I can later then evaluate the lists for next step

---

Where I need help:

1. Is this the most effecient way to do this? I have over 2,000 files that I need to locate. I'm open to other solution, this is something I just divised.
2. For some reason my coded broke, It stopped returning results to "collected-results.txt". My guess is that it broke somewhere in the second pipe right after the FD command. I haven't setup any condition in case it encounters an error or it can't find the file so it's hard for me to determine.

---

Additional Information:

• I'm using Mac, and running on Catalina
• Clearly this is not my area of expertise

Answer 1

"Missing" sounds like they do not exist where expected.
What makes you think they would be somewhere else?

If they are, I'd put the filenames in a list.txt file with enough minimal pattern to pick them out of the output of find.

$: cat list.txt
/apple.jpg$
/apricots.jpg$
/avocado.jpg$
/banana.jpg$
/blackberries.jpg$
/blackcurrant.jpg$
/blueberries.jpg$
/breadfruit.jpg$
/cantaloupe.jpg$
/carambola.jpg$

Then search the whole machine, which is gonna take a bit...

$: find / | grep -f list.txt
/tmp/apricots.jpg
/tmp/blackberries.jpg
/tmp/breadfruit.jpg
/tmp/carambola.jpg

Or if you want those longer partial paths,

$: find / | grep -f missing-files.txt

That should show you the actual paths to wherever those files exist IF they do exist on the system.

Answer 2

Is this the most effecient way to do this?

I/O is mostly usually the biggest bottleneck. You are running some software fd to find the files for one file one at a time. Instead, run it to find all files at once - do single I/O for all files. In shell you would do:

find . -type f '(' -name "first name" -o -name "other name" -o .... ')'

How can I iterate from a list of source files and locate those files on my disk drive?

Use -path to match the full path. First build the arguments then call find.

findargs=()
# Read bashfaq/001
while IFS= read -r patt; do
    # I think */ should match anything in front.
    findargs+=(-o -path "*/$patt")
done < <(
    # TODO: escape glob better, not tested
    # see https://pubs.opengroup.org/onlinepubs/009604499/utilities/xcu_chap02.html#tag_02_13
    sed 's/[?*[]/\\&/g' missing-files.txt
)
# remove leading -o
unset findargs[0]
find / -type f '(' "${findargs[@]}" ')'

Topics to research: var=() - bash arrays, < <(...) shell redirection with process substitution and when to use it (bashfaq/024), glob (and see man 7 glob) and man find.

Answer 3

From the way I understand it, you want to find all files what could match the directory structure:

path/to/file

So it should return something like "/full/path/to/file" and "/another/full/path/to/file"

Using a simple find command you can get a list of all files that match this criteria.

Using find you can search your hard disk in a single go with something of the form:

$ find -regex pattern

The idea is now to build pattern, which we can do from the file missing_files.txt. The pattern should look something like .*/\(file1\|file2\|...\|filen\). So we can use the following awk to do so:

$ sed ':a;N;$!ba;s/\n/\|/g' missing_files.txt

So now we can do exactly what you did, but a bit quicker, in the following way:

pattern="$(sed ':a;N;$!ba;s/\n/\|/g' missing_files.txt)"
pattern=".*/\($pattern\)"
find -regex "$pattern" > file_list.txt

In order to find the files, you can now do something like:

grep -F -f missing_files file_list.txt

This will return all the matching cases. If you just want the first case, i.e.

awk '(NR==FNR){a[$0]++;next}{for(i in a) if (!(i in b)) if ($0 ~ i) {print; b[i]}}' missing_files file_list.txt