-2

I am trying to copy files over from an old file structure where data are stored in folders with improper names to a new (better) structure, but as there are 671 participants I need to copy, I want to use regex in order to streamline it (each participant has files saved in the same format). However, I keep getting a cp: cannot stat error message saying that no file/directory exists. I had assumed this meant that I had missed a / or put "" in the wrong location but I cannot see anything in the code that would suggest it.

My code is as follows (which I add a lot of comments so other collaborators can understand):

#!/bin/bash
# This code below copies the initial .nii file.
# These data are copied into my Trial Participant folders.

# Create a variable called parent_folder1 that describes initial mask directory e.g. folders for each participant which contains the files.

parent_folder1="/this/path/here/contains/Trial_Participants"

# The original folders are named according to ClinicalID_scandate_randomdigits e.g. folder 1234567890_20000101_987654.
# The destination folders are named according to TrialIDNumber e.g. LBC100001.
# The .nii files are saved under TrialIDNumber_1_ICV.nii.gz e.g. LBC1000001_1_ICV.nii.gz.
# These files need copied over from their directories into the Trial Participant folders, using the for loop function.
# The * symbol is used as a wildcard.

for i in $(ls -1d "${parent_folder1}"/*_20*); do
        lbc=$(ls ${i}/finalMasks/*ICV* | sed 's/^.*\///'); lbc=${lbc:0:9}
        cp "${parent_folder1}/${i}"/finalMasks/*_1_ICV.nii.gz /this/path/is/the/destination/path/${lbc}/
done

# This code uses regular expression to find the initial ICV file.
# ls asks for a list, -1 makes each new folder on a new line, d is for directory.
# *_20* refers to the name of the folders. The * covers the ClinicalID, _20* refers to the scan date and random digits.
# I have no idea what the | sed 's/^.*\///' does, but I think it strips the path.
# lbc=${lbc:0:9} is used to keep the ID numbers. 
# cp copies the files that are named under TrialIDNumber(replaced by *)_1_ICV.nii.gz to the destination under the respective folder.
Wiktor Stribiżew
  • 607,720
  • 39
  • 448
  • 563
fsneden
  • 1
  • Replace `cp` with `echo cp`. What is the output? Are the file(s) indeed missing? What's the difference to the actual filenames? – Gereon Jan 06 '23 at 12:20
  • 1
    Thanks to [**not** parsing `ls` output](http://mywiki.wooledge.org/ParsingLs) – Gilles Quénot Jan 06 '23 at 12:21
  • @Gereon Anonymised it by changing clinically identifiable material, but the echo cp looks like this. `cp /DSTORE/BRICIA/LBC1936/LBC1936_Structural_NewProcessed/Wave1//DSTORE/BRICIA/LBC1936/LBC1936_Structural_NewProcessed/Wave1/1234567890_20100422_16224/finalMasks/*_1_ICV.nii.gz /DSTORE/BRICIA/LBC1936/fsneden/LBC1936_T1W/Scan1Wave2_T1W/LBC361256/` Files are not missing. It is stating the parent folder path that contains subfolders of every participant (named by clinical ID), then the subfolder where the file is found, and then the destination where the file SHOULD copy to. – fsneden Jan 06 '23 at 13:36
  • When the source file(s) actually exist, does the target directory `/DSTORE/BRICIA/LBC1936/fsneden/LBC1936_T1W/Scan1Wave2_T1W/LBC361256/` exist as well? – Gereon Jan 06 '23 at 14:14
  • Did you intend .../Wave1/DSTORE/... instead of .../Wave1//DSTORE/... ? Do you have the "//" in your actual code? or is that a mis-type here? – Eric Marceau Jan 08 '23 at 02:35

1 Answers1

0

So after a bit of fooling around, I changed the code a lot (took out sed as it confuses me), and came up with this that worked. Thanks to those who commented!


# Create a variable called parent_folder1 that describes initial mask directory.
parent_folder1="/original/path/here"

# Iterate over directories in parent_folder1
for i in $(ls -1d "${parent_folder1}"/*_20*); do
  # Extract the base name of the file in the finalMasks directory
  lbc=$(basename $(ls "${i}"/finalMasks/*ICV*))
  # Extract the LBC number from the file name
  lbc=${lbc:0:9}
  # Copy the file to the specific folder
  cp "${i}"/finalMasks/${lbc}_1_ICV.nii.gz /destination/path/here/${lbc}/
done
fsneden
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jan 09 '23 at 13:22