1

I'm trying to download files using wget and for some reason I think I'm encountering some problems with string interpolation. I have a list of files to be downloaded that I have successfully parsed etc (no small feat for me) and would like to incorporate these into a for loop wget statement combo which downloads these files en masse.

Please forgive me the URLs won't work for you because the password and data have been changed.

I have tried single and double quotes as well as escaping a few characters in the URL (the &s and @s which I presume are at the marrow of this).

   list of files

   files.txt

   /results/KIDFROST@LARAZA.com--B627-.txt.gz      
   /results/KIDFROST@LARAZA.com--B626-part002.csv.gz   
   /results/KIDFROST@LARAZA.com--B62-part001.csv.gz

wget statement works as a single command

   wget -O files/$i "https://tickhistory.com/HttpPull/Download?           user=KIDFROST@LARAZA.com&pass=RICOSUAVE&file=/results/KIDFROST@LARAZA.com--N6265-.txt.gz"

but a loop doesn't work

   for i in `cat files.txt`; do  
       wget -O files/$i "https://tickhistory.com/HttpPull/Download?  user=KIDFROST@LARAZA.com&pass=RICOSUAVE&file=$i"
   done
tripleee
  • 175,061
  • 34
  • 275
  • 318
Chase CB
  • 1,561
  • 1
  • 13
  • 20
  • Apart from the `for` vs `while` problem, your code looks fine. How does it break? Do you get an error message? – tripleee Jan 26 '14 at 07:31
  • /6319-report.csv.gz: No such file or directory /6264-confirmation.txt.gz: No such file or directory – Chase CB Jan 26 '14 at 09:49
  • despite these messages I know the files are there and, moreover, my path is "correct" (but obviously something is up). – Chase CB Jan 26 '14 at 09:51
  • The error message seems to be saying you have lost the `/results` prefix somehow. – tripleee Jan 26 '14 at 13:13

1 Answers1

0

Edit: Main issues was "files/$i" instead of "files$i" (see comments)

Don't use a for-loop/cat for reading files, use a while/read loop. Less problems with buffer-overflow/splitting/etc...

while read -r i; do
  wget -O "files$i" "https://tickhistory.com/HttpPull/Download?  user=KIDFROST@LARAZA.com&pass=RICOSUAVE&file=$i"
done < files.txt

Also make sure you quote variables to prevent expansion/splitting errors. Also not really sure why you're doing files/$i, since $i has full path.

Reinstate Monica Please
  • 11,123
  • 3
  • 27
  • 48
  • Correct but tangential; unlikely to actually solve the OP's problem. – tripleee Jan 26 '14 at 07:31
  • @tripleee I think either some weird expansion or the fact that files/$i (which becomes e.g. files//results/KIDFROST@LARAZA.com--B627-.txt.gz) doesn't make any sense is reasonably likely to solve the problem here. Though agree would help to have more info on error. – Reinstate Monica Please Jan 26 '14 at 07:37
  • files/$i would be where i want to save the downloads. many thanks for considering my problem. – Chase CB Jan 26 '14 at 09:46
  • @ChaseCB Yes, but since the paths in your input file already start with a `/`, it will expand to e.g. `files//results/KIDFROST@LARAZA.com--B627-.txt.gz` which can never exist. If you do have a `files` subdirectory then you probably want `wget -O "files$i" ...` – Reinstate Monica Please Jan 26 '14 at 10:46
  • thanks again the "files$i" worked.......i sincerely appreciate your help if you ever write a book or become a tutor let me know. – Chase CB Jan 26 '14 at 21:09