Multi-variable string not being returned via SFTP get command

Question

I'm working on a ksh script to retrieve a file every hour via sftp that will be put on a job scheduler to be run hourly. The script must navigate to a folder with yesterday's date (remote/path/yyyymmdd/). The filename also has yesterday's date and a timestamp (filename_yyyymmdd_hhmmss.dat). Since the job will be scheduled, my script has to include the previous hour - ex. if the job runs at 11:02, the file to retrieve would be filename_yyyymmdd_10mmss.dat. The minutes and seconds will always be the same - ex 4949. There will be multiple files in the remote directory and I only want to retrieve the latest one so that there are not multiple input files being processed by our jobs. The remote directory will also have other files being created regularly, so I can't retrieve just the last modified files.

I have variables to return yesterday's date and the previous hour, but the sftp command isn't returning the full filename and isn't retrieving the file. I've tried concatenating the variables, using brackets & quotes & parenthesis, assigning multiple variables to a single variable, and exporting the variables.

vdate=$(TZ=bb24 date '+%Y%m%d')
vhour=$(date '+%H')
prevhour=$((vhour - 1))

sftp user@host << EOF
lcd /my/dir/
cd /remote/path/$vdate/
get filename_$vdate_$prevhour*.dat
bye
EOF
exit

When running the script, the file cannot be found and the full filename isn't returned:

File "/remote/path/20190411/filename_20190411" not found.

instead of

File "/remote/path/20190411/filename_20190411_10*.dat" not found.

Every combination of variables that I try returns the same not found - ending after filename_$vdate.

I've tried some other combinations but always get the same not found:

newvar=${vdate}_${prevhour}
get filename_$newvar*.dat

and

newvar=${vdate}\\_${prevhour}
get filename_$newvar*.dat

File "/remote/path/20190411/filename_20190411" not found.

`*(s)ftp` clients are notoriously problematic (I didn't say buggy :-) ). Your code looks like it should work. I would manually run `sftp` and enter each of the cmds. When you get to `get file...` I would replace that with `dir file...*` and 1. confirm the file is really there and 2. see if you spot some reason why your `filename_$vdate_$prevhour*.dat` isn't working. If you can get it work by manually typing everything, then you have to look at what is being processed by `ksh`. Harder to do inside of a here-doc. .... — shellter, Apr 12 '19 at 19:22
Ah, maybe you need to use `mget` even if there is only one file. The `*` maybe throwing it off. .... Otherwise, in a separate terminal window, manually execute assignments to `filename_$vdate_$prevhour*.dat`, then `echo filename_$vdate_$prevhour*.dat` on the command line and copy/paste that value into your `sftp` session. Good luck! — shellter, Apr 12 '19 at 19:23
Please try with this format `get filename_${vdate}_${prevhour}*.dat` — lw0v0wl, Apr 13 '19 at 13:06
@shellter @edvin I tried both methods and was unsuccessful. Running the commands manually and echoing returns the desired output, but once I enter the commands manually through sftp the output doesn't match up. Is it possible there's a character limit or a limit on the number of variables in the sftp client? I may end up just retrieving `filename*` and having a script delete the copies that are not from the previous hour. — dcap4518, Apr 14 '19 at 18:14

lw0v0wl · Accepted Answer · 2019-04-15T02:15:17.030

You have a problem in your script at prevhour=$((vhour - 1)) this way a text 02 after you make subtraction, it will be 1 and not 01 and it will match to undesired files, or even none as 00 - 1 is -1

[edvin]$ vdate=$(TZ=bb24 date '+%Y%m%d')
[edvin]$ vhour=$(date '+%H')
[edvin]$ prevhour=$((vhour - 1))
[edvin]$ echo $vhour
03
[edvin]$ echo $prevhour
2
[edvin]$ prevhour=$(date -d '1 hour ago' '+%H')
[edvin]$ echo $prevhour
02

date's -d option not avaliable on some system.

I believe in that in your attempt the shell is considered the * as part of the variable prevhour as you did not put it into {} that separate variables from sorrunding text.

This is my working solution based by your attempt:

#!/bin/ksh

r_host='server2'
r_user='edvin'
l_dir='./content'
r_dir='./Test_folder'

# this still not cover the case of midnight
# it put 00 to 23 but day have to be yesterday as well
##vdate=$(TZ=bb24 date '+%Y%m%d')
##vhour=$(date '+%H')                      # not used
##prevhour=$(date -d '1 hour ago' '+%H')

# vtime = YYYYmmdd_HH -1 H
vtime=$(TZ=bb24 date -d '1 hour ago' '+%Y%m%d_%H')

sftp ${r_user}@${r_host} << EOF
lcd ${l_dir}
cd ${r_dir}
get filename_${vtime}*.dat
bye
EOF

exit

Output:

[edvin]$ ./script.ksh 
Connected to server2.
sftp> lcd ./content
sftp> cd ./Test_folder
sftp> get filename_20190415_02*.dat
Fetching /home/edvin/Test_folder/filename_20190415_020000.dat to filename_20190415_020000.dat
Fetching /home/edvin/Test_folder/filename_20190415_020100.dat to filename_20190415_020100.dat
Fetching /home/edvin/Test_folder/filename_20190415_020200.dat to filename_20190415_020200.dat
Fetching /home/edvin/Test_folder/filename_20190415_020300.dat to filename_20190415_020300.dat
Fetching /home/edvin/Test_folder/filename_20190415_020400.dat to filename_20190415_020400.dat
Fetching /home/edvin/Test_folder/filename_20190415_020500.dat to filename_20190415_020500.dat
Fetching /home/edvin/Test_folder/filename_20190415_020600.dat to filename_20190415_020600.dat
Fetching /home/edvin/Test_folder/filename_20190415_020700.dat to filename_20190415_020700.dat
Fetching /home/edvin/Test_folder/filename_20190415_020800.dat to filename_20190415_020800.dat
Fetching /home/edvin/Test_folder/filename_20190415_020900.dat to filename_20190415_020900.dat
Fetching /home/edvin/Test_folder/filename_20190415_021000.dat to filename_20190415_021000.dat
sftp> bye

There is many thing can go wrong still in this solution, like if remote directory not exist, not accessible, script will still go on with the rest of the command, same for the local directory and for the files as well. The connection also can run various problems you might want to handle. You like to schedule it so might a solution needed to avoid script spawn over and over again if one already run.

scp would be more preferred way to do this, as you use password less authentication.

If scp is not an option for some reason, with expect this can be handled quite well.

I ended up going with `mget *.dat` then after exiting sftp using `ls -1 filename*.dat | sort -r | tail +2 | xargs -i rm {}` to remove all but the latest version based on the timestamp in the filename. I wasn't able to get the commands to execute properly via sftp. — dcap4518, May 02 '19 at 13:20

Multi-variable string not being returned via SFTP get command

1 Answers1