7

I am writing a BASH shell script to upload all the files in a directory to a remote server and then delete them. It'll run every few hours via a CRON job.

My complete script is below. The basic problem is that the part that's supposed to figure out whether the file uploaded successfully or not doesn't work. The SFTP command's exit status is always "0" regardless of whether the upload actually succeeded or not.

How can I figure out whether a file uploaded correctly or not so that I can know whether to delete it or let it be?

#!/bin/bash

# First, save the folder path containing the files.
FILES=/home/bob/theses/*

# Initialize a blank variable to hold messages.
MESSAGES=""
ERRORS=""


# These are for notifications of file totals.
COUNT=0
ERRORCOUNT=0

# Loop through the files.
for f in $FILES
do
    # Get the base filename
    BASE=`basename $f`

    # Build the SFTP command. Note space in folder name.
    CMD='cd "Destination Folder"\n'
    CMD="${CMD}put ${f}\nquit\n"


    # Execute it.
    echo -e $CMD | sftp -oIdentityFile /home/bob/.ssh/id_rsa bob@ftp.example.edu

    # On success, make a note, then delete the local copy of the file.
    if [ $? == "0" ]; then
        MESSAGES="${MESSAGES}\tNew file: ${BASE}\n"
        (( COUNT=$COUNT+1 ))

        # Next line commented out for ease of testing
        #rm $f
    fi

    # On failure, add an error message.
    if [ $? != "0" ]; then
        ERRORS="${ERRORS}\tFailed to upload file ${BASE}\n"
        (( ERRORCOUNT=$ERRORCOUNT+1 ))
    fi

done

SUBJECT="New Theses"

BODY="There were ${COUNT} files and ${ERRORCOUNT} errors in the latest batch.\n\n"

if [ "$MESSAGES" != "" ]; then
    BODY="${BODY}New files:\n\n${MESSAGES}\n\n"
fi

if [ "$ERRORS" != "" ]; then
    BODY="${BODY}Problem files:\n\n${ERRORS}"
fi

# Send a notification. 
echo -e $BODY | mail -s $SUBJECT bob@example.edu

Due to some operational considerations that make my head hurt, I cannot use SCP. The remote server is using WinSSHD on windows, and does not have EXEC privileges, so any SCP commands fail with the message "Exec request failed on channel 0". The uploading therefore has to be done via the interactive SFTP command.

Will Martin
  • 2,431
  • 5
  • 19
  • 18

5 Answers5

8

If you batch your SFTP session, the exit status $? will tell you if it's a failed transfer or not.

echo "put testfile1.txt" > batchfile.txt
sftp -b batchfile.txt user@host
if [[ $? != 0 ]]; then
  echo "Transfer failed!"
  exit 1
else
  echo "Transfer complete."
fi

Edited to add: This is something we use in a production script at work, so I'm sure it's valid. We can't use scp because some of our customers are on SCO Unix.

UtahJarhead
  • 928
  • 7
  • 14
  • I haven't done extensive testing, but I think batch mode alters how `sftp` exits and what exit status it presents to you. Glad that did the trick. – UtahJarhead Apr 17 '12 at 15:01
  • sftp is turning out to be quite a tricky little animal. It neither supports the simple operation of putting a file into a one-liner (without shell gymnastics), nor does it bother letting you know whether there was an error unless you specifically ask it. It's like these limitations were added on purpose, even though they make no sense. Anyway, it turns out you can use batch and a dash to still read from stdin, so thanks, it worked. Using: echo 'put [somefile]' | sftp -b - [remotehost] – Teekin Dec 26 '17 at 14:20
  • For your scenario, are you sure you don't want `scp`? – UtahJarhead Dec 27 '17 at 16:35
1

It is the tradition of serverfault not to unduly question the preconditions, but I have to ask: is it not possible for you to either 1) mount the remote filesystem via SMB, or 2) use fuse/sshfs instead? (I assume here that the sending machine is a Linux box as you are using bash and ssh.)

To actually answer your question, I think your problem is simple. Consider:

quest@wonky:~$ false
quest@wonky:~$ if [ $? != "0" ] ; then echo asdf ; fi
asdf
quest@wonky:~$ if [ $? != "0" ] ; then echo asdf ; fi

Try instead like this:

quest@wonky:~$ false
quest@wonky:~$ res=$?
quest@wonky:~$ if [ $res != "0" ] ; then echo asdf ; fi
asdf
quest@wonky:~$ if [ $res != "0" ] ; then echo asdf ; fi
asdf

To explain: your second if statement "if [ $? != "0" ]; then" tests for the exit status of the latest statement, which is no longer sftp, but the previous if statement.

Then I wonder, will sftp really exit with non-zero if it has problems uploading files? Cursory tests indicate mine doesn't.

Bittrance
  • 3,070
  • 3
  • 24
  • 27
  • This is a good tip -- I didn't understand that each IF statement altered the exit status variable. So, +1, and I've adjusted my script to reflect this. But SFTP does actually exit non-zero in the script as written above -- I think because the "quit" command succeeds. Using a batch file (as in another answer) makes it generate the correct exit code, though, so between the two answers I think I've got it working. – Will Martin Apr 17 '12 at 14:28
0

Use rsync(.exe) and dig in --remove-source-files & co., don´t even care about failed uploads, he will! ;-)

Yes you can tie this up with ssh, see --rsh

Tabakhase
  • 318
  • 1
  • 4
0

Jarhead

how about run sftp Batch script on Windows OS to detect success/failure?

I use %errorlevel% to detect success/failure, sometime the %errorlevel% be returned by 0, but that session is fail and the file is not upload to the server. my script is below

how can I detect sftp uploaded correctly? THX so much

@echo off
(echo cd %4
  echo put %2
  echo exit)>sftpScript.txt
sftp -b sftpScript.txt %3
set result=%errorlevel%
if result EQU 0 (
  del sftpScript.txt
  del %2
) else (
  del sftpScript.txt
)
exit %result%
pkpk
  • 1
  • 1
0

** edited to actually use a batch file To expand on the batch based solution.
On the copy of sftp I'm running, a batch file exits on an error, even if multiple commands exist beyond the error point

By the logic originally posted, by checking $?, you're testing the result of the entire batch. If you're ok with removing some of the files that WERE successfully transmitted, then build the batch file with !rm following each put. If the put fails, the !rm won't run

(for F in *.pgp
do
echo put $F
echo !rm $F
done;echo quit) > Batchfile ; sftp -b Batchfile user@targetServer
Tom
  • 1
  • 1