List recursively all files on sftp

Question

I'd like to write bash script to recursively list all files (with fullpaths) on sftp and interact with paths locally afterwards (so only thing for what sftp is needed is getting the paths). Unfortunately the "ls -R" doesn't work there.

Any idea how to do that with some basic POC would be really appreciated

Available commands:

bye                                Quit sftp

cd path                            Change remote directory to 'path'

chgrp grp path                     Change group of file 'path' to 'grp'

chmod mode path                    Change permissions of file 'path' to 'mode'

chown own path                     Change owner of file 'path' to 'own'
df [-hi] [path]                    Display statistics for current directory or
                               filesystem containing 'path'
exit                               Quit sftp
get [-Ppr] remote [local]          Download file
help                               Display this help text
lcd path                           Change local directory to 'path'
lls [ls-options [path]]            Display local directory listing
lmkdir path                        Create local directory
ln [-s] oldpath newpath            Link remote file (-s for symlink)
lpwd                               Print local working directory
ls [-1afhlnrSt] [path]             Display remote directory listing
lumask umask                       Set local umask to 'umask'
mkdir path                         Create remote directory
progress                           Toggle display of progress meter
put [-Ppr] local [remote]          Upload file
pwd                                Display remote working directory
quit                               Quit sftp
rename oldpath newpath             Rename remote file
rm path                            Delete remote file
rmdir path                         Remove remote directory
symlink oldpath newpath            Symlink remote file
version                            Show SFTP version
!command                           Execute 'command' in local shell
!                                  Escape to local shell
?                                  Synonym for help

Sorry, I forgot to mention that it's not available as well... — Radek Feber, May 28 '15 at 10:42
Why do you have so few commands available ? That list is missing bash builtins ?? — 123, May 28 '15 at 10:46
These commands are listed directly from sftp (customer's one so nothing to do about it) — Radek Feber, May 28 '15 at 10:53
One suggestion: Ask for an SFTP client that has this functionality or at least can be scripted from bash in the [software recommendations stackexchange](https://softwarerecs.stackexchange.com/). — smheidrich, May 28 '15 at 11:01
I posted a solution using solely `echo` and `cd` but apparently the server doesn't even have an `echo` command, can you verify this? If this is true, there is no possible way to achieve your goal. — ShellFish, May 28 '15 at 11:23
@RadekFeber I just realised you are using SFTP not SSH. To do anything meaningful you should use use SSH instead. SFTP is purely for file transfer and provides only those limited commands, you cannot run scripts inside SFTP. — 123, May 28 '15 at 11:53
There is probably a solution that can be written in the local shell based on building up a recursive directory tree from multiple connections to different directories on the server. I'm not willing to try it right now myself, though. — kojiro, May 28 '15 at 11:58
possible duplicate of [How to list directory content of remote FTP, recursively](http://stackoverflow.com/questions/98224/how-to-list-directory-content-of-remote-ftp-recursively) — pasaba por aqui, May 28 '15 at 14:00
Clarification: the list of "commands" show in the question is the list of commands supported by sftp. — pasaba por aqui, May 28 '15 at 14:01

score 6 · Answer 1 · answered May 28 '15 at 14:31

This recursive script does the job:

#!/bin/bash 
#

URL=user@XXX.XXX.XXX.XXX
TMPFILE=/tmp/ls.sftp

echo 'ls -1l' > $TMPFILE

function handle_dir {
  echo "====== $1 ========="
  local dir=$1
  sftp -b $TMPFILE "$URL:$dir" | tail -n +2 | while read info; do
    echo "$info"
    if egrep -q '^d' <<< $info; then
       info=$(echo $info)
       subdir=$(cut -d ' ' -f9- <<< $info)
       handle_dir "$dir/$subdir"
    fi
  done
}

handle_dir "."

fill URL with the sftp server data.

Hesitantly +1 -- perhaps explicitly point out that this will open one `sftp` connection for every directory. — tripleee, May 30 '16 at 10:24

score 5 · Answer 2 · answered Apr 19 '17 at 11:06

I scan the whole Internet and find a great tool sshfs. Mount the remote directory tree through SSHFS. SSHFS is a remote filesystem that uses the SFTP protocol to access remote files.

Once you've mounted the filesystem, you can use all the usual commands without having to care that the files are actually remote.

sshfs helps me a lot, may give you help, too.

mkdir localdir
sshfs user@host:/dir localdir
cd localdir
find . -name '*'

List recursively all files on sftp

2 Answers2

Linked

Related