Getting a list of files recursively using SFTP in shell or Python with no additional libraries

Question

I am transferring files between two servers via SFTP using python's subprocess module. The only way I can connect to the remote server is via an SFTP connection.

I need to verify that the two directories on the local and remote server are identical after the transfer. This is pretty easy on the local server, a basic find command gives me what I need. However I have no clue how to get a similar result on the remote server.

Here's an example of the file structure, its identical on both machines.

JobDirectory
    Job1
        test.txt
        tonks.txt
    Job2
        wildlife.txt
    Job3
        jackinthebox.txt
        happygilmore.txt
        sadprogrammer.txt

So I need a command that'll get the filenames from Job1, Job2, and Job3 and return them to me.

Something like

echo "ls *.txt" | sftp -q user@host.example.com:/path

doesn't track too well here, since it needs a specific path. I could get a list of folders within the directory and run the sftp command against each of them, but that's a lot of remote connections.

The only remote access tools I can use are subprocess and Python's OS module. Something like Paramiko SFTP is not available.

@MartinPrikryl, this answer doesn't really meet my requirements. The first will open an SFTP connection for every directory and is written in bash, the 2nd uses a tool I do not have access too — ronald mcdolittle, Jan 16 '19 at 16:55
Who removed the other comments/posts here? They were useful and I was looking into their suggestions — ronald mcdolittle, Jan 16 '19 at 16:56
For the deleted comments and answer: All were deleted by their respective owners, once they have realized that they are wrong. -- So no, they were definitely *not* useful. — Martin Prikryl, Jan 16 '19 at 17:43

score 1 · Accepted Answer · answered Jan 16 '19 at 17:42

For an easy but inefficient solution, see the answer by @pasabaporaqui to List recursively all files on sftp.

With your obscure limitation, the only solution that uses one connection would be:

open an sftp subprocess in Python
feed sequence of ls commands to it, one for each directory
parsing the directory listings coming on standard output, producing more ls commands for each subdirectory found.

Getting a list of files recursively using SFTP in shell or Python with no additional libraries

1 Answers1