16

I've queued up over 10,000 files to be uploaded to a UNIX based FTP server using a freeware (Windows based) FTP client which as far as i can see has finished without error.

Now, when i view the remote directory (using the Windows software) the output is truncated to 10,000 filenames. This ever occurs when i use the Windows command line FTP tool. Is there a way i can see more than this limit using another piece of software? I just need to confirm all files where indeed uploaded.

Any ideas?

Any information regarding this limit is very welcome.

Gary Willoughby
  • 50,926
  • 41
  • 133
  • 199

8 Answers8

10

Please provide more information like specific OS and version. Also please list the names and versions of the ftp clients that you are using when you experience the problem.

A lot of files in a single directory can cause many programs to have problems. I would recommend reducing the number of files in a folder, by putting them in several folders.

The FTP client that you are using probably has an option to look at the raw log and you should be able to use that to determine if it's a problem with the server or your client. The clients could be using a slightly different method to retrieve the list of files.

The story is in the logs.

bruceatk
  • 5,118
  • 2
  • 26
  • 36
  • 2
    To correct this problem, edit pure-ftpd configuration file at /etc/pure-ftpd.conf and modify value of LimitRecursion, by default, it is set to 2000. LimitRecursion 2000 8 <----- change this line to the following. LimitRecursion 20000 8 LimitRecursion <-----'ls' recursion limits. 20000 <-----Maximum number of files to be displayed. 8 <-----Maximum subdirectories depth. Now upto 20000 files will listed by FTP Client. To make these changes in effect, you have to restart pure-ftpd. root@server [~]# /etc/init.d/pure-ftpd restart – Adriaan Nel Jun 29 '15 at 07:29
  • @AdriaanNel's solution works great. One note: make sure you also restart your local FTP application, not just by reconnecting but closing all instances and restarting it. I had a cache problem of some sort with FileZilla and needed a restart of the local client in addition to Adriaan's solution. – Elly Post Oct 07 '15 at 11:26
  • 1
    I used SFTP to avoid this issue on FileZilla. –  Jun 12 '16 at 17:25
6

I also ran into this issue recently in FileZilla (it's non-client specific, btw). The setting is known as LimitRecursion. Talk to your host to adjust this rate. I simply created a ticket to LiquidWeb and this was fixed in a matter of minutes. For anyone else running into this issue.

solepixel
  • 797
  • 2
  • 9
  • 19
  • 1
    For others: It's easy to adjust yourself if you're using Pure FTPD server. http://www.inmotionhosting.com/support/website/general-server-setup/incrrease-file-display-limit – Elly Post Feb 02 '17 at 17:33
  • I was crazy about this problem with hostgator. I thought I lost the files! Filezilla is showing 9998 files! When I check with cpanel file manager, there is much more. After your answer I learned that this is a setting set by the hostgator. Thanks! – Tarik Jul 06 '17 at 17:08
4

I just created 11,000 files on an FTP server I have access to and then used Firefox to display the directory with a URL beginning ftp://hostname/pub/.... Firefox displays all 11,000 files just fine.

Norman Ramsey
  • 198,648
  • 61
  • 360
  • 533
2

If your unix host supports sftp(most modern ones does) then you should use it, winscp is a free sftp client for windows. You can use it to view/synchronize local&remote directories. It will securely transfer all files with encryption and hash checking

Tutul
  • 726
  • 8
  • 15
2

What software is the server running for ftp? If its vsftp, then you might be reaching shell/kernel limits and not the ftp server limits. Also 10K is a tad to much for one directory, seriously might want to consider doing directory spanning ( /1-3 characters of file/4-5 characters of filename/fullname.suffix As long as its consistent, this will make it a lot more manageable to find files.

David
  • 17,673
  • 10
  • 68
  • 97
1

You can run CrossFTP client, and choose Tools -> Synchornize Directory to compare your local folder and the remote folder. If there is any files missing, you can easily find them.

Some FTP Servers will truncate the file list if it is too big, for example, ProFTPd. You need to check the FTP log message to make sure there is no truncate on the server side as well.

0

You should be able to get to the raw FTP interface or somehow otherwise be able to submit single ftp commands directly. If you can, this list gives you some options. . (It's a windows list, but unix should be no worse. The following uses unix ftp commands as best I recall.) The bang (!) is supposed to drop you to a cmd prompt (I'm sure there are security issues involved. If you can somehow "ls | wc" you'll have a count, or perhaps you can "ls" to a file you can ftp back with the file list.

dkretz
  • 37,399
  • 13
  • 80
  • 138
0

If you have shell/rd access just (ls -al|dir) > filename then do a diff on that file and a similar file from your local machine.

UnkwnTech
  • 88,102
  • 65
  • 184
  • 229