How to upload multiples files in R without receiving a 421 Too many connections error?

Question

I need to upload more than 65 files to my hosting ftp server with RCurl but I'm receiving an error after several files have been uploaded :

 < 421 Too many connections (8) from this IP
* We got a 421 - timeout!
* Closing connection 291

From what I see in the log it says Connection #283 to host ftp.myserver.com left intact so that no further authorisation is needed until the connection expire after 15 minutes of inactivity < 220 You will be disconnected after 15 minutes of inactivity..

At least it is my understanding because, for some reason, it resends USER and PASSWORD right after for the second file :

> USER admin@myserver.com
< 331 User admin@myserver.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /

I understand there are multiples authorisation requests (one per file) so that I'm flooding the server.

Is there a possibility to first open a connection to the FTP server then upload all the files ? like in FileZilla ?

Please find the R code I use :

for(file in list.files(localPath)) {

  ftpUpload(paste0(localPath,file), userpwd = userpwd, to=paste0("ftp://", ftpPath, file), verbose=TRUE)

}

Another SO member has the same problem here but it looks I already use something similar to the response provided to him and it dosen't work. Using R to upload many files

Thank you

EDIT (adding the log) :

> for(file in list.files(localPath)) {
+   
+   print(paste0("ftp://", ftpPath, file))
+   ftpUpload(paste0(localPath,file), userpwd = userpwd, to=paste0("ftp://", ftpPath, file), verbose=TRUE)
+ 
+ }
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-1.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#310)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 2 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||44516|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 44516
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#310)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-1.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.180 seconds (measured here), 1.15 Kbytes per second
* Connection #310 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-2.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#311)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 3 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||45077|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 45077
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#311)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-2.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.207 seconds (measured here), 1.00 Kbytes per second
* Connection #311 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-3.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#312)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 4 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||40390|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 40390
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#312)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-3.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.189 seconds (measured here), 1.12 Kbytes per second
* Connection #312 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-4.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#313)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 5 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||46329|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 46329
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#313)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-4.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.187 seconds (measured here), 1.11 Kbytes per second
* Connection #313 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-1.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#314)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 6 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||46556|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 46556
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#314)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-1.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.186 seconds (measured here), 2.75 Kbytes per second
* Connection #314 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-2.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#315)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 7 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||43730|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 43730
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#315)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-2.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.190 seconds (measured here), 2.69 Kbytes per second
* Connection #315 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-3.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#316)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 8 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||43399|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 43399
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#316)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-3.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.196 seconds (measured here), 2.61 Kbytes per second
* Connection #316 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-4.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#317)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 9 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||48536|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 48536
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#317)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-4.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
[1] "ftp://ftp.mywebsite.com/data/csv/data_coins_average-1.csv"
< 226-File successfully transferred
< 226 0.184 seconds (measured here), 2.77 Kbytes per second
* Connection #317 to host ftp.mywebsite.com left intact
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#318)
< 421 Too many connections (8) from this IP
* We got a 421 - timeout!
* Closing connection 318
Error in function (type, msg, asError = TRUE)  : 
  Uploading to a URL without a file name!

Have you tried closing any open connections (`?closeAllConnections`)? — Roman Luštrik, Nov 04 '16 at 11:29
Hi, I just closed a `gzcon` with `closeAllConnections()` but still I have the `421 Too many connections error`. — Florent, Nov 04 '16 at 11:35

score 0 · Answer 1 · answered Nov 04 '16 at 14:49

This is a workaround rather than a solution to my problem but it works like a charm. I created a text file with ftp commands inside then launch Ubuntu ftp with R system().

ftpcommands.txt :

open ftp.mywebsite.com
user admin@mywebsite.com mypassword
lcd ~/R/data/csv
cd data/csv
prompt
mput *.csv
bye

Then the R command :

system("ftp -n < ~/R/data/ftpcommands.txt")

score 0 · Answer 2 · answered Jul 16 '17 at 17:14

It looks like you are opening for each file a new connection, which isn't how the File Transfer Protocol was meant to be used. Some different solutions:

Open less connections in your R code.
Limit the maximum concurrent connections used by curl.
Increase the allowed limit of concurrent connections in the config of you FTP server. (Do this only if you really need it, as I mentioned, don't open for each file a connection.)

Check this detailed answer out to get a better understanding of the problem. Don't understand me wrong, it's good for the performance to have a little bit more than one connection at the same time, but not for each file, that's definitively too much. The most FTP clients use 2.

How to upload multiples files in R without receiving a 421 Too many connections error?

2 Answers2