proftpd locking up after a set number of processes reached

Question

We have a proftpd server (1.3.3g) running on centos6 which when it seems to get over approximately 150 processes running locks up and prevents any more connections.

The proftpd server connects to MYSQL to handle user authentication.

I have run the proftpd paranoid log and can see no failures in that as well as checking the secure log for any login failures with no problems in that.

Monitoring shows no CPU/Memory/Disk/Network spikes during the times that it falls over it just seems to lock up until the connections drop back down again. The machine should be fine to handle more than 150 concurrent users (E3-1271v3 32GB RAM).

PROFTPD CONFIG

# This is a basic ProFTPD configuration file (rename it to
# 'proftpd.conf' for actual use.  It establishes a single server
# 'proftpd.conf' for actual use.  It establishes a single server
# and a single anonymous login.  It assumes that you have a user/group
# "nobody" and "ftp" for normal operation and anon.

ServerName                      "PROHIBITED FTP ACCESS"
DeferWelcome                     off
#ServerType                      standalone

# Globbing
UseGlobbing off

TransferLog /var/log/ftpxferlog
UseReverseDNS off
IdentLookups off
UseFtpUsers off 
WtmpLog off
UseIPv6 off


# Restrict the range of ports from which the server will select when sent the
# PASV command from a client. Use IANA-registered ephemeral port range of
# 49152-65534
PassivePorts                    49152 65534

Port                            21
Umask                           022

TimeoutLogin                    120
TimeoutIdle                     300
TimeoutNoTransfer               300
TimeoutStalled                  300

# Default to show dot files in directory listings
ListOptions "-a +R" strict
# ListOptions "" maxdepth 3
# ListOptions "" maxdirs 10
ListOptions "" maxfiles 2000
AllowOverride off

# Set the user and group that the server normally runs as.
User                            www
Group                           www

# Set path locations
ScoreboardFile                  /var/run/proftpd.score
#DefaultRoot                    /data/filesroot/ftproot/pub
DefaultRoot                     /data/filesroot/ftproot

# Log formats
LogFormat                      default "%h %l %u %t \"%r\" %s %b"
#LogFormat                      anonymous "%h %l %u %t \"%r\" %s %b"
#LogFormat                      auth    "%v [%P] %h %t \"%r\" %s"
#LogFormat                      write   "%h %l %u %t \"%r\" %s %b"

# SQL authentication Dynamic Shared Object (DSO) loading
# See README.DSO and howto/DSO.html for more details.
<IfModule mod_dso.c>
   LoadModule mod_sql.c
   LoadModule mod_sql_mysql.c
   LoadModule mod_ifsession.c
</IfModule>

# Global settings
<Global>

        AuthOrder               mod_sql.c

        SQLEngine               on
        SQLAuthenticate         users groups
        SQLConnectInfo          xxx@xxx:3306 USERNAME PASSWORD
        SQLAuthTypes            Backend
        SQLUserInfo             ftpusers username passwd uid gid NULL NULL

        SQLDefaultHomedir       /data/filesroot/ftproot/

        RequireValidShell       off
        SQLGroupInfo            ftpgroups groupname gid members
        SQLDefaultGID           65533
        SQLDefaultUID           65533
        SQLMinID                350

        ServerIdent on "FTP Server ready."
        AllowOverwrite          yes
        IdentLookups            off
        DelayEngine             off

        # Logging
        # file/dir access
        #ExtendedLog            /var/log/proftpd/access.log WRITE,READ

        # Record all logins
        #ExtendedLog            /var/log/proftpd/auth.log AUTH

        # Paranoia logging level....
        #ExtendedLog            /var/log/proftpd/paranoid.log ALL
</Global>

<Limit LOGIN>
      Order allow, deny
      DenyAll
</Limit>

# Deny writing to the base server...
<Directory /data/filesroot/ftproot/pub/*>
    <Limit WRITE>
    DenyAll
    </Limit>
</Directory>

<Limit WRITE>
DenyAll
</Limit>

<Directory />
        HideNoAccess on
        <Limit WRITE>
        DenyAll
        </Limit>
</Directory>


<VirtualHost xxx.xxx.xxx.xxx>

        ServerAdmin             xxx@xxx.com
        ServerName              "FTP"

        DefaultRoot             /data/filesroot/ftproot
        SQLDefaultHomedir       /data/filesroot/ftproot/

        TransferLog             /data/logs/ftp/files/files.xferlog

        RequireValidShell       off
        AllowOverwrite          on
        AllowRetrieveRestart    on
        AllowStoreRestart       on
        MaxLoginAttempts        2
        MaxClients              2000 "Sorry, maximum users reached."
        MaxClientsPerUser       5
        MaxHostsPerUser         2

        # How quickly do we kick someone out?
        TimeoutLogin            45
        TimeoutIdle             15
        TimeoutNoTransfer       300

        # Port 21 is the standard FTP port.
        Port                    21

        # Umask 022 is a good standard umask to prevent new dirs and files
        # from being group and world writable.
        Umask                   022

        # Set the user and group that the server normally runs at.
        User                    www
        Group                   www

        # Set Anonymous access controls
        <Anonymous /data/filesroot/ftproot/pub>
                User                    www
                Group                   www
                UserAlias               anonymous www
                RequireValidShell       off
                MaxClients              1

                <Limit WRITE>
                DenyAll
                </Limit>
                # Don't write anonymous accesses to the system wtmp file (good idea!)
                WtmpLog      off
        </Anonymous>            

</VirtualHost>

150 concurrent connections seems very small for a FTP server. Any insights would be appreciated

1.3.3 is a quite old version of ProFTPD; any chance you can upgrade? Also, you might see if the [debug logging](http://www.proftpd.org/docs/howto/Debugging.html) reveals more clues... — Castaglia, Jan 18 '19 at 04:29
@Castaglia We installed with EPEL repo and it seems that's the highest version it goes up to on Centos6. I'm currently setting up a Centos7 box for use as a DNS round robin balancer so will see how performance is on that. The proftpd version on that is 1.3.5e. Just ran the "proftpd -nd10" command but didn't see any errors showing up in that. — Mike, Jan 19 '19 at 00:48
@Castaglia thanks for your help, actually found the issue due to your linking of that debug page. the nd-10 revealed it was a MySQL issue. The tables were MyISAM and not InnoDB so when it got busy the whole table locking caused a cascade effect. changing those tables to INNODB has solved the issues as its now only locking rows. — Mike, Jan 21 '19 at 14:08

score 2 · Accepted Answer · answered Jan 21 '19 at 14:12

2

The problem turned out to be that the database tables were MyISAM and not InnoDB so when it got busy the whole table locking caused a cascade effect. changing those tables to INNODB has solved the issues as its now only locking rows.

answered Jan 21 '19 at 14:12

Mike

21
3

proftpd locking up after a set number of processes reached

PROFTPD CONFIG

1 Answers1