2

I have a storage server running Nexenta (OpenSolaris kernel, Ubuntu userspace) with MySQL on top of a ZFS storage array, using innodb_file_per_table and ulimit -n set to 8K. mysqltuner.pl confirms the file limit and claims there are 169 files.

The following command:

pfiles `fuser -c / 2>/dev/null

indicates one mysqld process having 485 file/device descriptors (and they're almost all for files) so I don't know how reliable the tuning script is, but it is still way less than 8K and this list also finds no other process which is close to it's limit. The global total number of descriptors in use is around 1K.

So what can cause mysqld to be constantly streaming the following errors?

[date] [host] mysqld[pid]: warning: cannot open /etc/hosts.allow: Too many open files
[date] [host] mysqld[pid]: warning: cannot open /etc/hosts.deny: Too many open files

Everything appears to actually be operating fine, but the issue is constantly flooding the admin console and starts right away on a fresh boot (not only reproducible, but always from mysqld and always the hosts files, whose permissions are the default -rw-r--r-- 1 root root). I could, of course, suppress it from the admin console but I'd rather get to the bottom of it and still allow mysqld warnings/errors to reach the admin console.

EDIT: not only is the actual file descriptor well within sane limits, the issue also persists (with immediate appearance) even with the file limit raised to 65535 and always only on hosts.allow/deny.

Stefan Lasiewski
  • 23,667
  • 41
  • 132
  • 186
HonoredMule
  • 187
  • 11
  • Interesting... I ran into a similar problem with MySQL, but with FreeBSD 8.2 on ZFS (which utilizes the OpenSolaris/Illumos code): http://unix.stackexchange.com/questions/66863/freebsd-too-many-open-files-but-should-be-able-to-open-another-160-000-files – Stefan Lasiewski Mar 04 '13 at 22:19
  • Do you perform ZFS snapshots on the system and does this correspond with when the "Too many open files" errors begin? – Stefan Lasiewski Mar 04 '13 at 23:14

1 Answers1

1

Given that Nexenta was OpenSolaris-based at that time, you may be running into this issue. It does seem like file descriptor limits aren't being applied in a consistent manner throughout the system.

Are you actually using hosts.allow/deny in your setup?

What does lsof output look like?

ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • Curious. Luckily, none of my systems actually rely on use of hosts.allow/deny. Less luckily, I do not have `lsof` in Nexenta. – HonoredMule Dec 11 '12 at 22:43
  • I installed mine via apt. `SunOS deore 5.11 NexentaOS_134f i86pc i386 i86pc Solaris` and `Open Storage Appliance (v3.1.3.5)` - `dpkg -p nexenta-lsof` – ewwhite Dec 11 '12 at 22:50
  • Is that OpenIndiana-based? I'm still on v3.0.5 and have no available upgrades (expected given the lack of upgrade path from the original Ubuntu/ON-based project). – HonoredMule Dec 11 '12 at 22:57
  • I changed my apt sources list... It's still OpenSolaris-based. – ewwhite Dec 11 '12 at 23:00
  • There doesn't seem to be any way to verify or disprove that this is in fact the particular reason I'm encountering, so I guess it count's as _an_ answer. Until I can take the time to upgrade to OpenIndiana, I guess I'll have to live with it. I wish that thread had more details on the bug being reported. – HonoredMule Dec 13 '12 at 21:41
  • 1
    I believe that hosts.allow/deny are used by the `libwrap` library, which is used by most daemons on the system. So the errors with hosts.allow/deny is not the source of your troubles, but it is yet another symptom. – Stefan Lasiewski Mar 04 '13 at 23:42