Our agent stoped working and the only way to get it back is by restarting the agent in every node, the error we get:
ERROR [clojure-agent-send-off-pool-11618] df failed on execute;
returning default value - Cannot run program "df": error=24, Too many open files
Why it's need to have so much files open? How we can reset it without the need to restart the agent and make it automatically.
As per the DSE documentation on production settings, make sure that you have made the following adjustments to your /etc/security/limits.d/cassandra.conf file:
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited
Note: This assumes that you are running Cassandra as the cassandra user. If that is not the case, adjust accordingly.
Why it's need to have so much files open?
Cassandra ultimately writes its data to SSTable files on the underlying filesystem. It also has to serve reads from any of those aforementioned files. Additionally, if you have rows that have been updated a lot and thus contain obsoleted data over time (and compaction has not run), it is entirely possible that a single row might need to be read from multiple files.
In short, databases need to work with many files simultaneously, and Cassandra is no exception.