The above answer is incorrect for modern RHEL 5 and Ubuntu installations. It was true at one point, and I cannot identify when the change occurred, but it did occur before the OP on August 30, 2011 (given the CIS benchmark published in Aug 2011). deny= needs to be on an auth line, not an account line. Various sources on the Internet are outdated/incorrect on this point.
See the man pages for pam_tally:
http://linux.die.net/man/8/pam_tally
http://manpages.ubuntu.com/manpages/hardy/man8/pam_tally.8.html
... you will see that deny is an "auth" option, not an account option.
These are the correct settings:
(in system-auth/system-auth-ac in RHEL and common-auth in Ubuntu)
#Actually locks out the user; put BEFORE pam_unix.so auth line.
auth required pam_tally2.so deny=5 onerr=fail unlock_time=900
(in system-auth/system-auth-ac in RHEL and common-account in Ubuntu)
#Resets the failed counter if the user finally gets in successfully. This is only needed to support programs that do not call pam_setcred(3) correctly (like sshd). Put BEFORE pam_unix.so account line.
account required pam_tally2.so
Note that pam_tally2 has replaced pam_tally. Pam_tally still works, and if pam_tally alone is used the deny=5 still must be on the auth line and not the account line. Pam_tally2 is recommended in the current versions of the NSA and CIS RHEL hardening guides.
Some Internet sources suggest that you add the magic_root statement so that the root account won’t become locked if a user enters an incorrect sudo password. I have not found this to be true in testing. If a user enters an incorrect password for su, that may be true, but no one should be su’ing to root directly anyway, and if they are and enter the incorrect password, that seems OK to me that root would be locked. If you do have magic_root, some sources suggest that you also then need to add lines in /etc/pam.d/sshd; I have not tried that.
Pam_tally2 has the following improvements/changes:
- GLOBAL: New “silent” and “no_log_info” options
- AUTH: even_deny_root_account -> even_deny_root
- AUTH: per_user deprecated
- AUTH: New root_unlock_time and serialize option
- ACCOUNT: no_reset deprecated
- “faillog” no longer works; use pam_tally2 to list locked users or
reset users.
The correct setting is present in the NSA RHEL Hardening guide and newest CIS Benchmark for RHEL 5. The NSA RHEL Hardening guides notes that the behavior of pam_tally has changed during the lifetime of RHEL, and that the new, corrected settings may not work on systems that are not up-to-date.
NOTE: In RHEL 6, the syntax is different. This is shown in the CIS Benchmarks for RHEL 6, but I have not tested those settings.
Sources:
http://linux.die.net/man/8/pam_tally
http://man.he.net/man8/pam_tally2
http://manpages.ubuntu.com/manpages/hardy/man8/pam_tally.8.html
http://manpages.ubuntu.com/manpages/lucid/man8/pam_tally2.8.html
http://www.nsa.gov/ia/_files/os/redhat/NSA_RHEL_5_GUIDE_v4.2.pdf
https://benchmarks.cisecurity.org/tools2/linux/CIS_Redhat_Linux_5_Benchmark_v2.0.0.pdf