2

While tailing /var/log/auth.log I noticed that there where multiple entries being entered (instantly) by the minute for user "foo". I personally had only one connection open as user "root_bar" while tailing the auth.log (log sample below). As you can see, there is no IP information for this incoming SSH connections. What is the best way to trace the IP address for incoming SSH connections?

Aug 10 14:30:04 ps2000 suexec: (pam_unix) session opened for user root_bar by (uid=999)
Aug 10 14:30:04 ps2000 suexec: (pam_unix) session closed for user root_bar
Aug 10 14:30:06 ps2000 suexec: (pam_unix) session opened for user root_bar by (uid=999)
Aug 10 14:30:06 ps2000 suexec: (pam_unix) session closed for user root_bar
Aug 10 14:30:08 ps2000 CRON[16879]: (pam_unix) session closed for user root_bar
Aug 10 14:30:14 ps2000 suexec: (pam_unix) session opened for user root_bar by (uid=999)
Aug 10 14:30:14 ps2000 suexec: (pam_unix) session closed for user root_bar
Aug 10 14:30:16 ps2000 suexec: (pam_unix) session opened for user root_bar by (uid=999)
Aug 10 14:30:16 ps2000 suexec: (pam_unix) session closed for user root_bar
Aug 10 14:30:27 ps2000 suexec: (pam_unix) session opened for user root_bar by (uid=999)
Aug 10 14:30:27 ps2000 suexec: (pam_unix) session closed for user root_bar
Aug 10 14:30:39 ps2000 suexec: (pam_unix) session opened for user root_bar by (uid=999)
Aug 10 14:30:39 ps2000 suexec: (pam_unix) session closed for user root_bar

Disclaimer: servername, and all user information has been changed for security reasons.

Correction: The question "Tracing incoming SSH connections" has been properly answered by the posters below. The message suexec (pam_unix) session does not necessarily indicate any sshd activity as clarified by @aseq, and I posted this as a sshd question due to my ignorance. Since the original question, and it's answers are helpful, I'm accepting the most helpful answer. I think tracing suexec: (pam_unix) session is the candidate for a separate question.

Final update: I discovered that the messages above did have to do with sshd. After doing some tweaks in /etc/pam.d/common-auth I started seeing lines such as

Aug 10 16:45:23 candy_bass sshd[427]: (pam_unix) session opened for user summer_flag by (uid=0)
Aug 10 16:45:23 candy_bass sshd[427]: PAM pam_parse: expecting return value; [...sucess=1 default=ignore]
Aug 10 16:45:23 candy_bass sshd[427]: PAM pam_parse: expecting return value; [...sucess=1 default=ignore]
Aug 10 16:45:23 candy_bass sshd[427]: Accepted publickey for summer_flag from xxx.zzz.yyy.abc port 35964 ssh2
Aug 10 16:45:23 candy_bass sshd[427]: (pam_unix) session opened for user summer_flag by (uid=0)
Aug 10 16:45:23 candy_bass pam_limits[427]: setrlimit limit #11 to soft=-1, hard=-1 failed: Operation not permitted; uid=0 euid=0
Aug 10 16:45:23 candy_bass pam_limits[427]: setrlimit limit #12 to soft=-1, hard=-1 failed: Operation not permitted; uid=0 euid=0
Aug 10 16:45:23 candy_bass sshd[427]: (pam_unix) session closed for user summer_flag

So this is related to sshd, however, since this is so specific to a token-auth vendor (whose name I am not disclosing for privacy), I think this might be better solved by the vendor.

amateur barista
  • 498
  • 3
  • 7
  • 21
  • 1
    Check `/var/log/messages` and/or `/var/log/secure`. You should be able to get more info there. – tacotuesday Aug 10 '12 at 22:08
  • The logs above show no activity from the ssh server. Please add relevant log entries from the ssh server. – aseq Aug 10 '12 at 22:13
  • There's no mention of the user `foo` in that log sample. None of those sessions were opened by ssh either. They were all `suexec` or `cron`. – Ladadadada Aug 10 '12 at 22:14
  • @aseq Please mention to the relevant log entries that you want to see. – amateur barista Aug 10 '12 at 22:19
  • So `foo` is a real user on your system but you changed it to `root_bar` in the log sample so we wouldn't know you had a `foo` user? – Ladadadada Aug 10 '12 at 22:22
  • @amateur barista -- this is because `auth.log` is logging every suexec transaction. Everytime a connection is made via SSH, regardless of whether or not someone even tries to login, it is logging this information in your `auth.log`. Each log file has a separate and distinct function. They're not meant to necessarily correlate. – tacotuesday Aug 10 '12 at 22:23
  • 1
    The log entries for "multiple SSH sessions being opened and closed (instantly) by the minute for user foo". The log entries you pasted do not show any ssh server activity. – aseq Aug 10 '12 at 22:25
  • @amateurbarista We need to also know the Linux distribution you're using. Different distros log SSH connections in different places. However, they can still be modified by a hosting provider to log somewhere entirely different than the norm for the distro, which I've experienced several times. – tacotuesday Aug 10 '12 at 22:32
  • @nojak The question is tagged as debian-etch =) – amateur barista Aug 10 '12 at 22:32
  • @amateurbarista Doh! Sorry... I thought I saw it somewhere... Long day, I suppose. I know Debian generally stores SSH logins in `auth.log`, so I assumed it was Debian. – tacotuesday Aug 10 '12 at 22:35
  • @auth.log, you got it right on the log name, I one-upped your answer =) – amateur barista Aug 10 '12 at 22:37
  • @Ladadadada I edited the question to clarify that the sensitive information has been altered. – amateur barista Aug 10 '12 at 22:40

4 Answers4

3

How do those log entries look like?

The ssh server should log the IP addresses by default in /var/log/auth.log and other log files, such as:

Aug 1 12:21:30 example.host sshd[1174]: Failed password for invalid user example from 192.0.2.1 port 9460 ssh2
Aug 1 12:21:32 example.host sshd[1176]: Invalid user root from 192.0.2.10

If the log entries you are asking about do not have the string "sshd" in it I doubt they actually came from the ssh server and you need to look elsewhere. Look ate the string that comes after the hostname, it tells you which program was writing the log.

You may also check /etc/ssh/sshd_config and see if the loglevel is correct, the default on squeeze is:

# Logging
SyslogFacility AUTH
LogLevel INFO

Perhaps increasing verbosity may reveal more information. The log entry you added to your question should be preceded by log entries as pasted above.

aseq
  • 4,610
  • 1
  • 24
  • 48
  • Interesting. @aseq, I posted the log info *as is*, with the exception of the username and servername modifications I did (for security). So that means that `suexec: (pam_unix)` would be executing this? This is starting to make sense, since I recently installed a pam module for token authentication on the server... – amateur barista Aug 10 '12 at 22:23
  • I'm actually post this question with the vendor of my token auth module, it could be, possibly, but I'm not sure related. – amateur barista Aug 10 '12 at 22:36
  • Let us know how you resolved it. Also see my updated answer. – aseq Aug 11 '12 at 00:10
  • 1
    doing ssh user@server.com with the -v option reveals that the user *is* actually logging in successfully to the server via sshd, however, the PAM module for token auth is the one that is blocking out the user. FYI, at this point probably the problem is bad configuration of the pam module. I've expanded the log samples in my question. – amateur barista Aug 11 '12 at 00:56
1

You will see connections open and close for SSH whenever a connection is made, regardless of whether someone successfully logged in or not.

To view more information on successful and failed login attempts via ssh, look at /var/log/secure and/or /var/log/messages.

***Note that the location may vary depending on your distribution of Linux and/or your hosting provider.*

tacotuesday
  • 1,389
  • 1
  • 16
  • 27
  • thanks for pointing me to /var/log. Messages does have useful information (along with IP addresses). However while I'm "tailing" messages and auth.log at the same time, I still see many opening and closing sessions on auth.log, but no activity on messages. Something is happening before a message even gets posted into messages. I only see stuff go into messages when I myself log in. – amateur barista Aug 10 '12 at 22:24
0

If you have access to the root user on the server, you could leverage iptables to implement a 'LOG' statement for all NEW connections on tcp port 22. This would add information to the /var/log/messages file indicating what connections are coming inbound to your server.

If you are running IPv6, the firewall for there would be iptables6 I believe.

Nick V
  • 145
  • 2
  • 2
  • 7
0

I resolved the flurry of opening and closing sessions on auth.log problem by commenting out the ChallengeResponseAuthentication and UsePAM settings on /etc/ssh/sshd_config from 

ChallengeResponseAuthentication yes
UsePAM yes

to 

#ChallengeResponseAuthentication yes
#UsePAM yes

In other words, I temporarily disabled the PAM module while I figure out with it's vendor the correct settings.

amateur barista
  • 498
  • 3
  • 7
  • 21