1

I've got a bit of a puzzler - I am an administrator on both a Windows 10 and a Mac Catalina machine.

The Windows machine is currently running software that has a known bug - SSH commands are sent to a Mac by a large piece of .NET Software that is difficult to RE (due more to its size than complexity) over SSH. The 2 PCs are on the same network.

The SSH commands fail, but I don't know what they are. I would like to record these actions, or at least view them in some way, so I can change the behavior of the receiving end (Mac) to take the correct action (bug workaround).

Here is what I've tried to view the SSH commands:

  • Run Wireshark (on both sides), try to decode the Diffie-Hellman exchange via SSH keys (I don't know how this works exactly, so I'm fumbling my way through - e.g. I don't know how to use RSA keys to decode the conversation)
  • Try Strace - doesn't exist on Mac machines. Alternative option dtruss may work but I don't know the commands and have to bypass SIP.
  • Try to log SSH commands by using .bashrc auto-logging - this has now been replaced by .zprofile and .zshrc - this doesn't appear to work, and no history is left in z_history or bash_history. It works when I run interactive Terminal commands though. This may be due to commands being chained to the initial SSH command, e.g. ssh user@host SOMECOMMAND.
  • Run ProcMon, see if the commands are in some secret config file. Can't seem to find it.
  • Decompile using Reflector and dnSpy - I find some logic on how to transmit SSH commands, but no hard-coded actual commands.

What am I missing? Is there some obvious way to do this that I've overlooked?

I am relatively versed in making things work, so if I have an idea of the best path forward, then I'll focus my attention there. There just seem to be too many options in a large space to know how to proceed..

I've read there are maybe ways to do this via the sshd_config on the SSH host (Mac), but I have no idea which settings would actually work, and apparently Mac does things in a special way.

[Please feel free to move the appropriate StackExchange site, not sure which is best for this]

Coruscate5
  • 53
  • 9
  • As a minor update - I've tried to use pam_tty_audit.so to record commands in the **sshd** config file `/etc/pam.d/sshd`, but this lib crashes SSH – Coruscate5 Mar 30 '20 at 21:38
  • You say this is a known bug, have you thought to contact the developer's bug list to find out the commands they are trying to execute? The other option would be to look at the MAC end of the equation rather than trying to get the commands before they are executed, see what they are on the client side. Is the SSH session created(last)? anything logged in /var/log/dmsg? if you are not seeing a session created, then you are having a more basic problem of how the connection is being set up. Can you SSH to to the mac from the PC? – Rowan Hawkins Mar 30 '20 at 22:53
  • @RowanHawkins - SSH works fine - MSFT is aware of the problem, but I'm more curious how to do any sort of SSH recording. There doesn't appear to be a /var/log/dmsg on my Mac, unless I'm missing something. Further, if I run a standard SSH session (without chaining commands) I can record commands using 2 tricks, `script` and `expect`, which come from other techniques. But I still get nothing from the MSFT SSH session. – Coruscate5 Mar 31 '20 at 01:22
  • Honestly, at this point I'd bet that DTRUSS is my best bet, but every time I turn it on it gives me nearly unintelligible raw system logs - I can't seem to find anything that resembles actual commands in there. – Coruscate5 Mar 31 '20 at 01:24
  • Consider this URL suggestion, please. https://en.wikipedia.org/wiki/Tee_(command) With your MAC version, you may have something similar. With my Windows 10, TEE is available. – Wilson Hauck Apr 06 '20 at 12:28

2 Answers2

1

This looks like a typical case for auditing. I've no experience configuring it on Mac OS, therefore the references to some documentation and examples:

https://opensource.apple.com/source/OpenBSM/OpenBSM-21/openbsm/man/

https://krypted.com/mac-os-x/quick-dirty-openbsm-auditing-macos/

It looks like a type of auditd is also available:

https://opensource.apple.com/source/system_cmds/system_cmds-336.10/auditd.tproj/auditd.8.auto.html

Enabling auditing, and parsing logs should work and contain all commands that had been executed on the destination system.

E.g knowing the ssh user you can start with audit_user and configure ex syscalls to be logged for this specific user. In case the user is changed during the session it might not capture all, but at least up to the user switch command.

/etc/security/audit_user yoursshuser:ad,ex:no

For a quick start try something like auditreduce -u <userid> -c ex /dev/auditpipe | praudit

This should provide live logs from the users exec syscalls.

Suggest to forget any approaches through shell histories and the like, as you don't know if a shell is used. E.g. a script is copied over and shell is just switched. Interesting are ex syscalls, these contain all executions, no matter how these are triggered.

EDIT FOR MAC: Specifically for Mac (with System Integrity Protection [SIP] on, e.g. Catalina/El Capitan/etc), OpenBSM will not record bash commands that don't result in actual execution of a program.

If you would like to record incorrect commands (e.g. ASDFASDFASDF), then the best option appears to be using dtruss - dtrace may also work, but the difficulty in getting it to work properly appears to be higher.

First, disable SIP, then run the following to record all system calls for bash processes: 

sudo dtruss -n bash

Incorrect commands will be "searched for" in various environment locations, in the stat64 calls. E.g. stat64("/usr/bin/ASDFASDFASDF",...)

Coruscate5
  • 53
  • 9
hargut
  • 3,908
  • 7
  • 10
  • I spent a good 3-4 hours working on PAM to get auditd to do what it can do in Linux to no avail, but I'll take a look at the sources you provided to see if I missed something, thanks – Coruscate5 Apr 03 '20 at 19:04
  • I'd also mention that the `pam_audit_tty.so` lib either doesn't exist on Mac or is non-functional, because applying it crashes all SSH sessions – Coruscate5 Apr 03 '20 at 19:07
  • In my opinion the suggested solution should be unrelated to `pam` and `sshd`. Also I do think that the `pam_tty_audit` module was not made for sshd. – hargut Apr 03 '20 at 19:10
  • After a quick look it seems that `pam_tty_audit` is only available for Linux. It was made for recording keystrokes on tty consoles, these are special devices, typically used for login at the hardware. Therefore not suited for sshd command recording, and most likely failing on OSX because the library cannot be found. The available pam modules should be located at `/usr/lib/pam`. – hargut Apr 03 '20 at 19:23
  • Makes sense - and it does look like the auditing from openbsm is the only new option I'm seeing. Catalina did a security overhaul, so I'll have to start with the `filewatcher` utility to see if that works. If it *does*, then this would represent the most likely course of action that will work - I'm OK with system logs as long as I can interpret them somehow (e.g. from the `ex` calls) – Coruscate5 Apr 03 '20 at 19:27
  • It is surprising to me that this is entirely new art though - I was hoping not to be coding my own audit solution. – Coruscate5 Apr 03 '20 at 19:28
  • Please be aware that OSX underlying structures are not compareable to Linux. Some elements are working similar, but this area of the os `openbsm` has not much in common with Linux, it seems to be more from the Solaris side. – hargut Apr 03 '20 at 19:30
  • No coding should be needed to achive what you need, even though one of the provided links has examples in code in it. All you need should be available on the system, as it is shipped with the os. Just some config files or commands should work out. – hargut Apr 03 '20 at 19:33
  • I get `Couldn't open dev/auditpipe` from your quick command - is that SIP likely blocking me from writing to the location? This is while in a sudo shell – Coruscate5 Apr 03 '20 at 19:33
  • Looks like a missing `/` at `/dev`. – hargut Apr 03 '20 at 19:36
  • Removed the link with the code examples as it provides ways too much, and unnecessary detail for this question. SIP looks to be only DTrace related, it should not interfer with the auditpipe. – hargut Apr 03 '20 at 19:44
  • Sorry, the command is correct - I used `/dev/auditpipe` just missed a slash in the comment. Same error, but I'll try bypassing SIP next (I'll likely have to work on this after business hrs today :) ) – Coruscate5 Apr 03 '20 at 20:07
  • Is the device there? Which permisdions are on it? How does `ls -l /dev/audit*` give? – hargut Apr 03 '20 at 20:11
  • truncated output of `ls -la` in `/dev/` : `crw------- ... auditpipe` and `crw-r--r-- ... auditsessions` - I *suspect* SIP, but I can't unmount the root drive until a few hrs from now – Coruscate5 Apr 03 '20 at 20:35
  • OK - yes, OpenBSM works. There are a lot of things I had to do to make this work on Catalina - primarily, SIP mucks with the user's ability to do anything. Regardless of configuration, I can't get anything to print to `/dev/auditpipe` but there is an audit trail in `/var/audit/[DATETIME].not_terminated` which I guess is Apple's way of tracking. I will update the answer once I have repeatable steps – Coruscate5 Apr 07 '20 at 00:03
  • Thank you for the feedback & good luck in getting your issue solved. :-) – hargut Apr 07 '20 at 16:38
  • Just out of curiosity, I was unable to get "incorrect" (non-existent commands, which were the core of the original issue) to print to EX calls because I guess the system doesn't actually execute them. I'm guessing that perhaps if I go into `audit_event`, then I might be able to control "failed and successful" events for incorrect commands.. any thoughts? – Coruscate5 Apr 07 '20 at 17:01
  • 1
    Depends what "incorrect" means in this context, typically if the program is called e.g. with incrorect flags, the exec call should be listed, even if the program later on stops/fails e.g. due to argument issues. Another thing would be that a program to be called cannot be found, e.g. file not found. This should be traceable. – hargut Apr 07 '20 at 17:34
  • incorrect flags are traced - what is missing is a call that cannot be found. For example. in an ssh session if I issue a command like `asdfasdfasdf`, I cannot seem to grep this string anywhere in the audit trail, even with `flags=all;naflags=all` - have you been able to do this in the past? Funny enough, the grep returns only the grep command itself :) (e.g. EXEC grep args asdfasdfasdf). I'm confident that my config isn't getting rolled either, I set the values extremely high and am watching file sizes (400M for single audit file, 3000M for total) – Coruscate5 Apr 07 '20 at 17:39
  • As far as I can see from the docs the non-successful flags are prefixed with a "-". See example in https://www.freebsd.org/cgi/man.cgi?audit_user for user jdoe where "-fc" is described to trace failed file creates. To trace "command-not-found" issues I do think that "-fr" or "-fa" could be the appropriate flags. Tracing failed file-read or "file-attribute access" actions. – hargut Apr 07 '20 at 17:40
  • OK - so, I think this is a bash issue. I can now record live commands at `/dev/auditpipe`, but, `bash` doesn't actually issue a search command for an object it can't find in it's own cache, e.g. `asdfasdfasdf`. `bash` will return `0xFFFFFFFF` as an error, but I think it never actually checks the filesystem, but rather a cache of what is knows exists already at each location in the `PATH` env var. The only way I could capture the incorrect command itself is by manipulating bash in some way unfortunately. It already should `tee` to history, but doesn't work for ssh piped commands – Coruscate5 Apr 07 '20 at 19:39
  • Just tried to get a grip on that, it looks like bash issues under a Linux OS a series of `stat` syscalls before it reports the "command not found" message. So tracing `stat` could be a way to go. According to https://www.freebsd.org/doc/handbook/audit-config.html the `fa` flag `file attribute access Audit the access of object attributes such as stat(1) and pathconf(2).` monitors these calls. So `-fa` should show what you're looking for. To validate on OSX you can try to run `strace bash` and then type a command like `asdf` and `enter` and you can see the calls which bash issues. – hargut Apr 07 '20 at 20:08
  • No `strace` in OSX :) . There is a variant of dtrace called `dtruss` but, like openBSM, seems underdocumented (or maybe just unpopular). I would love to try what you just commented though.. even with `stat`, it should have been recorded with `flags=all`, which for whatever reason just isn't :/ . Do you happen to know enough about `dtrace` or `dtruss` to be able to write me a quick analog of `strace bash`? – Coruscate5 Apr 07 '20 at 20:12
  • Hey! I got it. Dtruss not so bad - `sudo dtruss -n bash` got me listening to all the `bash` sessions (there were many for some reason). You were right, it only checks to see if stat64 returns something it already knows about - so, I end up seeing `stat64("EACH\PATH\IN\ENV\asdfasdfasdf")`. That's finally all the way there! – Coruscate5 Apr 08 '20 at 02:19
0

If you have access to the shell where the commands are executed you could try to redirect the file descriptors in the shell's profile to a logfile. Or turn on history. Or use sysdig.

Spacerat
  • 34
  • 2