0

I'm using osquery to monitor servers on my network. The following osquery.conf captures snapshots, every minute, of the processes communicating over the network ports and publishes that data to Kafka:

{
  "options": {
    "logger_kafka_brokers": "cp01.woolford.io:9092,cp02.woolford.io:9092,cp03.woolford.io:9092",
    "logger_kafka_topic": "base_topic",
    "logger_kafka_acks": "1"
  },
  "packs": {
    "system-snapshot": {
      "queries": {
        "processes_by_port": {
          "query": "select u.username, p.pid, p.name, pos.local_address, pos.local_port, pos.remote_address, pos.remote_port from processes p join users u on u.uid = p.uid join process_open_sockets pos on pos.pid=p.pid where pos.remote_port != '0'",
          "interval": 60,
          "snapshot": true
        }
      }
    }
  },
  "kafka_topics": {
    "process-port": [
      "pack_system-snapshot_processes_by_port"
    ]
  }
}

Here's an example of the output from the query:

osquery> select u.username, p.pid, p.name, pos.local_address, pos.local_port, pos.remote_address, pos.remote_port from processes p join users u on u.uid = p.uid join process_open_sockets pos on pos.pid=p.pid where pos.remote_port != '0';
+--------------------+-------+---------------+------------------+------------+------------------+-------------+
| username           | pid   | name          | local_address    | local_port | remote_address   | remote_port |
+--------------------+-------+---------------+------------------+------------+------------------+-------------+
| cp-kafka-connect   | 13646 | java          | 10.0.1.41        | 49018      | 10.0.1.41        | 9092        |
| cp-kafka-connect   | 13646 | java          | 10.0.1.41        | 49028      | 10.0.1.41        | 9092        |
| cp-kafka-connect   | 13646 | java          | 10.0.1.41        | 49026      | 10.0.1.41        | 9092        |
| cp-kafka-connect   | 13646 | java          | 10.0.1.41        | 50558      | 10.0.1.43        | 9092        |
| cp-kafka-connect   | 13646 | java          | 10.0.1.41        | 50554      | 10.0.1.43        | 9092        |
| cp-kafka-connect   | 13646 | java          | 10.0.1.41        | 49014      | 10.0.1.41        | 9092        |
| root               | 1505  | sssd_be       | 10.0.1.41        | 46436      | 10.0.1.89        | 389         |
...
| cp-ksql            | 1757  | java          | 10.0.1.41        | 56180      | 10.0.1.41        | 9092        |
| cp-ksql            | 1757  | java          | 10.0.1.41        | 53878      | 10.0.1.43        | 9092        |
| root               | 19684 | sshd          | 10.0.1.41        | 22         | 10.0.1.53        | 50238       |
| root               | 24082 | sshd          | 10.0.1.41        | 22         | 10.0.1.53        | 51233       |
| root               | 24107 | java          | 10.0.1.41        | 56052      | 10.0.1.41        | 9092        |
| root               | 24107 | java          | 10.0.1.41        | 56054      | 10.0.1.41        | 9092        |
| cp-schema-registry | 24694 | java          | 10.0.1.41        | 50742      | 10.0.1.31        | 2181        |
| cp-schema-registry | 24694 | java          | 10.0.1.41        | 47150      | 10.0.1.42        | 9093        |
| cp-schema-registry | 24694 | java          | 10.0.1.41        | 58068      | 10.0.1.41        | 9093        |
| cp-schema-registry | 24694 | java          | 10.0.1.41        | 47152      | 10.0.1.42        | 9093        |
| root               | 25782 | osqueryd      | 10.0.1.41        | 57700      | 10.0.1.43        | 9092        |
| root               | 25782 | osqueryd      | 10.0.1.41        | 56188      | 10.0.1.41        | 9092        |
+--------------------+-------+---------------+------------------+------------+------------------+-------------+

Instead of snapshots, I'd like osquery to capture differentials, i.e. to only publish the changes to Kafka.

I tried toggling the snapshot property from true to false. My expectation was that osquery would send the changes. For some reason, when I set "snapshot": false, no data is published to the process-port topic. Instead, all the data is routed to the catchall base_topic.

Can you see what I'm doing wrong?

Update:

I think I'm running into this bug: https://github.com/osquery/osquery/issues/5559

Here's a video walk-through: https://youtu.be/sPdlBBKgJmY

I filed a bug report, with steps to reproduce, in case it's not the same issue: https://github.com/osquery/osquery/issues/5890

Alex Woolford
  • 4,433
  • 11
  • 47
  • 80

1 Answers1

0

Given the context, I can't immediately tell what is causing the issue you are experiencing.

In order to debug this, I would first try using the filesystem logger plugin instead of (or in addition to) the Kafka logger.

Do you get results to the Kafka topic when the query is configured as a snapshot? If so, are you able to verify that the results are actually changing such that a diff should be generate when the query runs in differential mode?

Can you see results logged locally when you use --logger_plugin=filesystem,kafka?

Zach
  • 1,263
  • 11
  • 25
  • When the snapshot is true, the messages are routed to the expected topic (i.e. `process-port`). When the snapshot is false, all the messages are routed to the "catchall" `base_topic`. That seems like a bug to me. – Alex Woolford Oct 15 '19 at 12:59
  • That does sound like a bug. Please file an issue in the github.com/osquery/osquery repository. – Zach Sep 26 '20 at 22:27
  • Thanks, @Zach. This was fixed in the latest release. – Alex Woolford Sep 26 '20 at 23:00