0

How to pull data from AWS Security hub automatically using a scheduler ? I am new to AWS on doing some analysis I found below :

  1. In Security Hub data is in Json format , we don't have option to do Export to csv/excel ?
  2. All Security hub findings/insights are automatically sent to eventbridge ? Is it true ? If yes where i can check the same in eventbridge ?

Are there any other options in order to pull data from security hub , every 12 hours automatically. I want to take the data from security hub and pass it to the ETL Process in order to apply some logic on this data ?

Is Eventbridge the only and best approach for this ?

Bokambo
  • 4,204
  • 27
  • 79
  • 130

1 Answers1

2

On:

  1. It is a JSON based but it's their own format named AWS Security Finding Format (ASFF)
  2. It is true (for all resources that SecurityHub supports and is able to see). It should be noted that Each Security Hub Findings - Imported event contains a single finding. In order to see those events you'll need to create an EventBridge rule based on the format for each type of event.

Once you have that set up, the event could trigger an automatic action like:

  • Invoking an AWS Lambda function
  • Invoking the Amazon EC2 run command
  • Relaying the event to Amazon Kinesis Data Streams
  • Activating an AWS Step Functions state machine
  • Notifying an Amazon SNS topic or an Amazon SQS queue
  • Sending a finding to a third-party ticketing, chat, SIEM, or incident response and management tool.

In general, EventBridge is the way forward, but rather than using a scheduled based approach you'll need to resort to an event-based one. In order to intercept all findings, instead of rule being triggered by just specific one, you'll need to adjust the filter and essentially create a catch-all rule for SecurityHub which will then trigger your ETL job.

EDIT (as requested in comment):

The filter in the rule would look like this:

{
  "source": [
    "aws.securityhub"
  ]
}

with regard to the ETL, it really depends on your use case, having Kinesis Data Firehose dumping it to S3 and then using Athena as you suggest on your own would work. Another common approach is to send the data to ElasticSearch (or now OpenSearch). This blog post described them both, you can adjust it based on your needs.

EDIT 2:

Based on the discussion in the comments section if you really want to use a cron based approach you'll need to use the SDK based on your preferred language and create something around the GetFindings API that will poll for data from SecurityHub. You can use this function in Python, which extracts data from SecurityHub to Azure Sentinel as an example

Nick
  • 1,203
  • 5
  • 8
  • Thank you. Can you throw more light on this - create a catch-all rule for SecurityHub which will then trigger your ETL job ? Any examples ? And what do you suggest for ETL job ? Should i save this data first in S3 bucket and use AWS Athena to query this data as i need aggregate this data with another table before dumping into final S3 bucket for dashboarding. – Bokambo Sep 28 '21 at 04:10
  • I have updated my answer with an example filter for the rule and another link – Nick Sep 28 '21 at 04:43
  • { "source": [ "aws.securityhub" ] } This will send all the findings and insights from security hub to event bridge ? If i understand correctly this is more of a event driven architecture approach , if there is findings/insights in securityhub every second , eventbridge will have that data which might be costly approach in terms of cost/resources. My requirement is to do every 12 hours pull the data , is it not possible with schedule approach with event bridge ? – Bokambo Sep 28 '21 at 05:43
  • As you have pointed out in the question they are sent to EventBridge either way. As other services are sending information to it, with that filter you are basically filtering "everything that comes from SecurityHub" and then you can perform transformation of the data. This is the native approach. Of course in AWS everything is possible, you can use a scheduler and create a lambda around the https://docs.aws.amazon.com/securityhub/1.0/APIReference/API_GetFindings.html and poll for data instead but you'll need to work your way out on getting everything based on the API limits (MaxResults=100) – Nick Sep 28 '21 at 06:52
  • I have made another update to my answer, with a link to a python function which you can use as an example. – Nick Sep 28 '21 at 07:16