I have access logging configured for my AWS ALB. It dumps these logs into an S3 bucket on an interval.
To view them you have to download then unzip the file and look through the text.
I'd like to see a list of the ALB HTTP requests in one place without having to go through the process mentioned above.
Does AWS offer anything like this?
The AWS Application Load Balancer saves log files into Amazon S3.
Amazon Athena can then be used to query the files saved in S3. The important part is knowing the file format.
See this excellent article: Athena & ALB Log Analysis
They use this query to create the table:
CREATE EXTERNAL TABLE IF NOT EXISTS logs.web_alb (
type string,
time string,
elb string,
client_ip string,
client_port string,
target string,
request_processing_time int,
target_processing_time int,
response_processing_time int,
elb_status_code int,
target_status_code string,
received_bytes int,
sent_bytes int,
request_verb string,
request_url string,
request_proto string,
user_agent string,
ssl_cipher string,
ssl_protocol string,
target_group_arn string,
trace_id string
)
PARTITIONED BY(year string, month string, day string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1',
'input.regex' = '([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*) ([-0-9]*) ([-0-9]*) ([-0-9]*) ([-0-9]*) ([^ ]*) ([-0-9]*) ([-0-9]*) \"([^ ]*) ([^ ]*) ([^ ]*)\" \"([^\"]*)\" ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*)'
) LOCATION 's3://{{BUCKET}}/AWSLogs/{{ACCOUNT}}/elasticloadbalancing/us-east-1/';
AWS puts those log files in S3.. That cannot be changed and I don't see any other better place AWS could put those logs also. Now, what matters is how you want to process those logs from S3 ! What is your requirement there !
Few options:
As mentioned by John; you can use Athena to directly query on this S3 bucket. It is something like you have these logs on your local filesystem and you running grep on it.
If you need to aggregate this data; create some reports/dashboards --> use EMR on these log files.
If you need to really view these files; you can always set up a cronjob on one of the server which runs every hour which does exactly what you do manually above.. This will make sure that you have log files ALWAYS ready for consumption.
We can even put these log files into CloudWatch logs, Kinesis stream.
It all depends on your requirement..
