0

I'd like to use AWS AccessLogs for processing website impressions using an existing batch oriented ETL pipeline that grabs last finished hour of impressions and do a lot of further transformations with them.

The problem with AccessLog though is that :

Note, however, that some or all log file entries for a time period can sometimes be delayed by up to 24 hours

So I would never know when all the logs for a particular hour are complete. I unfortunately cannot use any streaming solution, I need to use existing pipeline that grabs hourly batches of data.

So my question is, is there any way to be notified that all logs has been delivered to s3 for a particular hour?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
lisak
  • 21,611
  • 40
  • 152
  • 243
  • Are you referring to Amazon S3 [Server Access Logging](http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html) or Amazon CloudFront [Access Logs](http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html)? – John Rotenstein Aug 26 '17 at 02:12
  • @JohnRotenstein sorry for that, I'm talking about CloudFront AccessLogs being stored to s3 and then processed by me in an ETL pipeline. – lisak Aug 27 '17 at 07:36

1 Answers1

0

You have asked about S3, but your pull-quote is from the documentation for CloudFront.

Either way, though, it doesn't matter. This is just a caveat, saying that log delivery might sometimes be delayed, and that if it's delayed, this is not a bug -- it's a side effect of a massive, distributed system.

Both services operate an an incomprehensibly large scale, so periodically, things go wrong with small parts of the system, and eventually some stranded logs or backlogged logs may be found and delivered. Rarely, they can even arrive days or weeks later.

There is no event that signifies that all of the logs are finished, because there's no single point within such a system that is aware of this.

But here is the takeaway concept: the majority of logs will arrive within minutes, but this isn't guaranteed. Once you start running traffic and observing how the logging works, you'll see what I am referring to. Delayed logs are the exception, and you should be able to develop a sense, fairly rapidly, of how long you need to wait before processing the logs for a given wall clock hour. As long as you track what you processed, you can audit this against the bucket, later, to ensure that yout process is capturing a sufficient proportion of the logs.

Since the days before CloudFront had SNI support, I have been routing traffic to some of my S3 buckets using HAProxy in EC2 in the same region as the bucket. This gave me the ability to use custom hostnames, and SNI, but also gave me real-time logging of all the bucket traffic using HAProxy, which can stream copies of its logs to a log collector for real-time analysis over UDP, as well as writing it to syslog. There is no measurable difference in performance with this solution, and HAProxy runs extremely well on t2-class servers, so it is cost-effective. You do, of course, introduce more costs and more to maintain, but you can even deploy HAProxy between CloudFront and S3 as long as you are not using an origin access identity. One of my larger services does exactly this, a holdover from the days before Lambda@Edge.

Michael - sqlbot
  • 169,571
  • 25
  • 353
  • 427