0

I am programatically downloading aws bill from s3 bucket,parsing all line items and processes them. I am doing this process daily by remembering a marker based on timestamp of last lineitem in the bill. So, in the next run, I will process only the lineitems after my marker. But Aws is updating cost values by adding more line items back in time which I am ignoring in current approach. I can't process all lineitems as sometimes the files can be huge. Does aws mention this problem anywhere ? I couldn't find any documentation.

It would be helpful if anyone can give a better approach to resolve this.

Manohar
  • 127
  • 12

2 Answers2

0

Yes, AWS may update previous line items in its billing file. AWS keeps updating the line items in a file till 72 hours of its creation. The best way to parse the data is to parse the whole month of data at a time. In that case, if the day of month is after 3rd, you just need to process current month data.

upaang saxena
  • 779
  • 1
  • 6
  • 18
  • Do you have a citation from the documentation where the 72 hour window is mentioned? – Michael - sqlbot May 16 '17 at 00:31
  • No, but this is my observation. AWS keeps updating billing file for previous days as for few instances, data is not updated. It will update them and will give them a completely new ID, so your marker logic will definitely fail. – upaang saxena May 16 '17 at 05:27
  • Note, I'm not the OP, so it isn't "my" marker logic. If reprocessing is required, it would be good to know what is *really* needed. 72 hours sounds like a very official number, and if it isn't, then I would suggest that your answer needs to indicate that this is only your anecdotal observation from practical experience. – Michael - sqlbot May 16 '17 at 11:33
  • I have seen in some cases AWS updated bill 5 days back in time. Processing whole month bill every time is expensive for me, that's why I opted marker logic. – Manohar May 17 '17 at 13:34
0

It does indeed, as per their billing documentation. Two options for you-- track the assembly ID to determine whether there's been a change, or else turn on versioning in that bucket and validate whether or not there's a new/old version for the object in question since the last time you ran.

Corey Quinn
  • 66
  • 8
  • Versioning or tracking the assembly Id has it's disadvantages. It will update the vesions and make changes for adding new lineitems also. So how do we track changes only back in time? – Manohar May 17 '17 at 13:37