0

I am working with Azure functions to create triggers that aggregate data on an hourly basis. The triggers get data from blob storage, and to avoid aggregating the same data twice I want to add a condition that lets me only process blobs that were modified the last hour.

i am using the SDK, and my code for doing this looks like this:

''' Timestamp variables for t.now and t-1 '''
timestamp = datetime.now(tz=utc)
timestamp_negative1hr = timestamp+timedelta(hours=1)

''' Read data from input enivonment '''

data = BlockBlobService(account_name='accname', account_key='key')
generator = data.list_blobs('directory')

dataloaded = []
for blob in generator:
    loader = data.get_blob_to_text('collection',blob.name, if_modified_since=timestamp_negative1hr)
    trackerstatusobjects = loader.content.split('\n')
    for trackerstatusobject in trackerstatusobjects:
        dataloaded.append(json.loads(trackerstatusobject))

When I run this, the error I get is azure.common.AzureHttpError: The condition specified using HTTP conditional header(s) is not met. It is also specified that it is due to a timeout. The blobs are recieving data when I run it, so in any case it is not the correct return message. If I add .strftime("%Y-%m-%d %H:%M:%S:%z")to the end of my timestamp i get another error AttributeError: 'str' object has no attribute 'tzinfo'. This must mean that azure expects a datetime object, but for some reason it is not working for me.

Any ideas on how to solve it? Thanks

Nord112
  • 23
  • 3
  • 8
  • Can you try by using the following format: `.strftime("%Y-%m-%dT %H:%M:%SZ")`? – Gaurav Mantri Dec 01 '17 at 15:19
  • Thanks for suggestion, but I still get the same error `AttributeError: 'str' object has no attribute 'tzinfo'`. I suspect this is because Azure expects a datetime object, not a string. Maybe its something with the datet time format? – Nord112 Dec 01 '17 at 15:27
  • The error you are getting: azure.common.AzureHttpError: The condition specified using HTTP conditional header(s) is not met , means that if you blob storage's content did not change since your browser has accessed the resources, then it throws a 304 HTTP code. This thread has a similar problem: https://stackoverflow.com/questions/6112520/304-the-condition-specified-using-http-conditional-headers-is-not-met – Adam Smith - Microsoft Azure Dec 01 '17 at 21:45
  • But even after refreshing the browser while container is writing to new blobs the problem remains, are you sure its the same problem as the thread you are refering to? Thanks for the reply btw. – Nord112 Dec 01 '17 at 21:50
  • The best way to make sure it's the same issue is to gather the HTTP header requests and checking them, the same as the suggested previous post. It will be the only way to confirm if it's the same exact issue. Doing a bit of research made me find similar issues related to that error. – Adam Smith - Microsoft Azure Dec 01 '17 at 22:31
  • How do I gather HTTP requests for a timer trigger? Sorry if it is a stupid question. I tried adding some code to check the last time modified manually, but the function times-out instead :/ – Nord112 Dec 04 '17 at 11:24
  • @Nord112 It's not a stupid question at all, There are several ways to capture HTTP traffic (Headers of your choice), you can use either: Network Watcher - Scenario-based monitoring: https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-packet-capture-overview, and for automation based on HTTP headers or TCP(or whatever protocol) , you should checkout this link: https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-alert-triggered-packet-capture Iet me know if I should provide more details in the answer section. -Adam – Adam Smith - Microsoft Azure Dec 04 '17 at 21:40
  • Thanks for the links. Sadly my boss wont let me create a network watcher, so I need to find another way around it. If this is indeed the issue, is there a way to force the function to "re-access" the blob storage each time it is run to ensure this problem does not occur? – Nord112 Dec 05 '17 at 09:53

0 Answers0