Xively read data in Python

Question

I have written a python 2.7 script to retrieve all my historical data from Xively.

Originally I wrote it in C#, and it works perfectly.

I am limiting the request to 6 hour blocks, to retrieve all stored data.

My version in Python is as follows:

requestString = 'http://api.xively.com/v2/feeds/41189/datastreams/0001.csv?key=YcfzZVxtXxxxxxxxxxxORnVu_dMQ&start=' + requestDate + '&duration=6hours&interval=0&per_page=1000' response = urllib2.urlopen(requestString).read()

The request date is in the correct format, I compared the full c# requestString version and the python one.

Using the above request, I only get 101 lines of data, which equates to a few minutes of results.

My suspicion is that it is the .read() function, it returns about 34k of characters which is far less than the c# version. I tried adding 100000 as an argument to the ad function, but no change in result.

Carlos Rufo · Answer 1 · 2014-12-22T19:10:56.047

Left another solution wrote in Python 2.7 too.

In my case, got data each 30 minutes because many sensors sent values every minute and Xively API has limited half hour of data to this sent frequency.

It's general module:

for day in datespan(start_datetime, end_datetime, deltatime): # loop increasing deltatime to star_datetime until finish
    while(True): # assurance correct retrieval data 
        try: 
            response = urllib2.urlopen('https://api.xively.com/v2/feeds/'+str(feed)+'.csv?key='+apikey_xively+'&start='+ day.strftime("%Y-%m-%dT%H:%M:%SZ")+'&interval='+str(interval)+'&duration='+duration) # get data
            break
        except:
            time.sleep(0.3)
            raise # try again
    cr = csv.reader(response) # return data in columns
    print '.'
    for row in cr:
        if row[0] in id: # choose desired data
            f.write(row[0]+","+row[1]+","+row[2]+"\n") # write "id,timestamp,value"

The full script you can find it here: https://github.com/CarlosRufo/scripts/blob/master/python/retrievalDataXively.py

Hope you might help, delighted to answer any questions :)

Xively read data in Python

1 Answers1