8

I want to store a list of datetimes in a binary file in Python.

EDIT: by "binary" I mean the best digital representation for each datatype. The application for this is to save GPS trackpoints composed by (unix-timestamp, latitude, longitude, elevation), so the whole structure is little-endian "Long, float, float, float", with four bytes to each value.

NOTE: I don't use "unix-timestamp" due to any affection to the Unix platform, but only as an unequivocal way to represent the value of a datetime.

Currently, I am doing like the code below, but besides some timezone confusion that I'm still working out (my timezone is -3), I believe converting to int and back might not be the right way, since datetime and datetime64 are native types in python/numpy, if I'm not mistaken. Thus, a datetime64 would need eight bytes instead of the four I am using for the (long)unix-timestamp.

import datetime
import calendar
import struct

now = datetime.datetime.now()
print now

stamp = calendar.timegm(now.utctimetuple())
print stamp

binarydatetime = struct.pack('<L', stamp)
recoverstamp = struct.unpack('<L', binarydatetime)[0]
print recoverstamp

recovernow = datetime.datetime.fromtimestamp(recoverstamp)
print recovernow

So the main question is: "is this the pythonic way to converting naive datetime to binary and back?"

And the aditional question is: "if everything in this code is supposed to be naive, why do I have a timezone offset?"

Thanks for reading!

heltonbiker
  • 26,657
  • 28
  • 137
  • 252
  • Define 'binary' here. What other software do you expect to interoperate with? There is no standard, per se, of storing datetime in binary formats, most software pick their own preferred format, which may or may not be based on the UNIX epoch. – Martijn Pieters Aug 29 '13 at 16:56
  • @MartijnPieters I would like the datatypes to be written to file in a language and platform independent way. See my edit! – heltonbiker Aug 29 '13 at 19:42
  • 1
    consider sqlite database as a portable binary format to store gps trackpoints using its native datetime type for utc (or gps) time. – jfs Sep 02 '13 at 07:24
  • Did you solve this problem? ("reading" timestamp from bytes). I have the same. – garrythehotdog Dec 31 '20 at 11:17
  • @garrythehotdog see my own answer below – heltonbiker Jan 05 '21 at 16:45

3 Answers3

6

I have found a way using the Unix timestamp and storing it as an integer. This works for me because I don't need a subsecond resolution, but I think long integers would allow for microsecond resolution with some modifications of the code.

The changes from my original consist in replacing calendar.timegm by time.mktime and also utctimetuple by timetuple, to keep everything naive.

This:

import datetime
import struct
import time

now = datetime.datetime.now()
print now

stamp = time.mktime(now.timetuple())
print stamp

recoverstamp = datetime.datetime.fromtimestamp(stamp)
print recoverstamp

binarydatetime = struct.pack('<L', stamp)
recoverbinstamp = struct.unpack('<L', binarydatetime)[0]
print recoverbinstamp

recovernow = datetime.datetime.fromtimestamp(recoverbinstamp)
print recovernow

Returns this:

2013-09-02 11:06:28.064000
1378130788.0
2013-09-02 11:06:28
1378130788
2013-09-02 11:06:28

From this, I can easily write the packed binarydatetime to file, and read it back later.

heltonbiker
  • 26,657
  • 28
  • 137
  • 252
  • For me microseconds are important what do you mean by using long integers ? – le_lemon Jun 17 '20 at 10:53
  • @le_lemon the unix timestamp is too large to be stored on a 32 bit signed integer, so in order to properly store it with microsecond resolution you would need a larger number format, such as uint, long, or even better ulong (unsigned 64 bit integer). Please check all this, perhaps what I just said is not totally correct, but these are the principles. – heltonbiker Jun 23 '20 at 20:54
  • @heltonbiker looks like `stamp` needs to be converted to integer before packing: `binarydatetime = struct.pack(' – Andrey Aug 17 '21 at 11:09
2

By far the most simple solution is to use Temporenc: http://temporenc.readthedocs.org/

It takes care of all the encoding/decoding and allows you to write a Python datetime object to a file:

now = datetime.datetime.now()
temporenc.pack(fp, now)

To read it back, this suffices:

dt = temporenc.unpack(fp)
print(dt)
wouter bolsterlee
  • 3,879
  • 22
  • 30
0

If you want to save an object to a binary file, you may think about pickle

As far as I know, it writes a byte stream, so it's binary enough, but the result is

Python-specific

Also, I think it should work with datetime.

And you'll have this for saving:

with open('your_file', 'wb') as file:
    pickle.Pickler(file).dump(your_var)

And this for recovering

with open('your_file', 'rb') as file:
     recovered=pickle.Unpickler(file).load()
Wjars
  • 99
  • 5
  • Actually I want a language- and platform- independent way to store the `datetime` data type. See my edit. – heltonbiker Aug 29 '13 at 19:49
  • Ah, I had the thought pickle won't do. See, I cannot figure out another way to do this, so for now it's surely good enough. ''Pythonic'' is a very abstract notion ( as is [PEP 20](http://www.python.org/dev/peps/pep-0020/) ). I only – Wjars Aug 29 '13 at 20:14
  • You can use str.encode('encodage') to have hexadecimal, utf-8 or whatever representation of your string, but it's a bit off topic. (the 5 minutes edit session is way too short) Cannot help you. – Wjars Aug 29 '13 at 20:20
  • Actually I am not storing a string representing a date and time, but actually a "native" `datetime.datetime` object: http://docs.python.org/2/library/datetime.html#datetime.datetime – heltonbiker Aug 29 '13 at 20:39