0

I have this code:

import pantab
import pandas as pd
import datetime
df = pd.DataFrame([
    [datetime.date(2018,2,20), 4],
    [datetime.date(2018,2,20), 4],
], columns=["date", "num_of_legs"])

pantab.frame_to_hyper(df, "example.hyper", table="animals")

which causes this error:

TypeError: Invalid value "datetime.date(2018, 2, 20)" found (row 0 column 0)

Is there a fix?

Henrietta Martingale
  • 771
  • 8
  • 27

1 Answers1

1

This has apparently been a problem since time 2020. You have to know which columns are datetime columns because panda dtypes treat dates and strings as objects. There's a world where data scientists don't care about dates, but not mine, apparently. Anyway, here's the solution awaiting the day when pandas makes the date dtype:

[https://github.com/innobi/pantab/issues/100][1]

And just to reduce it down to do what I did:

def createHyper(xlsx, filename, tablename):
    for name in xlsx.columns:
        if 'date' in name.lower():
            xlsx[name] = pd.to_datetime(xlsx[name],errors='coerce',utc=True)
    pantab.frame_to_hyper(xlsx,filename,table=tablename)
    return open(filename, 'rb')

errors = 'coerce' makes it so you can have dates like 1/1/3000 which is handy in a world of scd2 tables utc = True was necessary for me because my dates were timezone sensitive, yours may not be.

I screwed up the hyperlink, It's not working. Damn it. Thank you to the anonymous editor who will inevitable show up and fix it. I'm very sorry.

Henrietta Martingale
  • 771
  • 8
  • 27
  • Author of pantab here. For a lot of pandas users, you could just do `df["date"] = pd.to_datetime(df["date"])` to convert that series to a datetime. pandas does not offer native date support, instead using nanosecond-precision based numpy arrays for date backing. With that said, you may also be interested in the hyperarrow project I just released. This is similar to pantab, but instead operates on PyArrow tables which offer native date support – Will Ayd Feb 11 '22 at 04:22