0

I want to test cudf but stuck with a first simple task of filtering by datetime. Code works perfect with pandas, but not with cudf.

import pandas as pd
#import cudf as pd 
import time 
import datetime 
import dateutil

if __name__ == "__main__":
    Zeit_start = datetime.datetime.now()
    AGdata_search = pd.read_csv("testdata.csv",parse_dates=['Datetime'],infer_datetime_format=True,cache_dates=False)
    AGdata_TEST = AGdata_search.loc[(AGdata_search['Datetime'] >= dateutil.parser.parse("2021-11-02 13:44:00+00:00"))] 
    AGdata_TEST.to_csv("output.csv", encoding='utf-8',index=False)

testdata.csv looks like

Datetime,Open,High,Low,Close,Adj Close,Volume 
2021-10-22 13:30:00+00:00,149.69,149.75,149.01,149.04,149.04,4032096.0 
2021-10-22 13:40:00+00:00,149.69,150.175,148.845,149.92,149.92,19671400.0
2021-10-22 13:50:00+00:00,149.975,150.18,149.5601,149.75,149.75,11911828.0 
...

With cudf throws "KeyError: 'Datetime'"

Environment (Win11 with wsl2, Ubuntu and a Docker container)
conda version : 4.10.3
python version : 3.8.10.final.0
virtual packages : __cuda=11.5=0
               __linux=5.10.60.1=0
               __glibc=2.27=0
               __unix=0=0
               __archspec=1=x86_64
  user-agent : conda/4.10.3 requests/2.25.1 CPython/3.8.10 Linux/5.10.60.1-microsoft-standard-WSL2 ubuntu/18.04.6 glibc/2.27
stoic_snyder
rapidsai/rapidsai:21.12-cuda11.0-runtime-ubuntu18.04-py3.7
CUDA_VER=11.0 DASK_XGBOOST_VER=0.2* RAPIDS_VER=21.12
Robert Crovella
  • 143,785
  • 11
  • 213
  • 257
dirk8b
  • 1
  • 1

1 Answers1

1

RAPIDS cuDF does not yet implicitly support UTC offset in this format. Therefore, you have to explicitly declare the datetime format codes before proceeding, as per our docs: https://docs.rapids.ai/api/cudf/stable/api_docs/api/cudf.to_datetime.html?highlight=datetime

following these codes: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior

import cudf as cudf 
import time 
import datetime 
import dateutil

if __name__ == "__main__":
    Zeit_start = datetime.datetime.now()
    AGdata_search = cudf.read_csv("testdata.csv")
    AGdata_search['Datetime'] = cudf.to_datetime(AGdata_search['Datetime'], format='%Y-%m-%d %H:%M:%S+%z') # this makes it work)
    AGdata_TEST = AGdata_search.loc[(AGdata_search['Datetime'] >= dateutil.parser.parse("2021-11-02 13:44:00+00:00"))]
    AGdata_TEST.to_csv("output.csv", encoding='utf-8',index=False)

where testdata.csv is

Datetime,Open,High,Low,Close,Adj Close,Volume 
2021-10-22 13:30:00+00:00,149.69,149.75,149.01,149.04,149.04,4032096.0 
2021-10-22 13:40:00+00:00,149.69,150.175,148.845,149.92,149.92,19671400.0
2021-11-22 13:50:00+00:00,149.975,150.18,149.5601,149.75,149.75,11911828.0

I adjusted your testdata.csv so that output.csv would return with some data in it, based on your search query

I believe that there is a cleaner way to do your operation, but I don't know your whole workflow, so this should get you on your way!

TaureanDyerNV
  • 1,208
  • 8
  • 9