0

I have been running python scripts in windows power shell for quite some time.

All of a sudden this morning I get this error message:

 line 3802, in get_loc
    return self._engine.get_loc(casted_key)

  File "pandas\_libs\index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'last trade time'

what is strange is that I can run the code 3 different ways:

  1. Integrated Terminal - python test_blah.py (works fine).
  2. Windows power shell - navigate to folder, then python test_blah.py (works fine).
  3. Windows power shell - python "the full path/test_blah.py" (throws error).

the full path looks like this:

python "G:\My Drive\darren\python\test_blah.py"

So I have created a min reproducible example here to show the above:

import pandas as pd

print('pandas version:' , pd.__version__)

def some_function():
    df = pd.read_csv('basic_GBP_stats.csv')
    print(df)  
    print('got here 01')

    try:
        df['time'] = pd.to_datetime(df['last trade time'])
        print('got here 02')
    
    except Exception as e:
        print('exception found:', e)


some_function()

The contents of the (example) CSV file are this:

       Unnamed: 0  symbol  num trades  ...  wavg sell px  wavg ratio %      last trade time
0               0  ADAGBP         309  ...      0.261606      0.235289  2022-12-13 21:02:32
1               1  BTCGBP        3949  ...  14127.293571    -10.086920  2022-12-13 20:25:34
2               2  ETHGBP        1349  ...   1025.139765      0.183433  2022-12-13 20:18:14
3               3  SOLGBP        1467  ...     11.241261      0.193498  2022-12-13 17:42:31
4               4  XRPGBP        1005  ...      0.314160      0.333005  2022-12-13 22:13:14
...           ...     ...         ...  ...           ...           ...                  ...
15295           1  BTCGBP        6215  ...  16612.732328     -3.116873  2023-04-24 08:50:25
15296           2  ETHGBP        2075  ...   1139.706164      0.152275  2023-04-24 07:05:07
15297           3  SOLGBP        2366  ...     12.127936      2.285372  2023-04-23 15:48:48
15298           4  XRPGBP        1451  ...      0.315087      0.836744  2023-04-24 07:26:02
15299           5  BNBGBP         405  ...    219.697751      0.476855  2023-04-22 12:35:53

The version of pandas that is used: 1.5.3.

What I notice is that the df['time'] = pd.to_datetime(df['last trade time']) for some reason produces a malformed dataframe (from method 3 but works fine with methods 1 and 2), which is the reason for the error when using method 3 above.

But I don't know why or how to fix this...

D.L
  • 4,339
  • 5
  • 22
  • 45
  • 2
    Do you have different versions of "basic_GBP_stats.csv" at different places in your directory structure? (1) and (2) would read the file from the same directory that the python script is in, whereas (3) would read it from whatever your working directory was when you ran the command. – slothrop Apr 24 '23 at 10:44
  • 1. you have to check the separators in the CSV-file. If there are different from coma, you have to define it in the `read_csv()`. 2. try to define the format: `df['time'] = pd.to_datetime(df['last trade time'], format="%Y-%m-%d %H:%M:%S)` then you will see, of there is some problem with your data in the columns "last trade time". – coco18 Apr 24 '23 at 11:04
  • hi @slothrop. There is only one instance of the file. The only thing that I can think of is that somehow the space in the file path causes an error (although this was not a problem before) ? – D.L Apr 24 '23 at 11:21
  • @coco18, if this was the case, then the example would fail every time (for methods 1 and 2) and not just with method 3. This indicates that the (python) code is working, but how it is being called does not work (for some reason)... – D.L Apr 24 '23 at 11:24
  • @D.L thanks - now I actually don't understand how (3) reads a file at all! But if it couldn't find a file, it wouldn't even get as far as showing the error you see. – slothrop Apr 24 '23 at 11:25
  • @slothrop, Yes, exactly. that is the odd part.... i see that a file is being read into a Dataframe, but that the Dataframe is malformed. – D.L Apr 24 '23 at 11:26
  • @D.L perhaps try adding `print(os.getcwd())` so you can see the working directory? Since read_csv is using the relative path `basic_GBP_stats.csv`, it's going to be taking the file from the working directory. – slothrop Apr 24 '23 at 11:26
  • @slothrop, i think that this might be the case. the `getcwd` is different when using the full path... – D.L Apr 24 '23 at 11:29
  • @D.L right. I can only think to triple-check that there isn't a different version of "basic_GBP_stats.csv" lurking in that working directory... – slothrop Apr 24 '23 at 11:39
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/253298/discussion-between-d-l-and-slothrop). – D.L Apr 24 '23 at 11:52

0 Answers0