-1

I'm trying to make a crypto trading bot in python, and i collect historical data (prices and stuff) with the library called ccxt, and store it in pandas dataframe. When i try to store more than 999 rows, i get the error shown below (KeyError: 999). Basically it won't let me store more than 999 rows. Is there any way to get more then 999 data in pandas dataframe?

The code is pretty large, but here are some snippets:

Data frame build:

exchange = ccxt.binance()
bars = exchange.fetch_ohlcv(pair_to_trade, timeframe=timeframe, limit=amount_of_timeframe)
df = pd.DataFrame(bars[:-1], columns=['timestamp', 'open', 'high', 'low', 'close', 'volume'])

Last_row_index:

 for i in range(1, amount_of_timeframe-1):  # where amount_of_timeframe > 1000
        last_row_index = i
        previous_row_index = last_row_index - 1

Data frame looks like this:

              timestamp     open     high  ...    upperband    lowerband  in_uptrend
0   2021-09-05 16:27:00  3913.51  3914.97  ...          NaN          NaN        True
1   2021-09-05 16:28:00  3914.04  3914.22  ...          NaN          NaN        True
2   2021-09-05 16:29:00  3910.01  3912.00  ...          NaN          NaN        True
3   2021-09-05 16:30:00  3911.84  3911.85  ...          NaN          NaN        True
...
995 2021-09-06 09:02:00  3956.10  3956.36  ...  3967.517857  3942.412143       False
996 2021-09-06 09:03:00  3954.32  3954.32  ...  3964.035000  3941.775000       False
997 2021-09-06 09:04:00  3951.52  3955.85  ...  3964.035000  3941.610714       False
998 2021-09-06 09:05:00  3955.58  3955.58  ...  3964.035000  3943.252857       False

Error:

File "D:/myfolders/supertrendStrategy/main.py", line 105, in testStrategy
    if not stdf['in_uptrend'][previous_row_index] and stdf['in_uptrend'][last_row_index]:
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\series.py", line 882, in __getitem__
    return self._get_value(key)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\series.py", line 989, in _get_value
    loc = self.index.get_loc(label)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexes\range.py", line 357, in get_loc
    raise KeyError(key) from err
KeyError: 999
chronovirus
  • 41
  • 5
  • 1
    The problem is that you are trying to access the index 999 when it doesn't exist in the dataframe. I don't see anywhere in your question where you are trying to add rows to the dataframe. – Mohammad Sep 06 '21 at 15:49
  • You are trying to access row index number 999, but there is none in your dataframe. Also, pandas dataframes can ontain way more rows than 999. – Odhian Sep 06 '21 at 15:50
  • 1
    The limitation is likely from the package/api you are using. What is `limit=amount_of_timeframe`? – not_speshal Sep 06 '21 at 15:51
  • What is `df.shape`? Dataframes can be huge. Memory blowout huge. It seems like you drop the last row in `bars[:-1]`. Could that be your problem? – tdelaney Sep 06 '21 at 15:51
  • I don't know your API, but "limit" could be setting a upper bound and the API may return less than that. You likely don't want to index from the limit without checking whether you really got that much data. In fact, it would make sense to always use the real size of the dataframe and only use a `amount_of_timeframe` size check as an error detection thing. – tdelaney Sep 06 '21 at 15:55
  • @tdelaney df.shape is (999, 15) – chronovirus Sep 07 '21 at 08:07
  • @Mohammad i am adding rows in the first snippet (data frame build). I am adding amount_of_timeframe rows, where amount_of_timeframe is greater than 999. `exchange.fetch_ohlcv()` is from ccxt library – chronovirus Sep 07 '21 at 08:11
  • @not_speshal yea, this seems to be the problem. sorry for wasting y'alls time. – chronovirus Sep 07 '21 at 08:42

1 Answers1

0

ccxt.binance() won't let you fetch more than 1000 candles. Try another API, that works with bigger data

chronovirus
  • 41
  • 5