Backtrader problems in resampling to daily

Question

Using backtrader, I would like to retrieve quotes at a five minute scale and resample at a daily scale.

I am able to resample five minutes to sixty minutes, but can't do daily. Here is the code:

from __future__ import absolute_import, division, print_function, unicode_literals
import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

_CLIENTID = 100

class St(bt.Strategy):
    def __init__(self):
        self.sma = bt.indicators.SMA(self.data)

    def logdata(self):
        txt = []
        txt.append('{}'.format(len(self)))
        txt.append('{}'.format(self.data.datetime.datetime(0).isoformat()))
        txt.append('{:.2f}'.format(self.data.open[0]))
        txt.append('{:.2f}'.format(self.data.high[0]))
        txt.append('{:.2f}'.format(self.data.low[0]))
        txt.append('{:.2f}'.format(self.data.close[0]))
        txt.append('{:.2f}'.format(self.data.volume[0]))
        logger.debug(','.join(txt))

    data_live = False

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, 'DATA Notification:', data._getstatusname(status), *args)
        if status == data.LIVE:
            self.data_live = True

    def next(self):
        self.logdata()
        if not self.data_live:
            return

_TICKER = "TSLA-STK-SMART-USD"
_FROMDATE = datetime.datetime(2021,1,4)
_TODATE = datetime.datetime(2021,1,29)
_HAS_STATS = False

def run(args=None):
    cerebro = bt.Cerebro(stdstats=_HAS_STATS)

    store = ibstore.IBStore(host="127.0.0.1", port=7497, clientId= _CLIENTID )
    cerebro.broker = store.getbroker()
    stockkwargs = dict(
        timeframe=bt.TimeFrame.Minutes,
        compression=10,
        rtbar=False,
        historical=True,
        qcheck=0.5,
        fromdate=_FROMDATE,
        todate=_TODATE,
        latethrough=False,
        tradename=None
    )

    data0 = store.getdata(dataname=_TICKER, **stockkwargs)
    # cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)
    cerebro.resampledata(data0, timeframe=bt.TimeFrame.Days, compression=1)

    cerebro.run()

if __name__ == "__main__":
    run()

I get a connection, but no days. Here is the output (with no bars printed):

Server Version: 76
TWS Time at connection:20210303 09:39:17 EST
***** DATA Notification: DELAYED
***** DATA Notification: DISCONNECTED

However, if I go on a 60-minute resample, all is well. The code

cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)
# cerebro.resampledata(data0, timeframe=bt.TimeFrame.Days, compression=1)

results in

Server Version: 76
TWS Time at connection:20210303 09:36:27 EST
***** DATA Notification: DELAYED
DEBUG:__main__:30,2021-01-05T17:00:00,749.65,754.40,735.11,753.21,6011.00
DEBUG:__main__:31,2021-01-05T18:00:00,753.39,754.18,751.60,753.34,1126.00
DEBUG:__main__:32,2021-01-05T19:00:00,753.32,753.32,750.49,752.90,1179.00
DEBUG:__main__:33,2021-01-06T04:00:00,748.00,752.55,746.66,751.88,331.00
DEBUG:__main__:34,2021-01-06T05:00:00,751.60,753.00,749.26,750.50,137.00
(omitted)
DEBUG:__main__:286,2021-01-28T17:00:00,833.00,833.25,831.36,831.61,116.00
DEBUG:__main__:287,2021-01-28T18:00:00,831.60,833.96,830.11,831.00,175.00
DEBUG:__main__:288,2021-01-28T19:00:00,830.51,832.00,829.45,829.45,358.00
***** DATA Notification: DISCONNECTED

I am using these versions:

Python 3.7.4
ib                   0.8.0
IbPy2                0.8.0
numpy                1.19.2
pandas               1.1.3

TL;DR I did several other experiments.

getData timeframe	getData compression	resample timeframe	resample compression	outcome
Minutes	5	Minutes	60	can see hourly, as expected
Minutes	5	Days	1	(empty)
(blank)	(blank)	Days	1	(empty)
Days	1	Days	1	(empty)
Days	1	(commented out)	(commented out)	(empty)

There might be a solution at https://community.backtrader.com/topic/1397/resampling-fixes-for-1-9-67-122, which says "Somewhere along the path to implement trading calendars a bug was introduced which Prevented resampling which was limited to compression only (for sure when the timeframe was bt.TimeFrame.Days although it could have affected other higher timeframes)" I will research this version and post if it solves the problem. — rajah9, Mar 08 '21 at 14:24
Found tag 1.9.67.122. This seems out of date: It was tagged on "Oct 11, 2018 · 79 commits to master since this tag." — rajah9, Mar 20 '21 at 14:00

score 2 · Accepted Answer · edited Mar 30 '21 at 12:22

You are right there was a known issue a few years a back with this. There is an easy work-around though. Instead of using the backtrader resampling facility, use Interactive Brokers.

You can bypass resample and call the data directly from IB. Just remember that you must add the shortest timeframe first to Backtrader.


for tf_com in [(bt.TimeFrame.Minutes, 1), (bt.TimeFrame.Days, 1)]:
    stockkwargs = dict(
        timeframe=tf_com[0],
        compression=tf_com[1],
        rtbar=False,
        historical=True,
        qcheck=0.5,
        fromdate=_FROMDATE,
        todate=_TODATE,
        latethrough=False,
        tradename=None
    )

    data0 = store.getdata(dataname=_TICKER, **stockkwargs)

    cerebro.adddata(data0)

You can adjust your timeframe and compression in the tuples in the first line. I will adjust it to 5 and 15 minutes so you can see the output.

for tf_com in [(bt.TimeFrame.Minutes, 5), (bt.TimeFrame.Minutes, 15)]:

************ OUTPUT ************

2021-03-25 12:00:00 636.77 632.71 
2021-03-25 12:05:00 634.69 632.71 
2021-03-25 12:10:00 632.71 632.71 
2021-03-25 12:15:00 640.39 643.00 
2021-03-25 12:20:00 640.68 643.00 
2021-03-25 12:25:00 643.00 643.00 
2021-03-25 12:30:00 641.33 640.00 
2021-03-25 12:35:00 637.84 640.00 
2021-03-25 12:40:00 640.00 640.00 
2021-03-25 12:45:00 640.34 633.95 
2021-03-25 12:50:00 636.15 633.95 
2021-03-25 12:55:00 633.95 633.95 
2021-03-25 13:00:00 633.50 637.43 
2021-03-25 13:05:00 634.95 637.43 
2021-03-25 13:10:00 637.43 637.43

EDIT: IN RESPONSE TO OP COMMENT

I beg to differ. The days are sampled by Interactive Brokers so you will have daily information. You must be sure when using 5 minute data at the same time not to go back too far with your start date. IB is not a historical data provider.

Here is the entire code for your reference.

import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import os
from dotenv import load_dotenv

load_dotenv()


class St(bt.Strategy):

    def __init__(self):
        self.data_live = False
        self.timeframes = {4: "minute", 5: "day"}


    def next(self):
        time = self.data.datetime.time()
        start_day = datetime.time(4, 20, 0)
        end_day = datetime.time(19, 40, 0)
        if time < start_day or time > end_day:
            print_date = True
        else:
            print_date = False

        if not self.data_live and print_date:
            print(
                f"{self.data.datetime.datetime()}   "
                
                f"data0: tf: {self.timeframes[self.datas[0]._timeframe]} "
                f"comp: {self.datas[0]._compression},   "
                f"{self.datas[0].close[0]:5.2f} "
                
                f"data1: tf: {self.timeframes[self.datas[1]._timeframe]} "
                f"comp: {self.datas[1]._compression} "
                f"{self.datas[1].close[0]:5.2f} "
            )
            return


_TICKER = "TSLA-STK-SMART-USD"
_FROMDATE = datetime.datetime(2021, 3, 10)
_TODATE = datetime.datetime(2021, 3, 24)
_HAS_STATS = False
_CLIENTID = os.getenv("CLIENTID")
_PORT = os.getenv("SOCKET_PORT")


def run():
    cerebro = bt.Cerebro(stdstats=_HAS_STATS)
    cerebro.addstrategy(St)

    store = ibstore.IBStore(host="127.0.0.1", port=int(_PORT), clientId=_CLIENTID)
    cerebro.broker = store.getbroker()

    for tf_com in [(bt.TimeFrame.Minutes, 5), (bt.TimeFrame.Days, 1)]:
        stockkwargs = dict(
            timeframe=tf_com[0],
            compression=tf_com[1],
            rtbar=False,
            historical=True,
            qcheck=0.5,
            fromdate=_FROMDATE,
            todate=_TODATE,
            latethrough=False,
            tradename=None,
        )

        data = store.getdata(dataname=_TICKER, **stockkwargs)

        cerebro.adddata(data)

    cerebro.run()


if __name__ == "__main__":
    run()

Here is the output with the middle of the days removed.

Please note that backtrader will set the next day value at the end of the last bar of the previous day.

2021-03-10 19:55:00   data0: tf: minute comp: 5,   664.56 data1: tf: day comp: 1 664.56 
2021-03-11 04:00:00   data0: tf: minute comp: 5,   699.10 data1: tf: day comp: 1 664.56 
2021-03-11 04:05:00   data0: tf: minute comp: 5,   701.50 data1: tf: day comp: 1 664.56 
2021-03-11 04:10:00   data0: tf: minute comp: 5,   699.00 data1: tf: day comp: 1 664.56 
2021-03-11 04:15:00   data0: tf: minute comp: 5,   701.00 data1: tf: day comp: 1 664.56 
2021-03-11 19:45:00   data0: tf: minute comp: 5,   698.05 data1: tf: day comp: 1 664.56 
2021-03-11 19:50:00   data0: tf: minute comp: 5,   698.09 data1: tf: day comp: 1 664.56 
2021-03-11 19:55:00   data0: tf: minute comp: 5,   698.50 data1: tf: day comp: 1 664.56 
2021-03-11 19:55:00   data0: tf: minute comp: 5,   698.50 data1: tf: day comp: 1 698.50 
2021-03-12 04:00:00   data0: tf: minute comp: 5,   674.25 data1: tf: day comp: 1 698.50 
2021-03-12 04:05:00   data0: tf: minute comp: 5,   676.00 data1: tf: day comp: 1 698.50 
2021-03-12 04:10:00   data0: tf: minute comp: 5,   669.00 data1: tf: day comp: 1 698.50 
2021-03-12 04:15:00   data0: tf: minute comp: 5,   669.46 data1: tf: day comp: 1 698.50 
2021-03-12 19:45:00   data0: tf: minute comp: 5,   693.30 data1: tf: day comp: 1 698.50 
2021-03-12 19:50:00   data0: tf: minute comp: 5,   692.80 data1: tf: day comp: 1 698.50 
2021-03-12 19:55:00   data0: tf: minute comp: 5,   692.99 data1: tf: day comp: 1 698.50 
2021-03-12 19:55:00   data0: tf: minute comp: 5,   692.99 data1: tf: day comp: 1 692.99 
2021-03-15 04:00:00   data0: tf: minute comp: 5,   689.00 data1: tf: day comp: 1 692.99 
2021-03-15 04:05:00   data0: tf: minute comp: 5,   692.13 data1: tf: day comp: 1 692.99 
2021-03-15 04:10:00   data0: tf: minute comp: 5,   692.12 data1: tf: day comp: 1 692.99 
2021-03-15 04:15:00   data0: tf: minute comp: 5,   692.31 data1: tf: day comp: 1 692.99 
2021-03-15 19:45:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 692.99 
2021-03-15 19:50:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 692.99 
2021-03-15 19:55:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 692.99 
2021-03-15 19:55:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 702.00 
2021-03-16 04:00:00   data0: tf: minute comp: 5,   704.01 data1: tf: day comp: 1 702.00 
2021-03-16 04:05:00   data0: tf: minute comp: 5,   704.98 data1: tf: day comp: 1 702.00 
2021-03-16 04:10:00   data0: tf: minute comp: 5,   704.99 data1: tf: day comp: 1 702.00 
2021-03-16 04:15:00   data0: tf: minute comp: 5,   706.20 data1: tf: day comp: 1 702.00 
2021-03-16 19:45:00   data0: tf: minute comp: 5,   673.65 data1: tf: day comp: 1 702.00 
2021-03-16 19:50:00   data0: tf: minute comp: 5,   674.00 data1: tf: day comp: 1 702.00 
2021-03-16 19:55:00   data0: tf: minute comp: 5,   674.10 data1: tf: day comp: 1 702.00 
2021-03-16 19:55:00   data0: tf: minute comp: 5,   674.10 data1: tf: day comp: 1 674.10 
2021-03-17 04:00:00   data0: tf: minute comp: 5,   672.00 data1: tf: day comp: 1 674.10 
2021-03-17 04:05:00   data0: tf: minute comp: 5,   675.80 data1: tf: day comp: 1 674.10 
2021-03-17 04:10:00   data0: tf: minute comp: 5,   677.19 data1: tf: day comp: 1 674.10 
2021-03-17 04:15:00   data0: tf: minute comp: 5,   676.03 data1: tf: day comp: 1 674.10 
2021-03-17 19:45:00   data0: tf: minute comp: 5,   699.66 data1: tf: day comp: 1 674.10 
2021-03-17 19:50:00   data0: tf: minute comp: 5,   699.90 data1: tf: day comp: 1 674.10 
2021-03-17 19:55:00   data0: tf: minute comp: 5,   699.74 data1: tf: day comp: 1 674.10 
2021-03-17 19:55:00   data0: tf: minute comp: 5,   699.74 data1: tf: day comp: 1 699.74 
2021-03-18 04:00:00   data0: tf: minute comp: 5,   685.00 data1: tf: day comp: 1 699.74 
2021-03-18 04:05:00   data0: tf: minute comp: 5,   686.35 data1: tf: day comp: 1 699.74 
2021-03-18 04:10:00   data0: tf: minute comp: 5,   688.32 data1: tf: day comp: 1 699.74 
2021-03-18 04:15:00   data0: tf: minute comp: 5,   692.50 data1: tf: day comp: 1 699.74 
2021-03-18 19:45:00   data0: tf: minute comp: 5,   652.10 data1: tf: day comp: 1 699.74 
2021-03-18 19:50:00   data0: tf: minute comp: 5,   651.00 data1: tf: day comp: 1 699.74 
2021-03-18 19:55:00   data0: tf: minute comp: 5,   650.56 data1: tf: day comp: 1 699.74 
2021-03-18 19:55:00   data0: tf: minute comp: 5,   650.56 data1: tf: day comp: 1 650.56 
2021-03-19 04:00:00   data0: tf: minute comp: 5,   661.00 data1: tf: day comp: 1 650.56 
2021-03-19 04:05:00   data0: tf: minute comp: 5,   663.00 data1: tf: day comp: 1 650.56 
2021-03-19 04:10:00   data0: tf: minute comp: 5,   663.60 data1: tf: day comp: 1 650.56 
2021-03-19 04:15:00   data0: tf: minute comp: 5,   666.48 data1: tf: day comp: 1 650.56 
2021-03-19 19:45:00   data0: tf: minute comp: 5,   652.50 data1: tf: day comp: 1 650.56 
2021-03-19 19:50:00   data0: tf: minute comp: 5,   652.02 data1: tf: day comp: 1 650.56 
2021-03-19 19:55:00   data0: tf: minute comp: 5,   652.20 data1: tf: day comp: 1 650.56 
2021-03-19 19:55:00   data0: tf: minute comp: 5,   652.20 data1: tf: day comp: 1 652.20 
2021-03-22 04:00:00   data0: tf: minute comp: 5,   665.97 data1: tf: day comp: 1 652.20 
2021-03-22 04:05:00   data0: tf: minute comp: 5,   664.00 data1: tf: day comp: 1 652.20 
2021-03-22 04:10:00   data0: tf: minute comp: 5,   665.00 data1: tf: day comp: 1 652.20 
2021-03-22 04:15:00   data0: tf: minute comp: 5,   663.94 data1: tf: day comp: 1 652.20 
2021-03-22 19:45:00   data0: tf: minute comp: 5,   668.39 data1: tf: day comp: 1 652.20 
2021-03-22 19:50:00   data0: tf: minute comp: 5,   668.56 data1: tf: day comp: 1 652.20 
2021-03-22 19:55:00   data0: tf: minute comp: 5,   669.35 data1: tf: day comp: 1 652.20 
2021-03-22 19:55:00   data0: tf: minute comp: 5,   669.35 data1: tf: day comp: 1 669.35 
2021-03-23 04:00:00   data0: tf: minute comp: 5,   670.25 data1: tf: day comp: 1 669.35 
2021-03-23 04:05:00   data0: tf: minute comp: 5,   665.33 data1: tf: day comp: 1 669.35 
2021-03-23 04:10:00   data0: tf: minute comp: 5,   664.11 data1: tf: day comp: 1 669.35 
2021-03-23 04:15:00   data0: tf: minute comp: 5,   662.93 data1: tf: day comp: 1 669.35 
2021-03-23 19:45:00   data0: tf: minute comp: 5,   661.00 data1: tf: day comp: 1 669.35 
2021-03-23 19:50:00   data0: tf: minute comp: 5,   661.08 data1: tf: day comp: 1 669.35 
2021-03-23 19:55:00   data0: tf: minute comp: 5,   663.00 data1: tf: day comp: 1 669.35 
2021-03-23 19:55:00   data0: tf: minute comp: 5,   663.00 data1: tf: day comp: 1 663.00

=== OP Verification ===

Here is what the data0 and data1 plot look like, adding a cerebro.plot() after the cerebro.run().

That's good looking daily data.

I have tried your workaround. No joy. Although you've pushed the timeframe and compressions into a tuple, the daily compression problem remains. Try a tuple with `bt.TimeFrame.Days, 1`, please. — rajah9, Mar 30 '21 at 01:50
Thanks for the full code and the explanation. Nice verification to show the close on `datas[1]` to show the daily close. I added a `cerebro.plot()` resulting graph to your answer, which graphically shows your solution. — rajah9, Mar 30 '21 at 12:27

Backtrader problems in resampling to daily

1 Answers1

EDIT: IN RESPONSE TO OP COMMENT