Example of how to avoid duplicate order ids and corrupt prints in a multithreaded app thanks to queues with IB TWS Python API

Question

My trading app written with Python and using the IB TWS API is multithreaded and I'm looking to solve once and for all the issues of corrupt prints and duplicate order ids, both of which can happen once multiple threads are running at the same time.

For the purpose of this question I've put together a simple script that has for goal nothing other than providing a support for solving it. What I'm hoping for is to see how someone more experienced would implement queues in order to make the script do 2 things:

1. print 'safely' in a non corrupt way
2. each order should execute without the 'duplicate order id' error code popping up

I know that I need to use queues but I'm just starting to read up on the topic and since the IB TWS API is fairly specific, I'm not sure how to implement. Thank you.

from ibapi.wrapper import EWrapper
from ibapi.client import EClient
from ibapi.order import Order
from ibapi.contract import Contract
import threading
import time

buyOrder1Id = 0
buyOrder2Id = 0
buyOrder3Id = 0
buyOrder4Id = 0
buyOrder5Id = 0
buyOrder6Id = 0
buyOrder7Id = 0
buyOrder8Id = 0
buyOrder9Id = 0
buyOrder10Id = 0

class TradingApp(EWrapper, EClient):

    def __init__(self):
        self.BidPrice = 0
        self.AskPrice = 0
        self.LastPrice = 0

        EClient.__init__(self,self)

    def error(self, reqId, errorCode, errorString):
        print("Error. Id: ", reqId, " Code: ", errorCode, " Msg: ", errorString)

    def nextValidId(self, orderId):
        super().nextValidId(orderId)
        self.nextValidOrderId = orderId

def websocket_con():
    app.run()

Underlying = Contract()
Underlying.localSymbol = "ESU2"
Underlying.secType = "FUT"
Underlying.currency = "USD"
Underlying.exchange = "GLOBEX"

BuyOrder1 = Order()
BuyOrder1.action = "BUY"
BuyOrder1.totalQuantity = 1
BuyOrder1.orderType = "MKT"
BuyOrder1.account = "DU2312534"
BuyOrder1.tif = "GTC"

app = TradingApp()
app.connect("127.0.0.1", 7497, 2000)

time.sleep(1)

# starting a separate daemon thread to execute the websocket connection
con_thread = threading.Thread(target=websocket_con, daemon=True)
con_thread.start()
time.sleep(1)# some latency added to ensure that the connection is established

def Level1():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder1Id = app.nextValidOrderId
        print("buyOrder1Id: ",buyOrder1Id)
        app.placeOrder(buyOrder1Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level2():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder2Id = app.nextValidOrderId
        print("buyOrder2Id: ",buyOrder2Id)
        app.placeOrder(buyOrder2Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level3():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder3Id = app.nextValidOrderId
        print("buyOrder3Id: ",buyOrder3Id)
        app.placeOrder(buyOrder3Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level4():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder4Id = app.nextValidOrderId
        print("buyOrder4Id: ",buyOrder4Id)
        app.placeOrder(buyOrder4Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level5():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder5Id = app.nextValidOrderId
        print("buyOrder5Id: ",buyOrder5Id)
        app.placeOrder(buyOrder5Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level6():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder6Id = app.nextValidOrderId
        print("buyOrder6Id: ",buyOrder6Id)
        app.placeOrder(buyOrder6Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level7():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder7Id = app.nextValidOrderId
        print("buyOrder7Id: ",buyOrder7Id)
        app.placeOrder(buyOrder7Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level8():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder8Id = app.nextValidOrderId
        print("buyOrder8Id: ",buyOrder8Id)
        app.placeOrder(buyOrder8Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level9():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder9Id = app.nextValidOrderId
        print("buyOrder9Id: ",buyOrder9Id)
        app.placeOrder(buyOrder9Id,Underlying,BuyOrder1)
        time.sleep(3)

def Level10():
    while True:
        app.reqIds(1)
        time.sleep(1)
        buyOrder10Id = app.nextValidOrderId
        print("buyOrder10Id: ",buyOrder10Id)
        app.placeOrder(buyOrder10Id,Underlying,BuyOrder1)
        time.sleep(3)

level1_thread = threading.Thread(target=Level1)
level1_thread.start()

level2_thread = threading.Thread(target=Level2)
level2_thread.start()

time.sleep(1)

level3_thread = threading.Thread(target=Level3)
level3_thread.start()

time.sleep(1)

level4_thread = threading.Thread(target=Level4)
level4_thread.start()

time.sleep(1)

level5_thread = threading.Thread(target=Level5)
level5_thread.start()

time.sleep(1)

level6_thread = threading.Thread(target=Level6)
level6_thread.start()

time.sleep(1)

level7_thread = threading.Thread(target=Level7)
level7_thread.start()

time.sleep(1)

level8_thread = threading.Thread(target=Level8)
level8_thread.start()

time.sleep(1)

level9_thread = threading.Thread(target=Level9)
level9_thread.start()

time.sleep(1)

level10_thread = threading.Thread(target=Level10)
level10_thread.start()

time.sleep(1)

Answer 1

I'm not familiar with Python, but I see the logical error - you're using "nextValiOrderId" in your app.1st thing you do - you start a bunch of threads.2nd thing that happens - each thread would send a request app.reqIds(1)

Now you need to understand what's reqIds() function logic (from this discussion):

The nextValidId method is a wrapper method. From the client, you need to call reqIds to get an Order ID. ib_api.reqIds(-1) The parameter of reqIds doesn't matter. Also, it doesn't have a return value. Instead, reqIds sends a message to IB, and when the response is received, the wrapper's nextValidId method will be called.

So you called reqIds(1) many times (almost at the same time), eventually the response will come in form of nextValidId() callback f-n, which sets app.nextValidOrderId to ever growing number, but the same time you reading that number from all your threads, which is complete mess.

Instead you could reorganize the code logic the following way:  - ask for reqIds(1) only once:

• then do nothing (no code executed after that request is sent) 
• the continuation logic will reside inside the body of the nextValidId() callback f-n: upon getting back a valid integer number from IB you would set it to some of you application variable and do not call reqIds(1) anymore.
• call your "main_continuation()" function will will use that number as an argument to pass to i-th thread, then in a loop you'd increment this number and pass it to the next thread, and so on.In this case none of the threads will be trying to use the same order ID.

But even this approach is not ideal.. in case you need to start 100s of threads they will exceed 50 requests / second rate (one of many other rate limits by IB API), thus may be add 30 milliseconds sleep between starting new threads (in the loop where you create them one-by-one). But now if you think over the design of your application a bit more - you'll see that you do NOT need many threads at all.

ps: I would strongly recommend for any Pythonista to carefully look at the ib_insync project, created by Ewald de Wit and it has growing community behind it and everybody is so happy with ease of use of ib_insync (written in python). This is their discussion forum.

and this is the ib_insync on the github Cheers,Dmitry

Answer 2

You need to add a thread lock to pull nextValidOrderId in a thread-safe manner.

Something like:

def generate_order_id(self):
    with self.order_id_lock:
        order_id = self.nextValidOrderId
        self.nextValidOrderId += 1
        print(f"The next valid order_id is: {self.nextValidOrderId}")
    return order_id

and in your --init-- part of the class:

self.order_id_lock = threading.lock()

The "with" will acquire and release the lock itself so don't need to worry about that.

You should also be inheriting EWrapper and EClient in your TradingApp class so that self will have access to those methods.

Also just fyi it's always a bad idea to use market orders algorithmically.