How could messages get out of sequence?

Question

I have used QuickFix/.NET for a long time, but in last two days, the engine appears to have sent messages out of sequence twice.

Here is an example, the 3rd message is out of sequence:

20171117-14:44:34.627 : 8=FIX.4.4 9=70 35=0 34=6057 49=TRD 52=20171117-14:44:34.622 56=SS 10=208

20171117-14:44:34.635 : 8=FIX.4.4 9=0070 35=0 34=6876 49=SS 56=TRD 52=20171117-14:44:34.634 10=060

20171117-14:45:04.668 : 8=FIX.4.4 9=224 35=D 34=6059 49=TRD 52=20171117-14:45:04.668 56=SS 11=AGG-171117T095204000182 38=100000 40=D 44=112.402 54=2 55=USD/XXX 59=3 60=20171117-09:45:04.647 278=2cK-3ovrjdrk00X1j8h03+ 10=007

20171117-14:45:04.668 : 8=FIX.4.4 9=70 35=0 34=6058 49=TRD 52=20171117-14:45:04.642 56=SS 10=209

I understand that the QuickFix logger is not in a separate thread.

What could cause this to happen?

score 1 · Answer 1 · answered Nov 28 '17 at 22:12

The message numbers are generated using GetNextSenderMsgSeqNum method in quickfix/n, which use locking.

public int GetNextSenderMsgSeqNum()
        {
            lock (sync_) { return this.MessageStore.GetNextSenderMsgSeqNum(); }
        }

In my opinion, the messages are generated in sequence and your application is displaying in different order. In some situations the sender and receiver are not in sync, where receiver expects different sequence number, the initiator sends the message to acceptor that different sequence number is expected.

In that case, sequence number to can be changed to expected sequence number using the method call to update sequence or goto store folder and open file with extension.seqnums and update the sequence numbers. I hope this will help.

score 0 · Answer 2 · answered Nov 28 '17 at 14:14

As the datetime is the exact same on both messages, this may be a problem of sorting. This is common across any sorted list where the index is identical on two different items. If this were within your own code I would suggest that to resolve it, you include an extra element as part of the key, such a sequence number

score 0 · Answer 3 · answered Nov 28 '17 at 16:29

Multiple messages sent by QuickFix with identical timestamps may be sent out of sequence.

A previous answer on StackOverflow suggested re-ordering them on the receiving side, but was not accepted: QuickFix - messages out of sequence

If you decide to limit yourself to one message per millisecond, say with a sleep() command in between sends, be sure to increase your processes' scheduling priority: https://msdn.microsoft.com/en-us/library/windows/desktop/ms685100(v=vs.85).aspx You normally get a very long sleep even though you asked for only one millisecond, but I've gotten roughly 1-2 ms with ABOVE_NORMAL_PRIORITY_CLASS. (Windows 10)

You might try to disable Nagle's algorithm, which aggregates multiple TCP messages together and sends them at once. Nagle in and of itself can't cause messages to be sent out of order, but QuickFix may be manually buffering the messages in some weird way. Try telling QuickFix to send them immediately with SocketNodelay: http://quickfixn.org/tutorial/configuration.html

Is there any disadvantages to setting SocketNoDelay to true? — ManInMoon, Jan 17 '18 at 13:34
@ManInMoon The disadvantage to SocketNoDelay is increased bandwidth. Every TCP packet has an overhead of 40 bytes or more. With socket delay enabled, you bundle multiple messages together into one packet and only pay the overhead once. — Ebonair, Jan 18 '18 at 15:33

How could messages get out of sequence?

3 Answers3