1

We have a Data loader service that uses NServiceBus to insert data(if not already present)into SQL DB. The queue is configured with Concurrencylevel > 1 as the data to load might get huge. Since the Concurrencylevel > 1, it results in duplicate inserts. Is there a way to handle this within NServiceBus.

Note: We have already considered and ruled out creating thread safe locks

KJE
  • 11
  • 2

1 Answers1

0

Generally speaking, there's no need to run the endpoint with Concurrency Level of one. You also don't need to manage the threading and fiddle with concurrency/locks when it comes to NServiceBus. There are other factors on how the system needs to be designed to make it work:

  • Different transports have different levels of transaction support. Choose one that supports Transactions. It means if the message is retried, you won't get duplicated messages/data.

  • Try to work your system with idempotency. It means that with the lack of transactions (not supported by the transport or disabled by the code) if you process a message twice, you won't have multiple data/side effects. The 'how' part requires better knowledge about the data you're dealing with and your domain.

Hadi Eskandari
  • 25,575
  • 8
  • 51
  • 65
  • Would like to add some clarity to my question - The queue message are not getting duplicated, but say the file to load has duplicate records - though the queue has logic to upsert, since it runs with multiple concurrency, all the parallel threads assume it is a new record and insert rather than the first message being inserted and the subsequent messages with the same record data getting updated – KJE Aug 14 '17 at 16:28
  • Do you have some sort of a unique constraint on your db column? If you send 1 message per entry of your file (instead of processing the whole file in one message), what happens in case of concurrent processing is that two threads pick two messages with duplicate business data, first one inserts, the second one fails due to violating the unique constraint and it gets retried. On the next retry, since it finds the record has already been inserted it can ignore the message. – Hadi Eskandari Aug 16 '17 at 02:12