0

I am using Channel from system.threading.channels in asp.net core 3.1 application where I am continuously writing into a channel and reading from it.

When I am reading a item from channel, I am making a database call to save the item.

Questions is, if database call fails, can I move to item back to channel to re-read it again?

private static async Task Consume(ChannelReader<int> c)
{
  try
   {
     while (true)
     {
         int item = await c.ReadAsync();
         // process item to the database
         //if database call fails, can we move the item back to Channel
     }
}
  catch (ChannelClosedException) {}
}
user584018
  • 10,186
  • 15
  • 74
  • 160
  • No. A reader is a reader. Besides, putting something that caused a failure in the same input queue can easily lead to an infinite loop. BTW that's not typical Channel code, the typical code is `await foreach(var item in c.ReadAllAsync())`. No need to handle `ChannelClosedException` in this case – Panagiotis Kanavos Sep 14 '20 at 14:21
  • In all queueing systems failed items are placed in retry queues, to ensure those objects that need retrying don't interfere with normal processing. If something fails too often, it's put in a dead letter queue – Panagiotis Kanavos Sep 14 '20 at 14:23
  • 1
    BTW the real question here isn't whether you can put something back in the channel. It's how to handle failures and retries in a pipeline. Channels aren't the only pipeline technology, TPL Dataflow offers higher-level processing blocks since 2012. In both cases the real question is how to build the pipeline to enable retries, how to route messages to different blocks based on properties like status etc, not how to put the bad message in the original input block – Panagiotis Kanavos Sep 14 '20 at 14:28
  • makes sense. but just wanted to know if something out of box available with `channel` – user584018 Sep 14 '20 at 14:29
  • It's a channel, not MSMQ. The code you posted looks like an attempt to convert Go code to .NET. You wouldn't put a bad message back into the original channel in Go either. What you ask is a transactional queue, something far more complex to get right than it looks. And even transactional queues like MSMQ use dead letter queues to prevent bad messages from flooding the input queue – Panagiotis Kanavos Sep 14 '20 at 14:34
  • Imagine for example 10 timeouts. If you put them back in the input queue, you'd end up processing all 10 of them, all over again. After a few loops, those bad messages would end up as 10 consecutive bad messages, delaying other messages – Panagiotis Kanavos Sep 14 '20 at 14:35
  • Or consider a single bad record, caused eg by a. out-of-bounds field. There's no point in retrying this, it will always fail. What you can do in this case is either route it to a "junk" block or logger. Perhaps a better way (and functional, as in F#) is to wrap all messages in a `Result<>` class and have each block just forward failures to the next block, only process the "good" messages. This is called [Railway Oriented Programming](https://fsharpforfunandprofit.com/rop/) – Panagiotis Kanavos Sep 14 '20 at 14:39
  • If you have to process eg 200 CSV files where some records can fail, it may be simpler to use a straightforward pipeline and either eject and log bad records (conditional routing), or forward them to the end (ROP) before logging them. – Panagiotis Kanavos Sep 14 '20 at 14:42

0 Answers0