2

I have read about BigQueue and the like to be a durable disk bound queue.

Is there anything for c# that is not durable but "pages" queue overflow to disk.

I have incoming byte[] after incoming byte[] and don't want to overload the memory. BlockingCollection is ideal because it allows me to have a processing thread and GetConsumingEnumerable(),

Is there any queue out there like BlockingCollection that keeps as much in memory as possible, isn't durable, pages overflow to disk?

maxfridbe
  • 5,872
  • 10
  • 58
  • 80
  • 1
    The operating system does that for you. If you don't need to be durable, you can just allocate as much memory as you need, and the operating system will automatically assign it to you, given that the page file is allowed to grow and you're running in 64 bit mode. – PMF Mar 03 '21 at 14:36
  • This is on a memory constrained system (docker) which has a 2gig max memory set so I wanted to be proactive and only allow 500mb worth of memory to be used at a time. – maxfridbe Mar 03 '21 at 14:52
  • I have done this before (manually swapping list elements to disk when the list gets large), but it is slow, complicated and requires quite a bit of code. I think there is a way to increase the docker memory size. – PMF Mar 03 '21 at 16:46

1 Answers1

0

I ended up using BlockingCollection<NewItem>() where NewItem has Size, Page(), and GetBytes() members. Keeping the complexity low. For now am paging to disk tempfiles. Would love to find a binary manager which could page smarter.

maxfridbe
  • 5,872
  • 10
  • 58
  • 80