0

My scenario is as follows:

I have 10 datasets I need to process. I will be using 10 threads to process them all in parallel (each can take up to an hour). Once I find some info in the dataset I want, I will write it to the sqlite database. I might also have to update an existing row. I won't be doing any selects or deletes until after all datasets are finished being processed.

From what I understand sqlite will not handle this scenario well since only 1 thread can lock the file to write and I don't want to hold up other threads to wait until the lock is aquired.

So my idea is that I create another thread to handle all these writes. When a processing thread finds something it wants to write to the db, it sends it to the writer thread. The writer thread can then create a new thread to write it to the db so it can handle if another request comes in and add it to a queue if something is already writing it to the db. Therefore, we only have 1 thread trying to actually write to the db.

My main question is as follows:

Will this work / is this sane? Also is there something that does this already?

I'm using python if that matters.

Thanks

TreeWater
  • 761
  • 6
  • 13
  • It sounds like you are making life difficult for yourself. Why not use a database that allows concurrent writes? What volume of data are you writing? How often? Have you considered Redis? Why are you using `sqlite`? – Mark Setchell Nov 27 '19 at 22:11
  • Just though the simplicity of sqlite would make my life easier but alas that is not the case. I went with postgres instead – TreeWater Dec 05 '19 at 05:18

0 Answers0