2

I am pulling multiple datasets from multiple sources and want to parallelize that using the threading library in Python 3.8.2. All datasets end in different contexts in a ConjunctiveGraph, which uses an IOMemory store. I saw that a ConcurrentStore exists, but there is no example, not documented that much and does not implement the Store interface, which makes it unusable.

While reading through the code of IOMemory, I saw places where the existence of elements in collections is checked and then things get added without that I saw locking code around.

So is IOMemory already thread-safe or do I need to 'fix' ConcurrentStore?

Torsten Knodt
  • 477
  • 1
  • 5
  • 20

1 Answers1

0

Is the answer perhaps dependent on whether the underlying Python objects are themselves threadsafe? Python set() underlies the RDFlib Graph.

You could also perhaps create independent Graphs and then merge them all into a ConjunctiveGraph or a Dataset once they have each been fully populated. That may bypass the issue.

Any work on ConcurrentStore would be appreciated though!

Nicholas Car
  • 1,164
  • 4
  • 7