Batch saves on Xodus

Question

Is there a way to save hundreds of thousands of items on Xodus quickly? Some sort of batch feature where we can specify a very big transaction for that?

The goal here is to update a large set of entities during an ingestion phase. We usually approach this by deleting all the items that will be updated and then add them again. Is there a way to delete an entity without finding it first (like a delete from from MySQL)? If this is not the recommended way to do this in Xodus' Entity Store, what is the recommended way?

Cheers

Would you like to save hundreds of thousands of items in a single transaction atomically in terms of ACID guarantees? — Vyacheslav Lukianov, May 28 '19 at 18:49
That would be nice yes. But what I would like to know if there are strategies to save data in bulk to Xodus — Bruno René Santos, May 29 '19 at 10:34
Well if there is an alternative to save without ACID but faster we would pursue it too — Bruno René Santos, May 29 '19 at 11:55

score 1 · Answer 1 · answered Jun 04 '19 at 17:05

The recommended way is to split all the items onto several batches with a separate transaction for each batch. Deleting all the items and them adding them again seems to be an overhead if not every field on an entity is updated. If you wish to the whole ingestion process to be atomic then probably it makes sense to define some kind of application level metadata (e.g. which fields of the entities should be used, etc.) and to update it at the end of ingestion phase.

Using the Environments API, you can try the "deleting all items and adding them again" strategy. You can remove the Store completely and then add all the items. This strategy also won't work for arbitrary number of items in a single transaction.

Batch saves on Xodus

1 Answers1