0

I created a CPF on insert and update of documents. These CPF pipelines create multiple xdmp:spawn-tasks to perform variety of tasks. I have couple of questions with this approach.

  1. Some of the spawned tasks will modify the original document. Will this trigger the update workflow of CPF? I can use a flag on the document that will indicate that it is an update from the spawned task. But is there a more elegant way to do this ?
  2. Do I need to worry about deadlock? I mean if two tasks that were spawned from same CPF try to update the same document same time, how can I avoid this?

Basically I am trying to use envelope pattern for my inserted documents and wrap all the artifact documents into one single document. The reason I am using CPF to generate this artifact documents is that I can dump documents using MLCP or any other way into the database, and let CPF worry about the processing, instead of using a custom REST endpoint and having all the document ingest through this custom REST endpoint.

Dave Cassel
  • 8,352
  • 20
  • 38
Ravi
  • 1,179
  • 6
  • 13
  • 2
    Why are you using spawn explicitly in the first place? Why not just execute the work as individual pipeline steps and let CPF take care of the orchestration? It’s unclear from your explanation what you’re trying to accomplish. – Justin Makeig May 23 '16 at 17:57
  • I was originally doing in the CPF task it self, but when I do this I am sometimes getting out of memory exceptions, as once of the things I do is extract text from the documents using "xdmp:document-filter"... But using task server I did not get any, thus the spawning of tasks.. And I don;t think size of the original document is the issue.. My initial assumption of memory not getting released or something... But with task everything works fine – Ravi May 23 '16 at 19:47
  • xdmp:document-filter allows streaming results, so it doesn't necessarily need to consume a lot of memory. Filtering can be a heavy process though. It might be worth opening another question on SO to share some more details and ask why you are getting memory exceptions.. – grtjn May 24 '16 at 08:19

1 Answers1

0

I would recommend not spawning tasks to apply updates to the document at hand. Such updates would interfere with the natural flow of CPF states. CPF itself is designed to take a document through multiple states, and have each transition contribute to the total transformation.

So, let the document go through multiple CPF states, and do your updates in one of more of such state transitions (pipelines). Each state transition is already a separate transaction, so no need to spawn tasks.

I don't think you need to worry about deadlocks, but I can easily imagine that parallel execution of updates will cause part of the updates to get lost as they might get overwritten by each other. That kind of depends on the exact code you are using. MarkLogic's transaction mechanism normally guards against that, but it is easy to write code that bypasses that mechanism.

HTH!

grtjn
  • 20,254
  • 1
  • 24
  • 35
  • 1
    Useful reading: [Preventing Deadlocks](http://docs.marklogic.com/guide/app-dev/transactions#id_85082) – Dave Cassel May 23 '16 at 18:11
  • I was originally doing in the CPF task it self, but when I do this I am sometimes getting out of memory exceptions, as once of the things I do is extract text from the documents using "xdmp:document-filter"... But using task server I did not get any, thus the spawning of tasks.. And I don;t think size of the original document is the issue.. My initial assumption of memory not getting released or something... But with task everything works fine.. – Ravi May 23 '16 at 19:46