where does p:insert() install the pipeline?

Question

I have an XQuery function which I am using to install a CPF pipeline programatically. I am testing this by executing the function call from the Query console, where my content source is set to the content db which has CPF enabled.

I was executing the following to troubleshoot what is going on. $processsed_pipline_config contains the pipeline config XML. The return returns only the pipeline that was inserted, so it looks to me like the pipeline is being inserted into a db that only has a single pipeline?

let $pipeline_id := p:insert($processed_pipeline_config)

return
  for $pipeline in p:pipelines()
    return $pipeline

When I go to the admin console, this inserted pipeline does not show up in the pipeline list for the content db.

Update

I also tried to invoke it in the context of the schema db, as suggested below. No luck either.

  let $pipeline_id :=
    xdmp:invoke-function(
      function() {
         p:insert($processed_pipeline_config)
      },
      <options xmlns="xdmp:eval">
        <database>{ xdmp:schema-database() }</database>
        <transaction-mode>update-auto-commit</transaction-mode>
        <isolation>different-transaction</isolation>
      </options>
    )

score 1 · Answer 1 · answered Feb 16 '16 at 16:41

1

You should run the pipeline API against the schema database for your content database, not your content database: that is where CPF will look for it.

p:insert will insert the pipeline into a pipeline collection with the default permissions.

I think your problem here is that you are fetching the list of pipelines in the same transaction as you add the pipeline. A transaction is not going to see the results of its own commit. If you run the lookup in a separate query, you should see your pipeline.

answered Feb 16 '16 at 16:41

mholstege

4,902
11
7

I actually do see the pipeline I added when I query using p:pipelines(), I just don't see it when I go to look at pipelines in the admin console under CPF=>Pipelines. Most likely am inserting the pipeline into the wrong location. I will attempt to run the insert in the context of the schema db. – TJ Tang Feb 16 '16 at 21:17
I tried as you suggested to run it against the schema database. No deal. When I query with the p:pipelines() function after the insert, I see it as the one and only pipeline. – TJ Tang Feb 16 '16 at 22:10
Did you make sure you ran the query in a separate transaction as well? I think simplest is to just use two QC queries, one to do the insert, and a second for the reading.. – grtjn Feb 17 '16 at 09:21
Consider using the Management API too for managing CPF resources - http://docs.marklogic.com/REST/management/content-processing-framework-(cpf) - you just make HTTP requests instead of dealing directly with the CPF APIs. If you're inclined towards using Java, check out ml-gradle too - here's a sample project for configuring CPF - https://github.com/rjrudin/ml-gradle/tree/master/examples/cpf-project . – rjrudin Feb 17 '16 at 14:17
@rjrudin thanks for the suggestion. I am looking to do this in server side XQuery as a convenience function in a library I am developing for use internally for granting roles based on some predefined rules. – TJ Tang Feb 18 '16 at 15:45

score 0 · Accepted Answer · answered Feb 18 '16 at 15:40

The solution is that the p:insert() call should be made against the triggers database, rather than content or schema. Essentially:

      xdmp:invoke-function(
        function() {
         p:insert($processed_pipeline_config)
        },
        <options xmlns="xdmp:eval">
          <database>{ xdmp:triggers-database() }</database>
          <transaction-mode>update-auto-commit</transaction-mode>
          <isolation>different-transaction</isolation>
        </options>
      )

After doing this, the pipeline showed up in the list in the Admin console.

where does p:insert() install the pipeline?

2 Answers2