marklogic MLCP output_uri_replace fail

Question

I'm trying to use a regular expression with -output_uri_replace, and it is failing.

Here is my options file:

import
-host
localhost
-port
8877
-username
xxxx
-password
xxxx
-input_file_path 
to-import
-output_uri_replace
^.*(/[^/]+/),$1

I consistently get this error:

$ import
java.lang.IllegalArgumentException: Invalid option argument for output_uri_replace :^.*(/[^/]+/),$1
        at com.marklogic.contentpump.Command.applyCommonOutputConfigOptions(Command.java:2422)
        at com.marklogic.contentpump.Command$1.applyConfigOptions(Command.java:470)
        at com.marklogic.contentpump.Command$1.createJob(Command.java:370)
        at com.marklogic.contentpump.ContentPump.runCommand(ContentPump.java:238)
        at com.marklogic.contentpump.ContentPump.main(ContentPump.java:74)

which is obviously not very helpful. I don't know why it doesn't report the error from the regex engine, which is capable of giving useful messages.

I tried with and without quotes. No difference.

Just to prove it works, here is what happens in Groovy (which uses the same regex engine as Java)

This is exactly the result I want. And by the way, wouldn't this be a common case? You have a bunch of stuff in the directory immediately below and you want to lop off everything but that?

Couldn't there be a working example in the documentation?

And why the @#$% doesn't this work in MLCP when it's clearly a valid regex?

I don't think it's the regex or the replacement. I'm seeing the same issue with just `foo,bar`. It looks like it requires single quotes surrounding the replacement https://github.com/marklogic/marklogic-contentpump/blob/25d395a1247b86ca5fd3f273b64430cba09242e7/src/main/java/com/marklogic/contentpump/Command.java#L2420 however, if I use `'foo','bar'` then I see it showing `invalid option argument for output_uri_replace :foo',bar` so I suspect something is wrong with quote stripping and requirements to have them. — Mads Hansen, Jul 12 '22 at 12:06

Mads Hansen · Accepted Answer · 2022-07-12T12:41:26.467

There isn't anything wrong with your regex.

It is just that the -options_file options have an odd requirement/expectation (maybe a bug) that the entire option value be wrapped in double quotes, and also that the replacement value be wrapped in single quotes.

https://docs.marklogic.com/guide/mlcp/import#id_42798

The -output_uri_replace option accepts a comma delimited list of regular expression and replacement string pairs. The string portion must be enclosed in single quotes:

I don't think that is very clear that the double quotes wrapping the option are required, and the error doesn't help you understand that.

I noticed that in the source code where that exception is thrown it is looking for single quotes wrapping the replacements value

// Replacement string is expected to be in ''
for (int i = 0; i < replace.length - 1; i++) {
   String replacement = replace[++i].trim();
   if (!replacement.startsWith("'") || 
       !replacement.endsWith("'")) {
        throw new IllegalArgumentException(
                "Invalid option argument for "
                + OUTPUT_URI_REPLACE + " :" + uriReplace);
    }
}

So, I tried wrapping the replacement value in quotes: ^.*(/[^/]+/),$1 But then it complained about a malformed option, and I see it was making calls to OptionsFileUtil.removeQuoteCharactersIfNecessary() So, then I tried wrapping both values in single quotes: '^.*(/[^/]+/)','$1' But then it complained about invalid argument and the quotes were goofy:

IllegalArgumentException: Invalid option argument for output_uri_replace :^.*(/[^/]+/)','$1

I also verified this with simple strings foo,bar and saw same results.

If you wrap the entire value in double quotes and wrap the replacement value in single quotes:

-output_uri_replace
"^.*(/[^/]+/),'$1'"

then I was able to get it to execute without error.

I was also able to pass the -output_uri_replace via commandline, and it works (as long as you wrap the replacement value with single quotes):

mlcp.bat -options_file options.txt -output_uri_replace "^.*(/[^/]+/),'$1'"

Thanks for the help. I hope this can at least be documented, and ideally the behavior improved. — miles zarathustra, Jul 12 '22 at 16:39
It helps to have customers voice their issues and pain points. I'd advise to open a Support ticket or create a GitHub issue: https://github.com/marklogic/marklogic-contentpump/issues/new — Mads Hansen, Jul 12 '22 at 16:41
Personally, I'd rather see the implementation improve, rather than document the requirement for double quotes wrapping the value (feels like a bug to me) — Mads Hansen, Jul 12 '22 at 16:43

marklogic MLCP output_uri_replace fail

1 Answers1