Error "Duplicate namespace in XML input" when retrieving a list over NETCONF using Opendaylight

Question

I am using Opendaylight (release 0.10.1) as NETCONF client to send requests to a Netopeer2 server.
My YANG model is a third-party model consisting of several modules which contain, among other attributes, several list elements.
This translates into an XML request and response pair that looks something like this (simplified):

Request:

<rpc message-id="m-3" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <get-config>
        <source>
            <running/>
        </source>
        <filter xmlns:ns0="urn:ietf:params:xml:ns:netconf:base:1.0" ns0:type="subtree">
            <TopElement xmlns="top-element">
                <id>1</id>
            </TopElement>
        </filter>
    </get-config>
</rpc>

Response:

<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-3">
    <data>
        <TopElement xmlns="top-element">
            <id>1</id>
            <ParentElement xmlns="parent-element">
                <id>1</id>
                <ChildElement1 xmlns="child-element1">
                    <id>1</id>
                    <attributes>
                        ...
                    </attributes>
                </ChildElement1>
                <ChildElement1 xmlns="child-element1">
                    <id>2</id>
                    <attributes>
                        ...
                    </attributes>
                </ChildElement1>
                <ChildElement2 xmlns="child-element2">
                    <id>1</id>
                    <attributes>
                        ...
                    </attributes>
                </ChildElement2>
                <ChildElement1 xmlns="child-element1">
                    <id>3</id>
                    <attributes>
                        ...
                    </attributes>
                </ChildElement1>
            </ParentElement>
        </TopElement>
    </data>
</rpc-reply>

However, such a request causes a "Master is down" error, which I have tried investigating using the Karaf log file. I tracked the problem down to a akka.pattern.AskTimeoutException timeout which seems to be caused by a parsing error of the XML response:

2020-04-08T09:09:11,733 | ERROR | nioEventLoopGroupCloseable-3-1 | AbstractFuture                   | 42 - com.google.guava - 25.1.0.jre |  -  | RuntimeException while executing runnable CallbackListener{org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceRpc$1@165744d4} with executor MoreExecutors.directExecutor()
java.lang.IllegalArgumentException: Failed to parse data response [data: null]
    at org.opendaylight.netconf.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:285) ~[386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:75) ~[386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceRpc$1.onSuccess(NetconfDeviceRpc.java:60) ~[386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceRpc$1.onSuccess(NetconfDeviceRpc.java:57) ~[386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1355) ~[42:com.google.guava:25.1.0.jre]
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398) ~[42:com.google.guava:25.1.0.jre]
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1024) [42:com.google.guava:25.1.0.jre]
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:866) [42:com.google.guava:25.1.0.jre]
    at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:689) [42:com.google.guava:25.1.0.jre]
    at org.opendaylight.netconf.sal.connect.netconf.listener.UncancellableFuture.set(UncancellableFuture.java:45) [386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.processMessage(NetconfDeviceCommunicator.java:338) [386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:270) [386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:49) [386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    at org.opendaylight.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:64) [379:org.opendaylight.netconf.netty-util:1.6.1]
    at org.opendaylight.netconf.nettyutil.AbstractNetconfSession.channelRead0(AbstractNetconfSession.java:187) [379:org.opendaylight.netconf.netty-util:1.6.1]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) [71:io.netty.transport:4.1.34.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [66:io.netty.codec:4.1.34.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [66:io.netty.codec:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) [71:io.netty.transport:4.1.34.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [66:io.netty.codec:4.1.34.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [66:io.netty.codec:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) [71:io.netty.transport:4.1.34.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:350) [71:io.netty.transport:4.1.34.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [68:io.netty.common:4.1.34.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [68:io.netty.common:4.1.34.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) [71:io.netty.transport:4.1.34.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) [68:io.netty.common:4.1.34.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [68:io.netty.common:4.1.34.Final]
    at java.lang.Thread.run(Thread.java:748) [?:?]
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[-1,-1]
Message: Duplicate namespace "child-element1" element "ChildElement1" in XML input
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:383) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:321) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:403) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:321) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:403) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.parse(XmlParserStream.java:214) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.traverse(XmlParserStream.java:244) ~[410:org.opendaylight.yangtools.yang-data-codec-xml:2.1.10]
    at org.opendaylight.netconf.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:281) ~[386:org.opendaylight.netconf.sal-netconf-connector:1.9.1]
    ... 34 more

Can it really be because of the list? The XML is compliant to RFC 7950:

7.8.5. XML Encoding Rules

A list is encoded as a series of XML elements, one for each entry in the list. Each element's local name is the list's identifier, and its namespace is the module's XML namespace (see Section 7.1.3). There is no XML element surrounding the list as a whole.

Any suggestion would be greatly appreciated. Thanks!

score 0 · Answer 1 · answered Apr 08 '20 at 18:15

0

I'm not sure I know the answer here, but could the problem just be that some of the list items are named the same? as in:

<ChildElement1 xmlns="child-element1">

I know the id's are unique, but that's what stood out to me.

answered Apr 08 '20 at 18:15

jamo

742
1
4
5

Yes, but that's how you do lists in XML, the element name doesn't have to be unique. It looks like Opendaylight added that restriction, but I have no idea how to work with it. – amethyst Apr 09 '20 at 05:45
I should also add that the XML list I posted has been created by issuing POST commands to Opendaylight. So I can create ChildElements in the Netopeer2 configuration, and I can retrieve one element at a time, but I can't retrieve the whole XML tree. – amethyst Apr 09 '20 at 05:58
would you mind opening a jira ticket for this issue with all the gory details to recreate it? I can try to find time to move it forward that way, but I'm not the expert here. Maybe we can get more help that way too. Here is jira: https://jira.opendaylight.org/ you may need to first get an LFID so you can log in. That's here: https://identity.linuxfoundation.org/ – jamo Apr 13 '20 at 17:54
Thank you for the suggestion @jamo. I just posted an answer with what I think might be the origin of my problem. Do you think it's still worth opening a Jira ticket? – amethyst Apr 14 '20 at 05:55
I'm not really sure actually. I don't know if order of the elements should matter or not. One thing that seems off to me is your comment about "Master is down" in the original post. I'm not following that. Either way, if you have the time and want to file a jira with the steps for both cases here, I can try to see what I can learn. Something does seem off here. – jamo Apr 15 '20 at 15:54
My guess is that the parsing fails because of the "Duplicate namespace" error, which causes a timeout to expire. Hence the "Master is down", since it's been 5 seconds from when the XML response was received. Knowing the origin of the problem, I find both error messages to be quite deceiving, but maybe it's just me :) – amethyst Apr 16 '20 at 07:52
looks like a ticket was opened for this: https://jira.opendaylight.org/browse/NETCONF-692 – jamo May 27 '20 at 21:19

amethyst · Answer 2 · 2020-04-16T11:01:57.893

I might have stumbled upon something.
I tried the same thing with a different NETCONF server (ConfD) and the same models, and it seems to work fine.
One difference I noticed is that when doing a get-request on TopElement or ParentElement, ConfD returns the child lists ordered, i.e. first all ChildElement1 elements and then all ChildElement2 elements, even though they where created in a random order. Sysrepo-netopeer2, on the other hand, returns the XML tree with the elements in the same order as they were created, so ChildElement1 elements and ChildElement2 elements are mixed (see my example).
I'm not sure if they are both good approaches, or if there is a right one and a wrong one.

EDIT: In my original post I quoted RFC 7950 and somehow missed the relevant part:

The XML elements representing list entries MUST appear in the order specified by the user if the list is "ordered-by user"; otherwise, the order is implementation dependent. The XML elements representing list entries MAY be interleaved with elements for siblings of the list, unless the list defines RPC or action input or output parameters.

So, assuming I am correct in my assumption, it looks like ODL is actually being too restrictive in this case.

score 0 · Answer 3 · answered May 09 '20 at 00:56

I guess the problem here is that the response from the server contains, at the /TopElement/ParentElement , the list entries (in this order):

ChildElement1[id=1]
ChildElement1[id=2]
ChildElement2[id=1]
ChildElement1[id=3]

So I assume both ChildElement1 and ChildElement2 are both lists. The problem here is that the 3 entries of ChildElement1 are not sequential, you have an entry of ChildElement2 in the middle.

I don't see anything in the NETCONF spec saying that that is an issue, but clearly OpenDaylight has a problem with it. I assume OpenDaylight consider the handling of the ChildElement2 entry as meaning that the ChildElement1 list has ended being delcared, so after it finds another entry later, it considers it as a separate list, and generates the exception Duplicate namespace "child-element1" element "ChildElement1" in XML input that you had.

If you want to avoid that problem, I'd recommend to sort the entries in the response from the server, so that entries of the same list stand together.

So to reiterate, the problem is not the order of individual list entries, but the fact that you are interleaving entries belonging to two different lists.

Error "Duplicate namespace in XML input" when retrieving a list over NETCONF using Opendaylight

3 Answers3