How should i debug a Microsoft CIM that fails because MSFT_NetRoute be "not found"?

Question

In general, how should one debug a failure related to CmdletizationQuery_NotFound messages, and why would the MSFT_NetRoute based queries register such an error?

Details

Specifically, my Kubernetes CNI provider (antrea) is posting an error message related to MSFT_NetRoute when it starts up, because it can't succesfully run a powershell query necessary for bootstrapping itself. The error message has a suspicious string, which:

• Might be related to powershell version (I disconfirmed later in this post, but its a reasonable hypothesis)
• Im also considering that maybe this error came from external things i havent installed...
• Finally, without fully understanding the nature of cmdletizationQuery error messages, its reasonable to assume this is related to a lower level networking setup issue.

   + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetRoute

The above snippet of this message, highlights the Get-NetRoute Related error which occurs below, which is puzzling because it seems to be indicating that Get-NetRoute is not a valid query to run. But https://docs.microsoft.com/en-us/powershell/module/nettcpip/get-netroute?view=windowsserver2019-ps seems to imply thet Get-NetRoute is available by default in powershell.

The error message

ROOT/StandardCimv2/MSFT_NetRoute class on the  CIM server: SELECT * FROM MSFT_NetRoute  WHERE ((DestinationPrefix LIKE
'0.0.0.0/0')) AND ((InterfaceIndex = 26)). Verify query parameters and retry.
At line:1 char:3
+ $(Get-NetRoute -InterfaceIndex 26 -DestinationPrefix 0.0.0.0/0 ).Next ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (MSFT_NetRoute:String) [Get-NetRoute], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetRoute

Powershell version

The powershell version on this machine is seemlingy pretty up to date... (5.1 i think is normal on windows server 2019)...

Name                           Value
----                           -----
PSVersion                      5.1.17763.1852
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.1852
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Note: I've tested this in similar versions of windows server 2019 with a similar 5.1.x powershell version (5.1.17763.1490) and not seen this failure. So,

Questions

• is there something i need to do to make Get-NetRoute work properly on my machines?
• is this related to something else (i.e. the way network interfaces are configured on the individual VM) ? Note that this is a VirtualBox VM, so if there was a issue with bridged or NAT networking, I woudln't be totally shocked.

Answer 1

So, in general, if you get a CmdletizationQuery_NotFound error, may not always be accompanied by info about the failing field if there are many fields in the query...

➜  sig-windows-dev-tools git:(antrea-node-ip-hardcoding) ✗ vagrant winrm winw1 --shell=powershell --command="Get-NetRoute -InterfaceIndex 123 "
No MSFT_NetRoute objects found with property 'InterfaceIndex' equal to '123'.  Verify the value of the property and retry.
At line:1 char:1
+ Get-NetRoute -InterfaceIndex 123
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (123:UInt32) [Get-NetRoute], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_InterfaceIndex,Get-NetRoute

It is because the specific CIMs query failed, and thus the object you were looking for wasnt found.

However, it appears that sometimes a CIMS query will fail to return a result without giving you the specific part of the query which failed (i.e. CmdletizationQuery_NotFound_InterfaceIndex above was nice and easy to read, but the same query below when we add the DestinationPrefix search field, gives a more cryptic error message)...

➜  sig-windows-dev-tools git:(antrea-node-ip-hardcoding) ✗ vagrant winrm winw1 --shell=powershell --command="Get-NetRoute -InterfaceIndex 7 -DestinationPrefix 0.0.0.0/1"
No matching MSFT_NetRoute objects found by CIM query for instances of the ROOT/StandardCimv2/MSFT_NetRoute class on the  CIM server: SELECT * FROM MSFT_NetRoute  WHERE ((DestinationPrefix LIKE '0.0.0.0/1')) AND ((InterfaceIndex = 7)). Verify query parameters and retry.
At line:1 char:1
+ Get-NetRoute -InterfaceIndex 7 -DestinationPrefix 0.0.0.0/1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (MSFT_NetRoute:String) [Get-NetRoute], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetRoute

In this specific case:

• there was no existing IP interface with id = 26 which ALSO had the 0.0.0.0/0 destination prefix

In more general cases, if you get a similar error, it could be caused by the fact that your just crafting a CIMS query that is not able to return any data.

Now, since the original question was related to a Kubernetes CNI provider, I'll address that part:

In Kubernetes on Windows

CNI providers such as antrea need this information when coming online, and the general thing to make sure of is the fact that your windows kubelet is setting its IP address correctly (i.e. via the node-ip field on startup).

In my case, I found that after setting this, this query was generated correctly, and it started looking at the right interface for this value (i.e. the one that corresponded to my Node's internal IP address).

There is the broader question of how, in general, DestinationPrefixes should be setup in Kubernetes windows VMs, but that is outside the scope of my original question, but in general, if you have setup your network correctly such that:

• your node-ip as shown by kubectl get nodes -o wide is the correct one that you want for your windows kubelet and
• that node-ip is associated w/ an interface which has a destination prefix IP address = 0.0.0.0/0

Then specifically the antrea CNI provider will be able to accurately determine the correct nextHop for its gateway, which ultimately is used to configure the OVS routing rules on a node for your pod networks.