Issues with running a second node on another processor

Question

I am trying to run a second node on a different processor, either an ARM or a second x86_64. I have a DomMgr running on one x86_64 and attempting to start a node on either another x86_64 or arm using nodeBooter. The DevMgr starts and registers with the DomMgr but when it starts the GPP device it "Requesting IDM CHANNEL IDM CHANNEL IDM_CHANNEL" and then immediately "terminate called after throwing an instance of 'CORBA::OBJECT_NOT_EXIST'". The DomMgr printed out to the console that "Domain Channel: IDM_Channel created". Is it supposed to register that in the NameService or why does the remote DevMgr get an invalid object ref when it tries to get it?

I did not realize I could clarify my question by editing it to add new findings. I'll do that from now on.

By using ORBtraceLevel on the remote DevMgr I found that I had different problem on my remote x86-based DevMgr and my ARM-based one, even though the normal error messages were the same. The x86 case was simply that I my exported DevMgr dcd used the same name and id as one running locally on the Domain. When I fixed that I have no problem with the x86-based remote DevMgr starting its GPP device and registering.

But this is NOT the problem for the ARM-based case. With traceLevel=10 I started DevMgr on both my x86 successfully and my ARM and compared the outputs. First I should mention that my ARM is running Ubuntu 16.04 on a RaspberryPi 3. The cpu is 64-bit but no distro for either Ubuntu or CentOS is available as 64-bit so the OS is 32-bit Ubuntu for now. I know that RedHawk 2.0 says it only now supports 64-bit CentOS so perhaps that is the problem, although I was able to build RedHawk with no trouble and most of it works fine. But trace does show two warnings

WARN Device_impl:172 - Cannot set allocation implementation: Property ### is 
of type 'CORBA::Long' (not 'int')

which do not show in the x86 case and I believe are due to the different sizes of int. If I do not start an Event Service on the domain, these same warnings show but I am able to start the GPP fine and run waveforms. So I do not know if this is related to my OBJECT_NOT_FOUND error in GPP or not but thought I should mention it.

Trace shows one successful

Creating ref to remote: REDHAWK.DEV.IDM.Channel
target id   :IDL:omg.org/CosEventChannelAdmin/EventChannel:1.0
most derived id:
Adding root/Files<3> (activating) to object table.

but on the second case it immedately shows

Adding root<3> (activating) to object table.

followed by

throw OBJECT_NOT_EXIST from GIOP_C.cc:281 (NO,OBJECT_NOT_EXIST_NoMatch)
throw OBJECT_NOT_EXIST from omniOrbRef.cc:829 (NO,OBJECT_NOT_EXIST_NoMatch)

and then GPP terminates with signal 6.

The successful x86 trace shows the same Creating ref and Adding root<3> but then has

Creating ref to remote: root/REDHAWK_DEV.IDM_Channel <...>

Can this be related to the 32-bit vs 64-bit or why would this happen only on the ARM based GPP?

Note that I have iptables accepting any traffic from my subdomain on x86s and is not running at all on the ARM. There is a lot of successful connections including queries with nameclt, so this is not (as far as I can tell) a network connection issue.

Answer 1

What version of REDHAWK are you running? What OS? Can you provide a list of all the omni rpms you have installed on your machine?

Answer 2

It sounds like something is miss-configured on your system, perhaps IPTables or selinux? Lets walk through a quick example to show the minimum needed configuration and running processes needed for a multi-node system. If this does not clear things up, I'd suggest rerunning the domain and device manager with TRACE level debugging enabled and examine the output for any anomalies or disable selinux and iptables temporarily to rule them out as issues.

I'll use a REDHAWK 2.0.1 docker image as a tool to walk through the example. The installation steps used to build this image can be found here.

• First we'll drop into a a REDHAWK 2.0.1 environment with nothing running and label this container as our domain manager

[youssef@axios(0) docker-redhawk]$docker run -it --name=domainMgr axios/redhawk:2.0

• Let's confirm that almost nothing is running on this container

[redhawk@ce4df2ff20e4 ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
redhawk      1     0  0 12:55 ?        00:00:00 /bin/bash -l
redhawk     27     1  0 12:57 ?        00:00:00 ps -ef

• Lets take a look at the current omniORB configuration file. This will be the box we run omniNames, omniEvents and the domain manager.

[redhawk@ce4df2ff20e4 ~]$ cat /etc/omniORB.cfg 
InitRef = NameService=corbaname::127.0.0.1:2809
supportBootstrapAgent = 1
InitRef = EventService=corbaloc::127.0.0.1:11169/omniEvents

• Since this will be the machine we are running omniNames and omniEvents on, the loopback address (127.0.0.1) is fine however other machines will need to reference this machine either by its hostname (domainMgr) or it's IP address so we can note it's IP now.

[redhawk@ce4df2ff20e4 ~]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:0E  
          inet addr:172.17.0.14  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:468 (468.0 b)  TX bytes:558 (558.0 b)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

• Note it only has a single interface so we do not need to specify an endPoint. However specifying the unix socket endpoint would provide a performance boost for any locally running components.

• We can now startup omniNames, omniEvents, and the domain manager and after each step see what is running. The "extra operand" output on omniNames is expected on newer versions of CentOS6 and is an issue with the omniNames init script.

[redhawk@ce4df2ff20e4 ~]$ sudo service omniNames start
Starting omniNames: /usr/bin/dirname: extra operand `2>&1'
Try `/usr/bin/dirname --help' for more information.
                                                           [  OK  ]

[redhawk@ce4df2ff20e4 ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
redhawk      1     0  0 12:55 ?        00:00:00 /bin/bash -l
omniORB     50     1  0 13:01 ?        00:00:00 /usr/bin/omniNames -start -always -logdir /var/log/omniORB/ -errlog /var/log/omniORB/error.log
redhawk     53     1  0 13:01 ?        00:00:00 ps -ef

[redhawk@ce4df2ff20e4 ~]$ sudo service omniEvents start
Starting omniEvents                                        [  OK  ]

[redhawk@ce4df2ff20e4 ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
redhawk      1     0  0 12:55 ?        00:00:00 /bin/bash -l
omniORB     50     1  0 13:01 ?        00:00:00 /usr/bin/omniNames -start -always -logdir /var/log/omniORB/ -errlog /var/log/omniORB/error.log
root        69     1  0 13:01 ?        00:00:00 /usr/sbin/omniEvents -P /var/run/omniEvents.pid -l /var/lib/omniEvents -p 11169
redhawk     79     1  0 13:01 ?        00:00:00 ps -ef

• I'm going to start up the domain manager in the foreground and grab the output of ps -ef via a "docker exec domainMgr ps -ef" in a different terminal

[redhawk@ce4df2ff20e4 ~]$ nodeBooter -D
2016-06-22 13:03:21 INFO  DomainManager:257 - Loading DEFAULT logging configuration. 
2016-06-22 13:03:21 INFO  DomainManager:368 - Starting Domain Manager
2016-06-22 13:03:21 INFO  DomainManager_impl:208 - Domain Channel: ODM_Channel created.
2016-06-22 13:03:21 INFO  DomainManager_impl:225 - Domain Channel: IDM_Channel created.
2016-06-22 13:03:21 INFO  DomainManager:455 - Starting ORB!

[youssef@axios(0) docker-redhawk]$docker exec domainMgr ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
redhawk      1     0  0 12:55 ?        00:00:00 /bin/bash -l
omniORB     50     1  0 13:01 ?        00:00:00 /usr/bin/omniNames -start -always -logdir /var/log/omniORB/ -errlog /var/log/omniORB/error.log
root        69     1  0 13:01 ?        00:00:00 /usr/sbin/omniEvents -P /var/run/omniEvents.pid -l /var/lib/omniEvents -p 11169
redhawk     80     1  0 13:03 ?        00:00:00 DomainManager DEBUG_LEVEL 3 DMD_FILE /domain/DomainManager.dmd.xml DOMAIN_NAME REDHAWK_DEV FORCE_REBIND false PERSISTENCE true SDRROOT /var/redhawk/sdr
redhawk     93     0  1 13:03 ?        00:00:00 ps -ef

• So we can see that we have omniNames, omniEvents, and the DomainManager binaries running. Time to move on to a new node for the device manager.

• In a new terminal I create a new container and call it deviceManager

[youssef@axios(0) docker-redhawk]$docker run -it --name=deviceManager axios/redhawk:2.0

• Confirm nothing is really running, then take a look at the omniORB configuration file.

[redhawk@765ce325f145 ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
redhawk      1     0  0 13:05 ?        00:00:00 /bin/bash -l
redhawk     28     1  0 13:06 ?        00:00:00 ps -ef

[redhawk@765ce325f145 ~]$ cat /etc/omniORB.cfg 
InitRef = NameService=corbaname::127.0.0.1:2809
supportBootstrapAgent = 1
InitRef = EventService=corbaloc::127.0.0.1:11169/omniEvents

• We need to change where the NameService and EventService IPs are pointing to either our domain managers hostname (domainMgr) or IP address (172.17.0.14) I will go with IP address.

[redhawk@765ce325f145 ~]$ sudo sed -i 's,127.0.0.1,172.17.0.14,g' /etc/omniORB.cfg

[redhawk@765ce325f145 ~]$ cat /etc/omniORB.cfg 
InitRef = NameService=corbaname::172.17.0.14:2809
supportBootstrapAgent = 1
InitRef = EventService=corbaloc::172.17.0.14:11169/omniEvents

• We can confirm this worked using nameclt list to show the entry in omniNames of the event channel factory and the domain.

[redhawk@765ce325f145 ~]$ nameclt list
EventChannelFactory
REDHAWK_DEV/

• Finally we can start up the device manager and inspect the running processes in a new shell via "docker exec deviceManager ps -ef"

[redhawk@765ce325f145 ~]$ nodeBooter -d /var/redhawk/sdr/dev/nodes/DevMgr_12ef887a9000/DeviceManager.dcd.xml 
2016-06-22 13:09:09 INFO  DeviceManager:446 - Starting Device Manager with /nodes/DevMgr_12ef887a9000/DeviceManager.dcd.xml
2016-06-22 13:09:09 INFO  DeviceManager_impl:367 - Connecting to Domain Manager REDHAWK_DEV/REDHAWK_DEV
2016-06-22 13:09:09 INFO  DeviceManager:494 - Starting ORB!
2016-06-22 13:09:09 INFO  Device:995 - DEV-ID:DCE:c5029226-ce70-48d9-9533-e025fb9c2a34 Requesting IDM CHANNEL IDM_Channel
2016-06-22 13:09:09 INFO  redhawk::events::Manager:573 - PUBLISHER - Channel:IDM_Channel Reg-Id21f4e766-c5c6-4c5b-8974-337736e71f87 RESOURCE:DCE:c5029226-ce70-48d9-9533-e025fb9c2a34
2016-06-22 13:09:09 INFO  DeviceManager_impl:1865 - Registering device GPP_12ef887a9000 on Device Manager DevMgr_12ef887a9000
2016-06-22 13:09:09 INFO  DeviceManager_impl:1907 - Device LABEL: GPP_12ef887a9000  SPD loaded: GPP' - 'DCE:4e20362c-4442-4656-af6d-aedaaf13b275
2016-06-22 13:09:09 INFO  GPP:658 - initialize()
2016-06-22 13:09:09 INFO  redhawk::events::Manager:626 - SUBSCRIBER - Channel:ODM_Channel Reg-Id0d18c1f4-71bf-42c2-9a2d-416f16af9fcf resource:DCE:c5029226-ce70-48d9-9533-e025fb9c2a34
2016-06-22 13:09:09 INFO  GPP_i:679 - Component Output Redirection is DISABLED.
2016-06-22 13:09:09 INFO  GPP:1611 - Affinity Disable State,  disabled=1
2016-06-22 13:09:09 INFO  GPP:1613 - Disabling affinity processing requests.
2016-06-22 13:09:09 INFO  GPP_i:571 - SOCKET CPUS USER    SYSTEM  IDLE    

2016-06-22 13:09:09 INFO  GPP_i:577 - 0      8    0.00    0.00    0.00    
2016-06-22 13:09:09 INFO  GPP:616 -  initialize CPU Montior --- wl size 8
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (docker0)
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (em1)
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (lo)
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (tun0)
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (vboxnet0)
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (veth70de860)
2016-06-22 13:09:10 INFO  GPP_i:602 - initializeNetworkMonitor: Adding interface (vethd0227d6)
2016-06-22 13:09:10 INFO  DeviceManager_impl:2087 - Registering device GPP_12ef887a9000 on Domain Manager

[youssef@axios(0) docker-redhawk]$docker exec deviceManager ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
redhawk      1     0  0 13:05 ?        00:00:00 /bin/bash -l
redhawk     35     1  0 13:09 ?        00:00:00 DeviceManager DCD_FILE /nodes/DevMgr_12ef887a9000/DeviceManager.dcd.xml DEBUG_LEVEL 3 DOMAIN_NAME REDHAWK_DEV SDRCACHE /var/redhawk/sdr/dev SDRROOT /var/redhawk/sdr
redhawk     40    35  1 13:09 ?        00:00:00 /var/redhawk/sdr/dev/devices/GPP/cpp/GPP PROFILE_NAME /devices/GPP/GPP.spd.xml DEVICE_ID DCE:c5029226-ce70-48d9-9533-e025fb9c2a34 DEVICE_LABEL GPP_12ef887a9000 DEBUG_LEVEL 3 DOM_PATH REDHAWK_DEV/DevMgr_12ef887a9000 DCE:218e612c-71a7-4a73-92b6-bf70959aec45 False DCE:3bf07b37-0c00-4e2a-8275-52bd4e391f07 1.0 DCE:442d5014-2284-4f46-86ae-ce17e0749da0 0 DCE:4e416acc-3144-47eb-9e38-97f1d24f7700  DCE:5a41c2d3-5b68-4530-b0c4-ae98c26c77ec 0 DEVICE_MGR_IOR IOR:010000001900000049444c3a43462f4465766963654d616e616765723a312e3000000000010000000000000070000000010102000c0000003137322e31372e302e313500a49e00001c000000ff4465766963654d616e61676572fef58d6a570100002300000000000200000000000000080000000100000000545441010000001c00000001000000010001000100000001000105090101000100000009010100
redhawk    398     0  0 13:09 ?        00:00:00 ps -ef

So we've successfully spun up two machines on the same network with unique IP addresses, designated one as the domain manager, omniNames, and omniEvents server and the other as a Device Manager / GPP node. At this point, we could connect to the domain manager either via the IDE or through a python interface and launch waveforms; we would expect these waveforms to launch on the sole device manager node.