1

Can someone pls gimme a hint on how to apply nvprof to Kinetica ?

1) I see the name of processes of Kinetica which sits upon GPUs is gpudb_cluster_cuda, and its parent process is gpudb_host_manager. I find gpudb_host_manager is started by /etc/rc.d/init.d/gpudb_host_manager.

2) Thus I modified it as below. This should work - even for its child processes. But it doesn't. No profiling data was produced for gpudb_cluster_cuda.

# vi /etc/rc.d/init.d/gpudb_host_manager
...
#        $START_PROG"$GPUDB_EXE start-host-manager 2>&1 | tee -a ${STARTUP_LOG}; ( exit \${PIPESTATUS[0]} )"
        $START_PROG"/usr/local/cuda-9.2/bin/nvprof --log-file /tmp/nvprof/%p.txt --export-profile /tmp/nvprof/%p.nvvp --print-gpu-trace --profile-child-processes $GPUDB_EXE start-host-manager 2>&1 | tee -a ${STARTUP_LOG}; ( exit \${PIPESTATUS[0]} )"
...

I applied nvprof to /etc/rc.d/init.d/gpudb, and it produces some traces but it does not use GPUs at all.

# vi /etc/rc.d/init.d/gpudb
...
#        $START_PROG"$GPUDB_EXE start 2>&1 | tee -a ${STARTUP_LOG}; ( exit \${PIPESTATUS[0]} )"
       $START_PROG"/usr/local/cuda-9.2/bin/nvprof --log-file /tmp/nvprof/%p.txt --export-profile /tmp/nvprof/%p.nvvp --print-gpu-trace --profile-child-processes $GPUDB_EXE start 2>&1 | tee -a ${STARTUP_LOG}; ( exit \${PIPESTATUS[0]} )"
...

Of course, I stopped and restarted these. Any comment would be welcome.

james-see
  • 12,210
  • 6
  • 40
  • 47
nasica88
  • 1,185
  • 10
  • 10

1 Answers1

2

I found that nvprof can be applied by editing /opt/gpudb/core/bin/gpudb as below.

[root@localhost ~]# vi /opt/gpudb/core/bin/gpudb
...
#    nohup $HOST_MANAGER_CMD >> $HOST_MANAGER_LOG_FILENAME 2>&1 &
    nohup /usr/local/cuda-9.2/bin/nvprof --log-file /tmp/nvprof/%p.txt --export-profile /tmp/nvprof/%p.nvvp --print-gpu-trace --profile-child-processes $HOST_MANAGER_CMD >> $HOST_MANAGER_LOG_FILENAME 2>&1 &
...
nasica88
  • 1,185
  • 10
  • 10