0

I am using a Jetson Xavier NXto capture frames from a USB camera using OpenCV. Then I implement a KCF tracker on the captured frames to track a certain object. In parallel I am sending the tracking data to an ESP32 capturing IMU data from an MPU6050. I can see a lot of inconsistency between when the tracker recorded movement to when the sensor recorded movement. sometimes it is 70 millisecond difference, and sometimes it even gets up to 150 millisecond difference. that much difference is really messing up my plans, even that 70 millisecond difference seems too much considering i am running on 60fps camera, that amounts to almost 5 frames delay. I checked the network delay with a scope, it is about 3 millisecond consistent using UART communication. Here's how I am grabbing frames:

video.open(
  "v4l2src device=/dev/video0 queue-size=1 ! video/x-raw,width=640,height=480 ! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), 
  format = I420 ! nvvidconv ! video/x-raw,format=BGRx,width=(int)640, height=(int)480, framerate=(fraction)60/1! 
  videoconvert !video/x-raw,format=BGR ! appsink drop=True sync=False"
);

I use cv::VideoCapture::read() to capture frames.

the IMU sensor is working at 200hz.

Can anyone explain both the delay in measurements, meaning between the sensor and the tracker, and the difference between measured delay, ranging from 70 to 150 milliseconds?

Bilal
  • 3,191
  • 4
  • 21
  • 49
KTBM
  • 13
  • 1
  • 7
  • Do you know the bandwidth of the data stream? And what's the image processing time? Running the KCF tracker at 60 Hz means the image processing time should be less than 16.7 milliseconds per frame. – Kani Dec 22 '22 at 13:37
  • I am not sure about the bandwidth but I am using USB3.0 to grab the frames so it shouldn't really be an issue unless i am missing something. The processing time is shorter than the image acquisition time, the tracker takes about 2-10 milisec to run depending on the amount of pixels i track. sometimes the program is stuck on read() for more than 16miliseconds which i can't explain why, but the tracker is usually good with time. – KTBM Dec 25 '22 at 12:33
  • It is not clear to me, but how do you use a UART connection over the network? You mentioned both UART delay and network delay, how do you send data from the NX to the ESP32? – Kani Dec 25 '22 at 13:13
  • Yes sorry, i meant i am using UART communication to transfer the data, adding network to it was a mistake. it is simply serial UART communication which i calculated the delay of by raising a GPIO from the jetson when i send a message, and doing the same on the ESP when getting the message, got less than 5ms – KTBM Dec 25 '22 at 15:03
  • I see, then your data transfer bandwidth bottleneck is going to be the baud rate of the serial port between the NX and the ESP32. If you are not hitting that limit, I am not sure what to look for. Also, sending a single image for measure the time and sending a continuous steam of data can very different. – Kani Dec 25 '22 at 15:30

0 Answers0