Faster R-CNN object detection and deep-sort tracking algorithm integration

Question

I have been trying to integrate the Faster R-CNN object detection model with a deep-sort tracking algorithm. However, for some reason, the tracking algorithm does not perform well which means tracking ID just keeps increasing for the same person.

I have used this repository for building my own script. (check demo.py) deep-sort yolov3

What I did:

1. 1 detection every 30 frames

2. created a list for detection scores

3. created a list for detection bounding boxes (considering the input format of deep-sort)

4. calling the tracker !!!

                # tracking and draw bounding boxes
            for i in range(0, len(refine_person_detection)):
   
                confidence_worker.append(refine_person_detection[i][4]) # scores
                bboxes.append([refine_person_detection[i][0], refine_person_detection[i][2],
                               (refine_person_detection[i][1] - refine_person_detection[i][0]),
                               (refine_person_detection[i][3] - refine_person_detection[i][2])]) # bounding boxes
   
                features = encoder(frame, bboxes)
   
                detections = [Detection(bbox, confidence, feature) for bbox, confidence, feature in
                              zip(bboxes, confidence_worker, features)]
   
                boxes = np.array([d.tlwh for d in detections])
                scores = np.array([d.confidence for d in detections])
                indices = preprocessing.non_max_suppression(boxes, nms_max_overlap, scores)
                detections = [detections[i] for i in indices]
   
                tracker.predict() # calling the tracker
                tracker.update(detections)
   
                for track in tracker.tracks:
                    k.append(track)
                    if not track.is_confirmed() or track.time_since_update > 1:
                        continue
                    bbox = track.to_tlbr()
                    cv2.rectangle(frame, (int(bbox[0]), int(bbox[1])), (int(bbox[2]), int(bbox[3])),
                                  (255, 255, 255), 2)
                    cv2.putText(frame, str(track.track_id), (int(bbox[0]), int(bbox[1])), 0, 5e-3 * 200,
                                (0, 255, 0), 2)
   

Here is an example of bad results that tracking ID increases.

Thanks in advance for any suggestion

Answer 1

I also study the same thing, I try to combine them, too. Have you done it yet, any progress?

Answer 2

The provided code it correct. However, the detection must be done every frame. Since the deep-sort uses the features within the bounding box for tracking, having a gap between the detection frames caused the issue of increasing numbers for the same person

P.S: @Mustafa please check the code above with every frame detection, should work. feel free to comment if it did not