5

I am using dlib's train_object_detector for face detection and I have roughly 6k images in a folder with which I am trying to train my model.

Also, I am using dlib's example python code(train_object_detector.py) for this purpose.

But the thing is, the program's RAM usage is insane. For roughly 300 images, it required approximately 15GB RAM and right now with my 6k images, I'm stuck.

For 6k images, while training, it required more than 100GBs of RAM and eventually program killed itself.

Has it always been like this? or am I doing something wrong? Is it normal to have this much of RAM usage?

It is almost not modified at all and pretty much same with the example code from dlib.

Note: The sizes of the images are between 10-100 KB.

Here is the code I'm using (remote): http://pastebin.com/WipU8qgq Here's the code:

import os
import sys
import glob
import dlib
from skimage import io


if len(sys.argv) != 4:
        print(
        "Give the path to the faces directory as the argument to this "
        "program with training and test xml files in order. For example: \n"
        "    ./train_object_detector_modified.py ../faces ../faces/training.xml ../faces/testing.xml")
    exit()
faces_folder = sys.argv[1]
training_xml_path = sys.argv[2]
testing_xml_path = sys.argv[3]

options = dlib.simple_object_detector_training_options()
options.add_left_right_image_flips = True
options.C = 5
options.num_threads = 8
options.be_verbose = True

dlib.train_simple_object_detector(training_xml_path, "detector.svm", options)
print 'training end'

print("")  # Print blank line to create gap from previous output
print("Training accuracy: {}".format(
    dlib.test_simple_object_detector(training_xml_path, "detector.svm")))

print("Testing accuracy: {}".format(
    dlib.test_simple_object_detector(testing_xml_path, "detector.svm")))


'''
# Now let's use the detector as you would in a normal application.  First we
# will load it from disk.
detector = dlib.simple_object_detector("detector.svm")

# We can look at the HOG filter we learned.  It should look like a face.  Neat!
win_det = dlib.image_window()
win_det.set_image(detector)

# Now let's run the detector over the images in the faces folder and display the
# results.
print("Showing detections on the images in the faces folder...")
win = dlib.image_window()
for f in glob.glob(os.path.join(faces_folder, "*.jpg")):
   print("Processing file: {}".format(f))
   img = io.imread(f)
   dets = detector(img)
   print("Number of faces detected: {}".format(len(dets)))
   for k, d in enumerate(dets):
       print("Detection {}: Left: {} Top: {} Right: {} Bottom: {}".format(
           k, d.left(), d.top(), d.right(), d.bottom()))

   win.clear_overlay()
   win.set_image(img)
   win.add_overlay(dets)
   dlib.hit_enter_to_continue()
'''
ABuyukcakir
  • 53
  • 6

1 Answers1

2

This is happening because you have a combination of big images and/or small bounding boxes. By default, dlib.train_simple_object_detector uses a detection window that is 6400 pixels in size. If images contain target boxes much smaller than this then those images are upsampled to make the objects big enough.

All these settings are fields in the options object.

Davis King
  • 4,731
  • 1
  • 25
  • 26
  • Alright, thanks for the quick answer, Mr. King. I will regularize all my images by either rescaling or removing the exceptions, and will put that version's RAM usage for the record, if anyone's interested. – ABuyukcakir Aug 18 '16 at 11:45
  • 2
    By the way, my detector failed to classify faces. I suppose it was because the images I have had multiple orientations and similar to what has been discussed here http://stackoverflow.com/questions/38349615/traning-dlib-object-detector-with-450k-instances, i might need to cluster. now, my question is, how much ram usage do you normally expect for training this dataset you used , namely, dlib_face_detector_training_data (mentioned in above link)? It is around ~45GB for me right now. Is this normal @DavisKing? – ABuyukcakir Aug 22 '16 at 10:47
  • @Davis King let me allow to ask you a humble question. In the options I didn't find any facility to train the model in batches. But if it was possible these sorts of memory overflow problems could be easily handled. Why you didn't provide this facility here? Is there any potential problems there if we trained the model using mini-batches? If not then how we can accomplish this? – hafiz031 Aug 08 '20 at 06:00
  • There is no sort of mini-batch mode. That isn't a thing that makes sense for this algorithm. – Davis King Aug 08 '20 at 12:51