Cognitive Services - How to add multiple Faces from a Live Stream to Azure FaceList ? (Python)

Question

Problem Background:

I have created an Azure FaceList and I am using my webcam to capture live feed and:

1. sending the stream to Azure Face Detect
2. getting Face Rectangle returned by Face Detect
3. using the returned Face rectangle to add Face Detected from Live Video Stream to my FaceList.

(I need to create Face List in order to solve the problem I explained in my other question which is answered by Nicolas, which is what I am following)

Problem Details:

According to Azure FaceList documentation at https://learn.microsoft.com/en-us/rest/api/cognitiveservices/face/facelist/addfacefromstream ,if there are multiple faces in the image, we need to specify the target Face to add to Azure FaceList.

The Problem is, What if we need to add all the detected faces (multiple faces) in Face List? Suppose there are 2 or more faces in a single frame of video, then how can I add those two Faces to Face List?

I have tried adding the face rectangles returned from Azure Face Detect into a Python List and then iterating Over List indexes, so that each face Rectangle can be passed to Azure FaceList one-by-one. But no use.

Still getting the error:

There are more than one faces in the image

My Code:

face_list_id = "newtest-face-list"
vid = cv2.VideoCapture(0)


count = 0


face_ids_live_Detected = []  #This list will store faceIds from detected faces
list_of_face_rectangles = []
face_rect_counter=0

while True:
        ret, frame = vid.read()
        check,buffer = cv2.imencode('.jpg', frame)
        img = cv2.imencode('.jpg', frame)[1].tobytes()
        base64_encoded = base64.b64encode(buffer).decode()
        print(type(img))
        detected_faces = utils.detect_face_stream(endpoint=ENDPOINT, key=KEY, image=img,face_attributes=attributes,recognition_model='recognition_03')
        print('Image num {} face detected {}'.format(count, detected_faces))
        count += 1
        color = (255, 0, 0)
        thickness = 2

        for face in detected_faces:
        
            detected_face_id = face['faceId']
            face_ids_live_Detected.append(detected_face_id)
            

            detected_face_rectangle = face['faceRectangle']
            list_of_face_rectangles.append(detected_face_rectangle)
            print("detected rectangle =",detected_face_rectangle)

            face_rect_for_facelist = list_of_face_rectangles[face_rect_counter]
            face_rect_counter +=1

       frame = cv2.rectangle(frame, *utils.get_rectangle(face), color, thickness)
       cv2.imshow('frame', frame)

       for face_id_live in face_ids_live_Detected:
                        similar_faces = face_client.face.find_similar(face_id=face_id_live, face_list_id=face_list_id)                
                        if not similar_faces:
                                print('No similar faces found !')
                                print('Adding Unknown Face to FaceList...')
                                facelist_result = utils.facelist_add(endpoint=ENDPOINT, key=KEY, face_list_id=face_list_id,data=img,params=face_rect_for_facelist)
                                persisted_face_id = facelist_result['persistedFaceId']
                        else:
                                print('Similar Face Found!')
                                for similar_face in similar_faces:
                                        face_id_similar = similar_face.face_id
                                        print("Confidence: "+str(similar_face.confidence))

From my utils file, code for function facelist_add is as follows:

def facelist_add(endpoint, key, face_list_id, data=None, json=None, headers=None,params=None, targetFace=None):
    # pylint: disable=too-many-arguments
    """Universal interface for request."""
    method = 'POST'
    url = endpoint + '/face/v1.0/facelists/'+face_list_id+'/persistedfaces'

    # Make it possible to call only with short name (without BaseUrl).
    if not url.startswith('https://'):
        url = BaseUrl.get() + url

    params={}

    # Setup the headers with default Content-Type and Subscription Key.
    headers = headers or {}
    if 'Content-Type' not in headers:
        headers['Content-Type'] = 'application/octet-stream'
    headers['Ocp-Apim-Subscription-Key'] = key


    params['detectionModel']='detection_03'
    

    response = requests.request(
        method,
        url,
        params=params,
        data=data,
        json=json,
        headers=headers)

    if response.text:
         result = response.json()
    else:
         result = {}

    return result

Answer 1

When you have several faces in a picture, you have to provide a 'targetFace' in your call to AddFace:

A face rectangle to specify the target face to be added into the face list, in the format of "targetFace=left,top,width,height". E.g. "targetFace=10,10,100,100". If there is more than one face in the image, targetFace is required to specify which face to add. No targetFace means there is only one face detected in the entire image.

See API documentation for this method: https://westeurope.dev.cognitive.microsoft.com/docs/services/563879b61984550e40cbbe8d/operations/563879b61984550f30395250

Answer 2

Thanks to everyone who helped and especially Nicolas R. I just found a mistake and corrected it. Now the program is working like Charm.

Actually, Azure 'Face Detect' returns face Rectangle in top,left,width,height sequence which I was feeding directly to the faceList's targetFace.

Now I just swapped the first two values of Face Rectangle and it becomes left,top,width,height which is what the documentation says, and it's working fine now.

Solution:

I have added a new function that takes the faceRectangle dictionary and swap first two values.

list_of_faceRect_valuesonly=[]

def face_rect_values(faceRect_dict):
        temp_list=[]
        for key,value in faceRect_dict.items():
                temp_list.append(value)
        temp_list[0], temp_list[1] = temp_list[1], temp_list[0]
        list_of_faceRect_valuesonly.append(temp_list)

In order to extract values from list, I did following:

face_rect_counter=0
face_rect_for_facelist = list_of_faceRect_valuesonly[face_rect_counter]
face_rect_counter +=1

Request to facelist_add function:

facelist_result = utils.facelist_add(endpoint=ENDPOINT, key=KEY, face_list_id=face_list_id,targetFace=face_rect_for_facelist,data=img)

I have also changed my facelist_add function a little bit:

def facelist_add(endpoint, key, face_list_id, targetFace=[],data=None ,jsondata=None, headers=None):
    # pylint: disable=too-many-arguments
    """Universal interface for request."""
    method = 'POST'
    url = endpoint + '/face/v1.0/facelists/'+face_list_id+'/persistedfaces'

    # Make it possible to call only with short name (without BaseUrl).
    if not url.startswith('https://'):
        url = BaseUrl.get() + url

    params={}
    

    # Setup the headers with default Content-Type and Subscription Key.
    headers = headers or {}
    if 'Content-Type' not in headers:
        headers['Content-Type'] = 'application/octet-stream'
    headers['Ocp-Apim-Subscription-Key'] = key

    list_of_targetfaces  =[]
    list_of_targetfaces.append(targetFace)


    params={'targetFace':json.dumps(targetFace)}
    params = {'targetFace': ','.join(map(str,targetFace))}

    print("Printing TargetFaces(facelist_add function) ...",params['targetFace'])
    

    params['detectionModel']='detection_03'


    url=url + "?"


    response = requests.post(url,params=params,data=data,headers=headers)
    


    print("Request URL: ", response.url)

    result = None

    # Prevent `response.json()` complains about empty response.
    if response.text:
        result = response.json()
    else:
        result = {}

    return result