Inquiry on Slowing Down the Graph's Operation Speed in MediaPipe

Question

I'm working on a feature that utilizes face_mesh_cpu. However, my inquiry is not limited to this particular use-case. In face_mesh_cpu, a FlowLimiterCalculator is used to control the flow of the graph. The FlowLimiterCalculator discards the input_stream until a specified output is produced, and then resumes accepting the input_stream when a stream specified as a back_edge is output. This seems to work stably on most systems.

However, I want to add one condition here: to limit the maximum number of operations per second. Initially, I added a sleep function to the calculators included in face_mesh to make it operate intermittently, but it seemed to compromise accuracy.

Upon further research, I found PacketResamplerCalculator and PacketThinnerCalculator. PacketResamplerCalculator adds or removes frames to operate at a specified FPS, while PacketThinnerCalculator only removes frames to operate at a specified FPS.

I modified the graph to have one of these calculators receive the input_stream from the webcam and send the output_stream to FlowLimiterCalculator.

I tried both, and here's what I observed:

In the case of PacketResamplerCalculator, lowering the frame_rate value seemed to work fine. However, raising the frame_rate value resulted in the output video, including annotations, being slower than the actual webcam video by a factor of 0.x.
In the case of PacketThinnerCalculator, regardless of the period value set, the output video, including annotations, was slower than the actual webcam video by a factor of 0.x.

For the first setting, I modified the graph as follows: I added PacketResamplerCalculator and modified FlowLimiterCalculator. FaceLandmarkFrontCpu and FaceRendererCpu receive throttled_input_video, but there are no modifications on that end, so I'll omit that part.

    node {
      calculator: "PacketResamplerCalculator"
      input_stream: "input_video"
      output_stream: "resampled_video"
      node_options: {
        [type.googleapis.com/mediapipe.PacketResamplerCalculatorOptions] {
          frame_rate: 15
          #allow_legacy_timestamp: true
        }
      }
    }

    node {
      calculator: "FlowLimiterCalculator"
      #input_stream: "input_video"
      input_stream: "resampled_video"
      #input_stream: "video_frames_scaled_downsampled"
      input_stream: "FINISHED:output_video"
      input_stream_info: {
        tag_index: "FINISHED"
        back_edge: true
      }
      output_stream: "throttled_input_video"
    }

The second setting was configured as follows for the graph.

    node {
      calculator: "PacketThinnerCalculator"
      input_stream: "input_video"
      output_stream: "video_frames_scaled_downsampled"
      options: {
        [mediapipe.PacketThinnerCalculatorOptions.ext]: {
          thinner_type: ASYNC
          period: 100
        }
      }
    }

    node {
      calculator: "FlowLimiterCalculator"
      #input_stream: "input_video"
      #input_stream: "resampled_video"
      input_stream: "video_frames_scaled_downsampled"
      input_stream: "FINISHED:output_video"
      input_stream_info: {
        tag_index: "FINISHED"
        back_edge: true
      }
      output_stream: "throttled_input_video"
    }

I thought that adding PacketThinnerCalculator in front of FlowLimiterCalculator would limit the maximum FPS while preventing issues even when the system performance is very poor. However, I'm currently using an M1 Max with 32GB of RAM, and I don't understand why there's a delay in the output screen with that setting. Am I missing something?

Limiting the maximum FPS of the input being passed through FlowLimiter while maintaining real-time performance on the Mediapipe output screen.

Inquiry on Slowing Down the Graph's Operation Speed in MediaPipe

0 Answers0