0

Not sure where to ask this. Please redirect me if SO is not the place.

I want make a web app that accurately tracks pose in a stationary video of someone pedaling a stationary bike. The joints can be marked with some stickers to make the process easier and more accurate. Basically, I want to do what does this app.

First i tried markerless tracking using pose estimation models such as mediapipe's Blazepose and google's MoveNet. However, these are not accurate enough. I would also like to track some additional landmarks (ball of the foot,...).

Then I tried OpenCV.js's Lukas-Kanade optical flow method. But the algorithm lost the tracked point quickly. Even when i placed a colored tape on the part of the body that i wanted to track.

I also tried template matching a single marked point in opencv but it was not very robust, and it would probably not work well when using more markers.

What other methods can I try? Since the app i send the video of requires stickers to be placed, I though it is using something like Lukas-Kanade. But as I said, when I tried it, it wasn't able to track the marked point. Because the app is only on iOS I thought it may be using this API. However, this is only my speculation.

Edit: added example video: https://www.youtube.com/watch?v=eCNyyABfWSE I tried shooting in slowmo to have more fps, but the quality suffered because of this. Also i didn't have blue or green tape so I had to use yellow, which is not very visible on the sweater or on my wrist. But the markers on the pants should be trackable right?

KarelPrdel
  • 99
  • 1
  • 1
  • 7
  • input data please (video). -- all those methods you suggested are ill suited to tracking retroreflective markers, which is the type of marker you _should_ be using (and is probably used in the video of that other app you show). this just needs some simple blob detection based on color and brightness, and then tracking by association, picking the nearest new detection as a continuation of the track. -- research this some more, until you find the materials that say this. – Christoph Rackwitz Feb 03 '23 at 22:34
  • Thanks, I'll have a look at simple blob detection. I think it may work when used with some bright sticker color (the video does not mention reflective stickers, but says the color should stand out). – KarelPrdel Feb 03 '23 at 23:29
  • @KarelPrdel Please provide a testable video file (link) to advise about. Even a short clip like 3 seconds is still giving us 90 frames (at 30 FPS) to discuss possible solution(s). If providing a test video clip, also please resize it to a 480p or 720p resolution (_eg:_ Don't be that guy that just dumps a 32-bit 8K clip at 1 gigabyte per second and calls it _"for testing purposes"_, just saying before you upload anything that might be un-usable to others. Keep it a simple small file with markers visible). – VC.One Feb 05 '23 at 15:25
  • I, for one, am happy with 2K video. too little resolution and objects get harder to identify. here's a screenshot from the youtube video: https://i.stack.imgur.com/83jBW.jpg looks like they already processed that or detected the markers. the green dots look too regular. – Christoph Rackwitz Feb 05 '23 at 17:37
  • I believe your high frame rate video is suitable for tracking. just make the user place tracking points on all those markers, and then some very simple detection approaches should be able to stay on those markers and follow. – Christoph Rackwitz Feb 06 '23 at 23:26
  • @ChristophRackwitz Thanks! For detection I want to try some kind of color thresholding, than grouping the thresholded pixels that are near the selected joints into keypoints, and then tracking these keypoints using something like meanshift. Or is there a better approach? – KarelPrdel Feb 07 '23 at 09:36

0 Answers0