How to tracking with bounding boxes that only cover the visible part of objects

Question

I'm using DeepSORT to tracking people. The bounding boxes of objects only cover the visible part so Kalman Filter estimate wrong direction of motion. How can I fix it?

EMA state of Kalman Filter, ignore low confident boxes