I think it has something to do with trying to fix an object's collision. I have a box collider on a cube, and when I pick it up, sometimes I go zooming around (even can use it to move vertically!). It's annoying and not sure how to fix this.
Upon letting go of the object, there are no more issues.