Well, the difference is that when you remove the object in the scene it is removed from the scene, i.e. is no longer among the children there. Whereas when it's just set to invisible, it still stays in the scene data structure and can be used in calculations for example to rotate some other object towards it.
But yes for the rendering there is no difference in the end, both are ways to omit that object from drawing.
A practically useful difference is that if you need to hide & show objects a lot, setting the visible flag is quick and light whereas manipulating the scene is a bit more complex heavier operation. So to temporarily hide and object that you know you'll show again soon, is a good idea to configure the visibility flag, and to remove an object that you might not bring back anymore better remove it from the scene. Or indeed if you need it for calculations like rotating something towards it (and it perhaps moves in some hierarchy itself).
In order to actually free memory, you need to remove the object from the scene but also dispose the data it is using like shown in e.g. freeing memory in three.js