First of all my agenda is to be able to use spark codes inside jupyterhub. In other words I want to connect a remote spark cluster to jupyterhub. After searching about it I came up with two solutions:1)Livy and 2)spark magic. I have tried Livy but since it doesn't support spark version 3 I have put that one aside, also another way we considered was via spark magic but we couldn't install or even find a good documentation about it.
what came into our minds was somehow merge the two image of spark and jupyterhub together to gain what we need. Does anyone have any idea how it can be done or a better suggestion? Even a good documentation about spark magic that we can use would be wonderful. thank you for your help.
