I noticed that there are some cloud providers that offer a managed solution for kubernetes, I'm wondering what are the various components involved in this offering.
My gut feeling is telling me there would be some components involved, like, NetworkPolicy, OPA, but I can't wrap my head on how should be the overall project overview, since the API are the same for every user of the cluster.
Are they deploying various control planes or are they deploying a separate control plane for each user?
I presume you are referring to managed solutions like GKE, EKS, and AKS in the major cloud providers. Each cloud provider has a different "secret sauce" for how they manage the cluster control planes, but the common theme is that the control plane (kube-apiserver, etcd, kube-controller-manager, kubelet, etc.) is hidden away from you as the cluster owner.
From your perspective as the creator/owner of one of these clusters, it's an entirely isolated cluster. You get your own IP address for your own kubernetes API endpoint for your own cluster. Only data for your cluster shows up in your API.
Now, as far as what they're doing behind the scenes to make it look like this -- that is not public knowledge, but we can make some reasonable guesses as to how they might be doing it.
One approach is that they're actually building isolated control planes, complete with their own VMs, for each cluster. There are frameworks like ClusterAPI that make it possible to do this in an automated fashion. But this is not a very resource-efficient way to run thousands or millions of Kubernetes clusters on a major cloud platform, so this is probably not how the big players are doing it.
Another approach is to run separate control plane components in their own isolated containers on a pool of VMs. Perhaps even another Kubernetes cluster. So when a request for a new Kubernetes cluster comes in, the orchestrator just spawns a new kube-apiserver Deployment, a new etcd StatefulSet, etc. and plumbs them up to the designated worker node pools. I imagine this is more or less what Google and Amazon are doing, but probably a lot more sophisticated than what I'm describing.
And finally, an approach would be to eliminate the "upstream" Kubernetes API and management components (aside from kubelet and kube-proxy) all together, and instead create some bespoke, scalable, multi-tenant Kubernetes control plane service. This seems like something Google might do, as it would be the easiest to scale and would thus allow them to deliver the most clusters with the best performance at the lowest cost. However, it strays from the upstream Kubernetes code and so it would take a large, well-funded and dedicated development team to make this work.
If you are considering building your own multi-tenant Kubernetes infrastructure, I suggest you consider an orchestration tool like Rancher to do this, rather than doing it yourself. Using something like Rancher will allow you to create roles to control access for your tenants and create preconfigured cluster "templates". The tenants click a button to deploy a new Kubernetes cluster using whatever VM orchestration tool you have (such as VMware, Digital Ocean, or a cloud provider like AWS or GCP). Rancher in particular also has the ability to run multi-tenant clusters, in the sense of building a single, large Kubernetes cluster that many application developers use at the same time. In this case, you assign namespaces to each application team (Rancher has an abstraction they call "Projects" for this), and each application team may fully manage all the resources in their namespace(s), but not anything at the cluster level. It's then up to a "platform operations" team (I presume that's you?) to manage the cluster itself.
