Currently I'm working with Kubeflow. It is a quite large set up with about 30 different deployments. The default manifests of the Kubeflow team is using the standard Kustomize tool to provide patches for different environments like on-prem, cloud, testing, etc.
However, I still feel quite overwhelmed and limited with all those configurations. The only way that I can quickly navigate and manipulate variables for the whole project is to use the search and/or replace function from IDE (yes I know Kustomize can do variables in each environment but I want to do it for all deployments). This sucks, as it is not reversible once replaced. Another problem is that many folders are not just named base or reverse so many folders are named base, making it quite hard to findout some original fields. I also struggled when I want to combine just a few parts of different environment. For example, they provided 3 different environment for dex auth, 1 using email, 1 using GitHub and 1 using Google. I want my setup to have all 3 of them but now I cannot reuse the config from those environment easily. I had to create my own environment and dig up each of those environment to see what changes did they make in the patches.
I have never tried setting up Jsonnet configurations (maybe with tools like Tanka) on a similar cluster. I'm seeing a few big companies using Jsonnet. The two that I know are Grafana (they even created Tanka) and Databricks (they create their own Jsonnet compiler). What are the pros and cons of doing k8s configurations management in Jsonnet compared to Kustomize - the currently most popular choice? Is it worth learning and managing k8s config using Jsonnet (maybe with Tanka)?
