I am trying to determine column-level lineage between a target table and a number of source tables. The columns that end up in the target table come from one or more of the source tables and may have been transformed by one or more intermediate processes. Trouble is, I have no access to the intermediate processes - all I have are source tables and a target table. I am trying to find out whether there exists a class of solutions or tools for column level (fine-grained) lineage that assumes black-box processes.
Solutions that my searches turned up appear to determine lineage from code, e.g. SQL queries, etc, which obviously requires some foresight and pre-integration. I am working with legacy systems and data from different organizations for which getting access to transformation processes is just not going to happen. Searches for blackbox column level lineage didn't return anything I consider useful.
I am about to sketch out a custom solution but didn't want to undertake such a huge task without making sure something already exists. It seems unlikely that no one has tackled this problem.
Separately, I also would like to know whether there exists open-source standalone visualization tools for column-level lineage that was generated using another process.
