There are two databases, Db_A and Db_B, each with their own data dictionary. Most of the data in my database, Db_A, will fit in some field or another of the target database Db_B. Many values from Db_A will require reformatting before being inserted into fields in Db_B, and some values to be inserted into Db_B will need to be derived from multiple fields in Db_A. Very few fields in Db_A will be transferable to Db_B without at least some processing. Some fields may require a lot of processing (especially those which are derived). Unfortunately the processing steps are not very consistent. Each field will essentially require its own unique conversion.
In other words, I have a large set of fields. Each field needs to be processed in a specific way. These fields may change and the way they need to be processed may change. What is the best way of implementing this system?
One way I've done this in the past was to have a central function which loops through each field, calling that fields function. I created one function per field and used a csv file to map fields to functions and to the parameters the functions need. That way if a new field is created, I can just update the csv file and write a method to handle its conversion. If the way a field is converted needs to be changed, I can just change the corresponding method.
Is this a good way of doing it? Any suggestions? I'm using Python.
