1

I will be importing data from SQL Server into Microsoft Dynamics CRM (Dataverse) using a custom Console application (C#) I will work on. One of the requirements is that we can restart the import in case something happens and the application fails - and the application should continue from the last imported row. How can I achieve this?

I have two ideas for now, but I would like to hear other opinions (or maybe best practices from people that have already done this):

  • An additional column in the SQL table which indicates whether the row has been migrated (requires writing data back to the DB which will impact the speed)
  • Always get all data from SQL and check if the IDs are existing in CRM (if we stop almost at the end, we will have to go through all rows just to get to one that is not imported yet)

The rows will be not ordered and will be the same on the next try.

ddelic
  • 89
  • 13
  • 1
    Usually you add a DataTime to database to solve this type issue. You can then query database/table for latest entry and then add only objects that have a date later than last entry in database. – jdweng Aug 18 '22 at 10:32
  • It dépends on many things : is the data ordered, is the data the same on the next try, has the data unique id, etc. You can look at orchestration frameworks such as Airflow to see how they deal with this – Romka Aug 18 '22 at 10:35
  • 1
    Both of your ideas will work. The 1st one has 2 disadvantages: A - it forces you to create an additional column in every SQL table you want to import and B - Writing takes longer than reading, which will degrade the performance. The 2nd one seems better, as you don't need to change the database and you'll only read more, but not actually write anything (which takes longer). A 3rd option would be doing your 1st option but instead of changing the database and write on it, creating a json log file that would contain the same information written on disk. This log file would be updated on every run – Luis Gouveia Nov 12 '22 at 10:53

0 Answers0