0

[Newbie question.]

I have the following scenario,

  • Source systems publishes an event for a new customer
  • Data warehouse consumes these events and saves them to a NewCustomer table
  • A SSIS process will run frequently (every 10 mins), processing these into the customer DIM table, and will either mark the rows as processed or remove them from the NewCustomer table

How do I mark/remove processed customers from the NewCustomer table, without removing any newly added customers which may have appeared during process?

dbones
  • 4,415
  • 3
  • 36
  • 52

1 Answers1

0

Think of a linear process and state. This is the way I understand your problem.

  1. An event causes a record to be created in NewCustomer.
  2. The state of this record is "New".
  3. The SSIS process picks up the record from NewCustomer and deposits it in CustomerDIM. It then marks the record in NewCustomer as "processed".
  4. The SSIS process removes any record in NewCustomer marked as "processed". Any new record showing up in the meantime is not marked as processed, so it's not removed, until the next SSIS session.

So maybe I don't understand your problem.

pdhpoolplayer
  • 1
  • 1
  • In the SSIS package, how do you mark NewCustomer rows as processed (or just remove them) once they have been transferred into the CustomerDIM? (step 3). I have a Data Flow with a source, and destination, how do I now update the source as being processed – dbones Dec 04 '15 at 18:01