0

This has been discussed in stackoverflow before but I couldn't find a case/answer that might apply to my situation:

From time to time I have raw data in text to be imported into SQL, for almost every case I must try out several times as SSIS wizard doesn't know what's the max size of each field and the default is 50 characters. Only after it fails I can know from the error message which (first) field was truncated and I then increase the field's size.

There might be more than one field that needs getting its size increased, and the SSIS wizard only gives one error each time it encounters a truncate, as you can see this is very tedious, I want to find a way to have a quick inspect to the data first to determine the max size of each field.

I came across an old post on stackoverflow: Here is the post

Unfortunately it might not work on my case: my raw data could have as many rows as 10 Million (yes, in one single text file which is over GB).

I am kind of do not think there would be a way to get that, but just still want to post my question here hoping to get some clue.

Thank you very much.

Community
  • 1
  • 1
PasLeChoix
  • 311
  • 1
  • 5
  • 21
  • Can you use any programming language? There are plenty of questions of people posting how to do so in Java, C#, etc.... – John Bustos Nov 08 '16 at 18:58
  • Thank you. The question here is probably not programming but the size of the raw data, I have a feeling this is actually a mission impossible because it seems to be any script must read each row before determine the field's length. I could be wrong though, that's why I post this question. – PasLeChoix Nov 08 '16 at 19:18
  • You would definitely need to read every row, but, if done via an efficient reader that reads row-by-row, it's not a big deal since it could stil lbe very fast and have a small memory-footprint. That's why a program would be best IMHO. – John Bustos Nov 08 '16 at 19:40
  • Also, maybe something like this: http://stackoverflow.com/a/17498835/1693085 – John Bustos Nov 08 '16 at 19:44

0 Answers0