3

I am reading many csv files and extracting columns 1, 6, 8, and 20. The delimiter is '","' because the csv is using double quotes. This is working great unless some of the data is flagged, in which case the everything is thrown out of whack. Data Example:

 Date            Year    Month   Day    Flag     Min T    Flag   Max T

 30/11/2007     2007    11       30     [Blank]  -14.9   [Blank]  -20.3

 01/12/2007     2007    12       1         *     -16     [Blank]  -20.1

Script reads perfect up to the * then everything is shifted so all I get is NaNs.

fid1 = fopen(File, 'r');
Date = textscan(fid1, '%q %*s %*[^\n]', 'Delimiter', ',', 'HeaderLines', 25);
fclose(fid1);
Date = datenum(Date{1, 1}, 'yyyy-mm-dd');
fid1 = fopen(File, 'r');

Data = textscan(fid1, '%*s %*s %*s %*s %f %f %*s %*s %*s %*s %*s %f %*[^\n]', 
'Delimiter', '","', 'HeaderLines', 25,'treatAsEmpty', {'M', '*', 'E', 'T', 'A', 'C', 
'L', 'N', 'Y', 'S', 'F'}, 'multipledelimsasone', true);
fclose(fid1);

So I suppose my question is:

Can I Preformat each file with a loop to remove the flags OR change the textscan to ignore the flags all together so that my delimiter works.

Thanks for your input!

Evan M Kraemer
  • 47
  • 1
  • 5

1 Answers1

0

Are the 'flags' the asterisks? You can use a terminal command like

Date = textscan(system(['sed "s/\*/ /g" ' File ]));

to remove the asterisk and scan the result.

RussH
  • 329
  • 2
  • 17
  • There are several different flags, * is the one in this portion of the example file. M, T, S, C and so on also occur as flags. – Evan M Kraemer Jan 14 '13 at 22:30
  • are you just looking to grab the dates? – RussH Jan 14 '13 at 23:47
  • @RussH You can't know for sure that `sed` is installed. Also, it's better not to avoid the `system` command if there's a simpler native solution (for instance one that employs `regexp`). – Eitan T Jan 15 '13 at 07:50
  • I need dates from string into serial # and then the other columns are min/max temp, total precip values comma delimited with double quotes. It is a daily time series of 60 years. – Evan M Kraemer Jan 15 '13 at 15:08