1

I have a CSV as follows, where the line is terminated by "+++" instead of new line. How to load the csv by doing a line break where the string "+++" is present?

VTS,51,0071,9739965515,NM,GP,INF01,V,19,072219,291014,0000.0000,N,00000.0000,E,07AE

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205+++VTS,51,0071,9739965515,NM,GP,INF01,V,18,072311,291014,0000.0000,N,00000.0000,E,C24E+++VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358

VTS,51,0071,9739965515,NM,GP,INF01,V,18,072319,291014,0000.0000,N,00000.0000,E,012F
VTS,51,0071,9739965515,NM,GP,INF01,V,19,072326,291014,0000.0000,N,00000.0000,E,B2E6+++VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072333,291014,0000.0000,N,00000.0000,E,9896
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072340,291014,0000.0000,N,00000.0000,E,9B23

First, I need to break the line where a new line or "+++" symbol is present and load the data. Then, again filter with the value 01 in second column.

Expected output:

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205
VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358
VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0
vefthym
  • 7,422
  • 6
  • 32
  • 58
Anas A
  • 199
  • 4
  • 19

1 Answers1

1

PigScript:

A = LOAD 'input.csv' AS (line:chararray);
B = FOREACH A {
                 splitRow = TOKENIZE(line,'+++');
                 GENERATE FLATTEN(splitRow) AS newList;
              }
C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',16));
D = FILTER C BY $1==01;
DUMP D;

OutPut:

(VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205)
(VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358)
(VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0)
Sivasakthi Jayaraman
  • 4,724
  • 3
  • 17
  • 27
  • can you please explain the above step soo that i can understand what internally working – Anas A Nov 06 '14 at 10:58
  • when i added an extra line E = foreach D generate $7,$15; DUMP E; the result which i get is (060037,061114,0068,00,4000,00,999,149,9594) (060113,061114,0068,00,4000,00,999,152,B927) why unnecessarily coming all those rest of the filed instead of $7 and $15 – Anas A Nov 06 '14 at 11:01
  • Change the line "C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',23));" ie, instead of 16 give 23. basically its total number of columns. I mistakenly gave as 16. – Sivasakthi Jayaraman Nov 06 '14 at 18:07