Perl - Use Data::Dumper to write back to a file

Question

I have a large .csv file (2 - 3 million records). I need to concatenate the first three fields (with underscores) and append it to each record, then I need to sort file based on that new field and three other fields. I am able to do that (am testing it with a 4 record file for now) - but I'm not sure how to write it back to the file in the same .csv form - instead of the way Data::Dumper formats each line as a separate variable. Here is the code I have so far - I have a couple of Print (to screen) lines to see what it's doing -

#!/usr/bin/perl/

use strict;
use warnings;
use Data::Dumper;

my $filename = '/testpath/test.csv';
#$filename = 'test.csv';

open my $FH, $filename
  or die "Could not read from $filename <$!>, program halting.";

# Read the header line.
chomp(my $line = <$FH>);
my @fields = split(/,/, $line);
#print "Field Names:\n", Dumper(@fields), $/;
print Dumper(@fields), $/;

my @data;
# Read the lines one by one.
while($line = <$FH>) {

# split the fields, concatenate the first three fields,
# and add it to the beginning of each line in the file
    chomp($line);
    my @fields = split(/,/, $line);
    unshift @fields, join '_', @fields[0..2];
    push @data, \@fields;
}
close $FH;
print "Unsorted:\n", Dumper(@data); #, $/;

@data = sort {
    $a->[0] cmp $b->[0] ||
    $a->[20] cmp $b->[20] ||
    $a->[23] cmp $b->[23] ||
    $a->[26] cmp $b-> [26]
} @data;


open my $OFH, '>', '/testpath/parsedTest.csv';
print $OFH Dumper(@data);
close $OFH;

exit;

I'm assuming it's in the "print $OFH Dumper(@data);" line that I need to re-format it back to its original form.

And please be kind as I am a novice.

__________EDIT__________________________________

Here are the four lines from the test .csv file - first line is the header record:

STORE_NBR,CONTROL_NBR,LINE_NBR,SALES_NBR,QTY_MISTINT,REASON_CODE,MISTINT_COMM,SZ_CDE,TINTER_MODEL,TINTER_SERL_NBR,SPECTRO_MODEL,SPECTRO_SERL_NBR,EMP_NBR,TRAN_DATE,TRAN_TIME,CDS_ADL_FLD,PROD_NBR,PALETTE,COLOR_ID,INIT_TRAN_DATE,GALLONS_MISTINTED,UPDATE_EMP_NBR,UPDATE_TRAN_DATE,GALLONS,FORM_SOURCE,UPDATE_TRAN_TIME,SOURCE_IND,CANCEL_DATE,COLOR_TYPE,CANCEL_EMP_NBR,NEED_EXTRACTED,MISTINT_MQ_XTR,DATA_SOURCE,GUID,QUEUE_NAME,BROKER_NAME,MESSAGE_ID,PUT_TIME,CREATED_TS
1334,53927,1,100551589,1,6,Bad Shercolor Match,16,IFC 8112NP,01DX8005513,,,77,10/23/2015,95816,,OV0020001,,MANUAL,10/21/2015,1,0,,1,MAN,,CUST,,CUSTOM MATCH,0,TRUE,TRUE,O,5394A0E67FFF4D01A0D9AD16FA29ABB1,POS.MISTINT.V0000.UP.Q,PROD_SMISC_BK,414D512050524F445F504F533133333464EB2956052C0020,10/23/2015 10:45,10/23/2015 10:45
2525,67087,1,650462328,1,4,Tinted Wrong Product,14,IFC 8012NP,Standalone-5,,,11,10/23/2015,104314,,A91W00353,,,10/20/2015,0.25,0,,0.25,,,COMP,,CUSTOM MATCH,0,TRUE,TRUE,O,1AC5D8742D47435EA05343D57372AD32,POS.MISTINT.V0000.UP.Q,PROD_SMISC_BK,414D512050524F445F504F533235323531C2295605350020,10/23/2015 10:46,10/23/2015 10:47
1350,163689,1,650462302,1,3,Tinted Wrong Color,14,IFC 8012NP,06DX8006805,,,1,10/23/2015,104907,,A91W00351,COLOR,6233,10/23/2015,0.25,0,,0.5,ENG,,SW,,PALETTE,0,TRUE,TRUE,O,F1A072BCC548412FA22052698B5B0C28,POS.MISTINT.V0000.UP.Q,PROD_SMISC_BK,414D512050524F445F504F53313335307BC12956053C0020,10/23/2015 10:52,10/23/2015 10:52

Hope that's not too convoluted to read.

Answer 1

Data::Dumper outputs a format that is valid perl, and is good for debugging, but not for writing a CSV file. You could write the CSV by hand:

foreach my $row (@data) {
    print $OFG join(',', @$row), "\n";
}

but you really should use a specialized module, in this case Text::CSV, both for reading and writing the CSV – it will handle all the border cases (such as fields with embedded commas).

The synopsis contains a good example of both reading and writing; I won't repeat that here.

Answer 2

You don't have to rebuild the line if you just store it in @data too!

my @data;
while(my $line = <$FH>) {
    chomp($line);
    my @fields = split(/,/, $line);
    push @data, [ "$line\n", join('_', @fields[0..2]), @fields[19, 22, 25] ];
}

@data = sort {
    $a->[1] cmp $b->[1] ||
    $a->[2] cmp $b->[2] ||
    $a->[3] cmp $b->[3] ||
    $a->[4] cmp $b->[4]
} @data;

print($OFH $_->[0]) for @data;

If your input didn't contain NULs, you could even use the following faster approach:

print $OFH
   map { /[^\0]*\z/g }
   sort
   map {
      chomp;
      my @fields = split /,/;
      join("\0", join('_', @fields[0..2]), @fields[19, 22, 25], "$_\n")
   }
   <$FH>;

---

But yeah, you should probably use a legit CSV parser.

use Text::CSV_XS qw( );

my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 1 });

my @data;
while (my $row = $csv->getline($FH)) {
    push @data, [ join('_', @$row[0..2]), $row ];
}

@data = sort {
    $a->[0]     cmp $b->[0]     ||
    $a->[1][19] cmp $b->[1][19] ||
    $a->[1][22] cmp $b->[1][22] ||
    $a->[1][25] cmp $b->[1][25]
} @data;

$csv->say($OFH, $_->[1]) for @data;

The following is the fast approach using a CSV parser:

use Text::CSV_XS qw( );

my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 2 });

print $OFH
   map { /[^\0]*\z/g }
   sort
   map {
      $csv->parse($_);
      my @fields = $csv->fields();
      join("\0", join('_', @fields[0..2]), @fields[19, 22, 25], $_)
   }
   <$FH>;

Answer 3

Was unable to use the Text::CVS_XS because it was not available on our server, unfortunately - but did find adding this single "print" line worked -

open my $OFH, '>', '/swpkg/shared/batch_processing/mistints/parsedTest.csv'; 
print $OFH join(',', @$_), $/ for @data; 
close $OFH; 

Tested out fine with the small file, now to test on the actual file!