Importing data without headers and rearranging column data

Question

I have a 100 .dat files like the one in the left hand pane, which I need to import without headers and then sort rows.

workspace

ANSWER for doing it manually, file by file:

data=sortrows(data,2); #sort all columns of data via the 2nd col


fid=fopen('pole_2_TurbI_chamber_05_RSM.xy');
[x ~] = textscan (fid, '%f %f', 'HeaderLines', 4);  # reads file correctly
fclose(fid);

v(:,1)=cell2mat(x(:,1)); # convert from cell to array
v(:,2)=cell2mat(x(:,2));
v=sortrows(v,2);         # sort rows according to column 2

% fig
plot(v(:,1),-v(:,2),'ro');

How can I extend this to all the files in my directory? Perhaps giving each imported variable the file name... if possible. Regards,

Can't you put a loop around your manual solution to make it an automatic solution? — Pursuit, Mar 04 '12 at 17:26

Barton Chittenden · Answer 1 · 2012-03-05T07:00:32.743

On a posix system, an individual file can be sorted using

sort -k 2 /tmp/sortme.txt

The output will be written to stdout.

If you want to sort a group of files, you would wrap everything in a for loop:

for i in *.dat
do
    sort -k 2 $i > $i.tmpsort -k 2 
    mv $i.tmp > $i
done

(in this example, make sure that you don't have any pairs of original input files that are named x.dat and x.dat.tmp, or you'll clobber x.dat.tmp).

Here's a version written in Perl, which should be portable your system (whatever you're running...). The script strips all lines which do not start with the digits 0-9.

#! /usr/bin/perl
use strict;

sub getcol2 {
     $_[0] =~ /\d+\.?\d*\s+(-?\d+\.\d+)/;
     print "$1\n";
     return( $1 ); 
}

for my $file ( @ARGV ) {
    my $INPUT;
    my @data;
    open($INPUT, "<", $file) or die "Cannot open '$file' for input: $!";


    while( <$INPUT> ) {
         #print "$_";
         push @data, $_ if(/^\d/);
    }
    close $INPUT;
    @data = sort { getcol2( $b ) <=> getcol2( $a ) } @data;    

    my $OUTPUT;
    open( $OUTPUT, ">", $file ); 

    for my $line ( @data ) {
        print $OUTPUT $line;
    }
    close( $OUTPUT ); 
}

I've called it 'sortdata.pl', it can be called as

perl sortdata.pl *.dat

It will overwrite data files; make sure that you back up the originals.

I actually have headers in the files too, which ideally I'd like to strip out. But since I'm not on Linux... is there any way around this? — HCAI, Mar 04 '12 at 17:47
Thank you for posting the working code. That will come in handy! However I don't want to change the files themselves. How can I import all the files in my directory by using a wild card or loop? eg I wanted all files called: 'pole_*_TurbI_chamber_05_RSM.xy' — HCAI, Mar 06 '12 at 11:39
I don't really know... I haven't used matlab since I was in college, running on Sun3 workstations. That was about 20 years ago. I doubt that I ever imported a file in that environment. I just found the 'strip headers and sort a bunch of files' problem interesting. — Barton Chittenden, Mar 06 '12 at 13:53

Importing data without headers and rearranging column data

1 Answers1