1

I have several thousand csv files I wish to reformat. They all have a standard filename with incremental integer, eg. file_1.csv, file_2.csv, file_3.csv, and they all have the same format: 

CH1
s,Volts
-1e-06,-0.0028,
-9.998e-07,-0.0032,
-9.99e-07,-0.0036,

For 10,002 lines. I want to remove the header, and I want to separate the two columns into separate files. I have the following code which produces the results I want when I consider a single input file:

tail -10000 file_1.csv |
awk -F, '{print $1 > "s.dat"; print $2 > "Volts.dat"}'

However, I want something that will produce the equivalent files for each csv file, say, replace s.dat with s_$i.dat or similar, but I'm not sure how to go about this, and how to call in each separate csv file in a loop rather than explicitly stating it as file_1.csv.

economy
  • 4,035
  • 6
  • 29
  • 37
Hodgesaargh
  • 11
  • 3

1 Answers1

1

awk to the rescue!

 awk -F, 'FNR>2{print $1 > "s_"FILENAME".dat"; 
                print $2 > "Volts_"FILENAME".dat"}' file*

or reading the filename from the data files

$ awk -F, 'FNR==2{s="_"FILENAME".dat";h1=$1s;h2=$2s} 
            FNR>2{print $1 > h1; print $2 > h2}' file*
karakfa
  • 66,216
  • 7
  • 41
  • 56
  • Thanks, I didn't know about the FNR variable! That works, but it produces output files with names like `s_file_1.csv.dat` whereas I'd like to ditch the `file_` and `.csv` whilst just retaining the number. I have `awk -F, 'FNR>2{print $1 > "s_"substr(FILENAME,5,1)".dat"; print $2 > "Volts_"substr(FILENAME,5,1)".dat"}' file*` which works but only while the number is in single digits. Is there a way to vary the length of the selected substring? – Hodgesaargh Dec 04 '15 at 23:18
  • Yep! I found it! Apparently this works quite well `awk -F, 'FNR>2{print $1 > "s_"substr(substr(FILENAME,0,length(FILENAME)-4),5)".dat"; print $2 > "Volts_"substr(substr(FILENAME,0,length(FILENAME)-4),5)".dat"}' file*` it removes both the `file_` and the `.csv` so should work once the number of digits increases. – Hodgesaargh Dec 04 '15 at 23:28