2

I would like to concatenate multiple .txt files in one single file, putting also the file name as the first column before each file (in order to understand from which file the data comes from). The code I am using below does this, but only for the first row.

for i in *.txt; do echo -n "$i," && cat "$i"; done > tmpfile; mv tmpfile all-files.txt;

for example output like this:

filename1.txt,COVERAGE SUMMARY,,Aligned bases in genome,80754336928,100.00
filename1.txt,COVERAGE SUMMARY,,Average alignment coverage over genome,26.55
filename2.txt,COVERAGE SUMMARY,,Aligned bases in genome,88896465740,100.00
filename2.txt,COVERAGE SUMMARY,,Average alignment coverage over genome,33.40
user3224522
  • 1,119
  • 8
  • 19

3 Answers3

2

I'd propose the use of awk:

for f in *.txt; do awk "{print \"$f, \" \$0}" "$f"; done
leu
  • 2,051
  • 2
  • 12
  • 25
2

suggesting gawk command:

To print the filename followed by , in first line of each file.

gawk 'BEGINFILE{printf("%s,",FILENAME)}1' *.txt

To print the filename followed by , in every line of each file.

awk '{print FILENAME "," $0}' *.txt
Dudi Boy
  • 4,551
  • 1
  • 15
  • 30
  • one doesn't need gawk to achieve that BEGINFILE effect - this one is identical, plus not splitting unnnecessary fields :::::::::::::::::::::::::: mawk -F'^$' 'NF – RARE Kpop Manifesto May 30 '22 at 07:27
2

3 different variations, and sub() is a smidgen faster :

CODE

{m,g} 'sub("^",FILENAME",")' FS='^$' *.txt

BENCHMARKING

 ( time ( mawk2  '$NF=FILENAME","$NF' FS='^$' "${m3t}" )  | pvE9 >/dev/null)

 out9: 2.34GiB 0:00:02 [ 896MiB/s] [ 896MiB/s] [ <=> ]

 2.33s user 0.34s system 99% cpu 2.685 total

———————

 ( time ( mawk2  '$!_=FILENAME","$!_' FS='^$' "${m3t}" )  | pvE9 >/dev/null) | lgp3 

 out9: 2.34GiB 0:00:02 [ 876MiB/s] [ 876MiB/s] [<=> ]

 2.38s user 0.34s system 99% cpu 2.744 total

———————

( time ( mawk2  'sub("^",FILENAME",")' FS='^$' "${m3t}" )  | pvE9 >/dev/null) | lgp3 
 
 out9: 2.34GiB 0:00:02 [ 987MiB/s] [ 987MiB/s] [ <=> ]

 2.10s user 0.32s system 99% cpu 2.441 total
RARE Kpop Manifesto
  • 2,453
  • 3
  • 11