Merge multiple .txt files from multiple directories in R

Question

Merge .txt files from different sub directories

I have a folder that is filled with sub folders of past dates (01_14 for example), inside each date folder there are 11 files named 01.txt, 02.txt... How can I merge all the .txt files into one data frame, with one column with the name of the folder from where it came from and a column with the name of file from where it came from?

My hierarchy would look something like this:

\Data
     \01_14
           01.txt
           02.txt
           ...
           11.txt 
     \02_14
           01.txt
           02.txt
           ...
           11.txt 
     \03_14
           01.txt
           02.txt
           ...
           11.txt 

Answer 1

When I need to read multiple files, i use a read.stack helper function which is basically a wrapper to read.table but it allows you to also add extra columns on a per-file basis. Here's how I might use it with your scenario.

dir<-"Data"

subdir<-list.dirs(dir, recursive=F)

#get dir/file names
ff<-do.call(rbind, lapply(subdir, function(x) {
    ff<-list.files(x, "\\.txt$", include.dirs = FALSE, full.names = TRUE)
    data.frame(dir=basename(x), file=basename(ff), 
        fullpath=ff, stringsAsFactors=F)
}))

#read into data.frame
read.stack(ff$fullpath, extra=list(file=ff$file, dir=ff$dir))

Answer 2

Try this:

fileNames <- list.files("Data", recursive = TRUE, full.names = TRUE)
fileContents <- lapply(fileNames, function(fileName) 
  paste(readLines(fileName, warn = FALSE), collapse = "\n"))
meta <- regmatches(fileNames, regexec(".*Data/(.*)/(.*)$", fileNames))
merged <- mapply(c, fileContents, lapply(meta, "[", -1), SIMPLIFY = FALSE)
as.data.frame(t(do.call(cbind, merged)))