Read a directory of txt files line by line into an R dataframe with filenames as one column

Question

I have a directory of text files. I want to read the contents of these text files, line by line into an R dataframe. The text files contain unstructured text. The desired dataframe output is:

file; line
1.txt; "line 1 in 1.txt"
1.txt; "line 2 in 1.txt"
2.txt; "line 1 in 2.txt"
...

I have written the code below, but it leads to errors. I also guess there is a more straightforward way to do this, with for example readr and dplyr.

files <- list.files(path="./data", pattern = "*.txt", full.names = TRUE) # read data folder txt files

my_lines <-list() # create temp list for reading lines
df <- data_frame( "file" = character(0), "line" = character(0))

for (file in files){
    my_lines <- readLines(file) # read lines from file into a list
    for (line in my_lines){
        df$file<-file
        df$fline<-line
    }
}

You can read the whole file at once and then add a new column of filename. At the end, you can concatenate all `data.frame`s through `rbind` or sth like `bind_rows`. — mt1022, Apr 23 '17 at 15:51
If you like the tidyverse, you could find some helpful tutorials about data importing and cleaning from http://r4ds.had.co.nz — mt1022, Apr 23 '17 at 16:12

score 3 · Accepted Answer · answered Apr 23 '17 at 20:42

An alternative solution without loop:

> file = list.files(path="C:/...", pattern = "*.txt",full.names=T)
> line = lapply(file,readLines) 
> file = unlist(mapply(rep,file,sapply(line,length),SIMPLIFY=FALSE,USE.NAMES=FALSE))
> df=data.frame(file=file,line=unlist(line))

Setting full.names to TRUE makes for quite long file names... If you set the working directory beforehand, the path and the full.namesargument to list.files() wouldn't be necessary and your data-frame would only contain the actual file names without path.

score 1 · Answer 2 · answered Apr 23 '17 at 16:19

A simple (but inefficient) solution is:

files <- list.files(path="./data", pattern = "*.txt", full.names = TRUE)
fls <- NULL
lns <- NULL
for (file in files) {
  my_lines <- readLines(file)
  for (line in my_lines) {
    fls <- c(fls, file)
    lns <- c(lns, line)
  }
}
df <- data.frame(file=fls, fline=lns)
print(df)

   file          fline
1 1.txt line1_in_1.txt
2 1.txt line2_in_1.txt
3 2.txt line1_in_2.txt
4 2.txt line2_in_2.txt

Read a directory of txt files line by line into an R dataframe with filenames as one column

2 Answers2