How to make a unique dataframe from multiple .txt files?

Question

I'm trying to make a data frame from multiple .txt files. It may be important to say, those .txt files used to be .dat files, which I've just converted by changing the .dat to .txt. Anyway, it opens normally with a double click.

My code is:

path = "/home/marlon/Shift One/Projeto Philips/Consolidação de Arquivos/dados/17448"

files = []
for i in os.listdir(path):
    if os.path.isfile(os.path.join(path,i)):
        files.append(i)

df = pd.DataFrame()
for i in files:
    frame = pd.read_csv(i)
    df = df.append(frame)
print(df)

The output is:

...

  File "pandas/_libs/parsers.pyx", line 384, in pandas._libs.parsers.TextReader.__cinit__

  File "pandas/_libs/parsers.pyx", line 695, in pandas._libs.parsers.TextReader._setup_parser_source

FileNotFoundError: File b'monitor_System_HumTechRoom.txt' does not exist

Thanks for your help!

@G.Anderson they are four files whose names are 'monitor_System_HumExamRoom.txt', 'monitor_System_TempTechRoom.txt', 'monitor_System_HumTechRoom.txt', 'monitor_System_tempExamRoom.txt'. That's all! — Marlon Henrique Teixeira, May 23 '19 at 20:53
So the entries don't include the full path from the script to the file? `"/home/marlon/Shift One/Projeto Philips/Consolidação de Arquivos/dados/17448/monitor_System_HumExamRoom.txt" != "monitor_System_HumExamRoom.txt"` — G. Anderson, May 23 '19 at 20:55
@G.Anderson, I don' t think so, as it is displayed in the code about. The path is determined in `path`. — Marlon Henrique Teixeira, May 23 '19 at 20:57
No it is not. The full error message tells you _where_ the error occurred. — DYZ, May 23 '19 at 20:57
If you don't pass the path+filename to `pd.read_csv()`, then the script will look in its local folder, and not find the file.. Have you tried `frame = pd.read_csv(path+'/'+i)` or similar? — G. Anderson, May 23 '19 at 20:59

score 0 · Accepted Answer · answered May 23 '19 at 21:18

0

Your list "files" contains only the contents of the directory "path" instead of the absolute paths to the files in there. Pandas are looking for "monitor_System_HumTechRoom.txt" instead of "/home/marlon/Shift One/Projeto Philips/Consolidação de Arquivos/dados/17448/monitor_System_HumTechRoom.txt" hence the FileNotFoundError. Do I understand correctly that you want to read all files from a directory as csvs into a single dataframe and then print the dataframe?

No need to loop twice. Try something like this:

csvdir = "/home/marlon/Shift One/Projeto Philips/Consolidação de Arquivos/dados/17448"

dataframes = []
for csv in os.listdir(csvdir):
    fullpath = os.path.join(csvdir, csv)
    if os.path.isfile(fullpath):
        # Read a csv into a dataframe and append to a list of dataframes
        dataframe = pd.read_csv(fullpath)
        dataframes.append(dataframe)

# Concatenate all created dataframes into one
df = pd.concat(dataframes)
print(df)

answered May 23 '19 at 21:18

pnovotnyq

547
3
12

Thank you very much! It got my problem! – Marlon Henrique Teixeira May 24 '19 at 19:23
By the way, what if instead of concatenating the frames I want to make each frame in "dataframes" a column? :) – Marlon Henrique Teixeira May 24 '19 at 19:29
Not sure I understand. Does `pd.concat(dataframes, axis=1)` do what you want? – pnovotnyq May 24 '19 at 19:35
Ow yeah! Man thanks! I’m beginner I apologize for those stupid questions! – Marlon Henrique Teixeira May 24 '19 at 19:37
1

That's alright! I'm no pro either. I personally found this video very helpful. [SciPy 2017](https://www.youtube.com/watch?v=oGzU688xCUs&t=11997s) – pnovotnyq May 24 '19 at 20:19

How to make a unique dataframe from multiple .txt files?

1 Answers1