Python: Import comma separated data from a TXT file

Question

I built a data logger with a Raspberry Pi Pico. The Data Logger saves the temperature every six minutes in the TXT file. I am currently failing to import the data from the TXT file into Python.

The data looks like this: 1,20.5,2,21.0,3,21.0,4,21.0,5,21.0,6,21.0,7,21.0,...

The data set thus contains two variables, which are separated by commas, an incrementing counter and the temperature with one decimal place. The dataset has about 240 measurements.

So far I have tried different methods to import the data. I have turned the TXT file into a CSV file and tried importing as a dataframe using pandas:

temp_df = pd.read_csv('temp_data.csv', header=None)

This gives me a df with one observation and a three-digit number of variables. I cannot import the dataset to have only two variables and about 240 observations.

I also tried to import the data as lists:

import csv 
file = open("temp_data.csv", "r")
temp_list = list(csv.reader(file, delimiter=","))
file.close()
print(temp_list)

Which results in the error: "TypeError: 'list' object is not callable".

--> All in all, I need a solution that uses the TXT file directly and creates a df. I am still very inexperienced with Python and hope for your help! Thanks in advance

Thanks for the recommendation --------------------------------------------------------------------------- TypeError Traceback (most recent call last) ~\AppData\Local\Temp/ipykernel_15780/510848478.py in 2 3 file = open("temp_data.csv", "r") ----> 4 temp_list = list(csv.reader(file, delimiter=",")) 5 file.close() 6 TypeError: 'list' object is not callable — jonas.weik, Jul 04 '22 at 14:41
It looks like you override `list` in your code. Do you use `list` as name somewhere in the code? — buran, Jul 04 '22 at 15:30

score 0 · Answer 1 · answered Jul 04 '22 at 14:28

0

You should try pd.read_fwf. It is for reading fixed width format.

https://pandas.pydata.org/docs/reference/api/pandas.read_fwf.html

answered Jul 04 '22 at 14:28

Abhishek

1,585
2
12
15

OP does not work with fixed-width file – buran Jul 04 '22 at 15:29

score 0 · Answer 2 · answered Jul 04 '22 at 14:39

I would use numpy here. Numpy are python lists with nice functionalities. They allow you to save/retrieve the list in a .npy file.

eg.

import numpy as np
import pandas as pd

data = [1,20.5,2,21.0,3,21.0,4,21.0,5,21.0,6,21.0,7,21.0]
data_numpied = np.array(data,dtype=float)
np.save("numpy_data_storage.npy",data_numpied,allow_pickle = True)

# Restart / exit here

new_data = [42,42,1.0]
old_data = np.load("numpy_data_storage.npy",allow_pickle=True)
new_data2 = np.concatenate((old_data,new_data))

dataframe = pd.DataFrame(new_data2,columns = ["Temperature"])

However, the ".npy" file is not human-readable.

mcrot · Answer 3 · 2022-07-04T15:00:38.413

0

Given the input file

1,20.5,2,21.0,3,21.0,4,21.0,5,21.0,6,21.0,7,21.0`

as example, try this:

import pandas as pd
df = pd.read_csv('temp_data.csv', header=None)
temp_df = pd.DataFrame(index=df.iloc[0,0::2].astype(int).values, data=dict(temp=df.iloc[0,1::2].values))
print(temp_df)

Result:

edited Jul 04 '22 at 15:00

answered Jul 04 '22 at 14:43

mcrot

446
2
12

score 0 · Accepted Answer · answered Jul 04 '22 at 14:54

Does this help?

import pandas


data = dict()

with open("data.txt") as f:
    for line in f.readlines():
        splitted = line.split(",")
        data = {**data, **dict(zip(splitted[::2], splitted[1::2]))}

as_dataframe = pandas.DataFrame({"counter": data.keys(), "temperature": data.values()})

Python: Import comma separated data from a TXT file

4 Answers4