Python: Import comma separated data from a TXT file-CodePudding

I built a data logger with a Raspberry Pi Pico. The Data Logger saves the temperature every six minutes in the TXT file. I am currently failing to import the data from the TXT file into Python.

The data looks like this: 1,20.5,2,21.0,3,21.0,4,21.0,5,21.0,6,21.0,7,21.0,...

The data set thus contains two variables, which are separated by commas, an incrementing counter and the temperature with one decimal place. The dataset has about 240 measurements.

So far I have tried different methods to import the data. I have turned the TXT file into a CSV file and tried importing as a dataframe using pandas:

temp_df = pd.read_csv('temp_data.csv', header=None)

This gives me a df with one observation and a three-digit number of variables. I cannot import the dataset to have only two variables and about 240 observations.

I also tried to import the data as lists:

import csv 
file = open("temp_data.csv", "r")
temp_list = list(csv.reader(file, delimiter=","))
file.close()
print(temp_list)

Which results in the error: "TypeError: 'list' object is not callable".

--> All in all, I need a solution that uses the TXT file directly and creates a df. I am still very inexperienced with Python and hope for your help! Thanks in advance

CodePudding user response：

Does this help?

import pandas


data = dict()

with open("data.txt") as f:
    for line in f.readlines():
        splitted = line.split(",")
        data = {**data, **dict(zip(splitted[::2], splitted[1::2]))}

as_dataframe = pandas.DataFrame({"counter": data.keys(), "temperature": data.values()})

CodePudding user response：

You should try pd.read_fwf. It is for reading fixed width format.

https://pandas.pydata.org/docs/reference/api/pandas.read_fwf.html

CodePudding user response：

I would use numpy here. Numpy are python lists with nice functionalities. They allow you to save/retrieve the list in a .npy file.

eg.

import numpy as np
import pandas as pd

data = [1,20.5,2,21.0,3,21.0,4,21.0,5,21.0,6,21.0,7,21.0]
data_numpied = np.array(data,dtype=float)
np.save("numpy_data_storage.npy",data_numpied,allow_pickle = True)

# Restart / exit here

new_data = [42,42,1.0]
old_data = np.load("numpy_data_storage.npy",allow_pickle=True)
new_data2 = np.concatenate((old_data,new_data))

dataframe = pd.DataFrame(new_data2,columns = ["Temperature"])

However, the ".npy" file is not human-readable.

CodePudding user response：

Given the input file

1,20.5,2,21.0,3,21.0,4,21.0,5,21.0,6,21.0,7,21.0`

as example, try this:

import pandas as pd
df = pd.read_csv('temp_data.csv', header=None)
temp_df = pd.DataFrame(index=df.iloc[0,0::2].astype(int).values, data=dict(temp=df.iloc[0,1::2].values))
print(temp_df)

Result: