.txt file handling in python for ML data-CodePudding

I have .txt files that contain data for ML. Each file has 19 entries that is repeated again and again in the text files. I need to get these files in python in such a way that every entry becomes a column and their corresponding numerical values are placed in that column. I need to do this with all 19 entries set in the txt file(like '1' below). or a text file that contains only numerical data would also be okay(like '2' below). If you can write a code for this in python that would be awesome otherwise please describe some details how to do that

OUTPUT I WANT:

1click to view image or 2click to view image

TXT FILE:

Frame.id: 263126,Timestamp: 697287019071, Hand_number: 1 ,hand_Id_type: 238right hand ,hand_finger's_number:2,hand direction: (-0.142081, 0.865413, -0.480493) ,Palm position: (35.2841, 284.522, 330.828) ,Palm normal: (-0.686854, -0.435733, -0.581694) ,Finger_type(tipposition) TYPE_THUMB(-36.7239, 301.602, 330.845) ,Finger_type(tipposition) TYPE_INDEX(-14.9321, 347.039, 280.375) ,Finger_type(tipposition) TYPE_MIDDLE(5.5661, 258.191, 321.318) ,Finger_type(tipposition) TYPE_RING(20.0886, 251.219, 320.136) ,Finger_type(tipposition) TYPE_PINKY(27.5919, 259.584, 310.508) Frame.id: 263127,Timestamp: 697287037765, Hand_number: 1 ,hand_Id_type: 238right hand ,hand_finger's_number:2,hand direction: (-0.167599, 0.827034, -0.536587) ,Palm position: (31.7441, 283.449, 322.619) ,Palm normal: (-0.776892, -0.445873, -0.444562) ,Finger_type(tipposition) TYPE_THUMB(-38.8083, 304.444, 330.446) ,Finger_type(tipposition) TYPE_INDEX(-22.0532, 344.008, 273.496) ,Finger_type(tipposition) TYPE_MIDDLE(-0.161068, 258.09, 319.489) ,Finger_type(tipposition) TYPE_RING(13.8233, 250.236, 317.138) ,Finger_type(tipposition) TYPE_PINKY(20.1892, 257.352, 305.739) Frame.id: 263128,Timestamp: 697287057570, Hand_number: 1 ,hand_Id_type: 238right hand ,hand_finger's_number:2,hand direction: (-0.179139, 0.817551, -0.547284) ,Palm position: (30.8754, 280.871, 315.444) ,Palm normal: (-0.750039, -0.473481, -0.461797) ,Finger_type(tipposition) TYPE_THUMB(-40.4781, 299.689, 321.24) ,Finger_type(tipposition) TYPE_INDEX(-23.5209, 339.286, 264.435) ,Finger_type(tipposition) TYPE_MIDDLE(-0.157164, 254.483, 311.627) ,Finger_type(tipposition) TYPE_RING(14.1716, 247.067, 309.742) ,Finger_type(tipposition) TYPE_PINKY(20.45, 254.358, 298.274) Frame.id: 263129,Timestamp: 697287076710, Hand_number: 1 ,hand_Id_type: 238right hand ,hand_finger's_number:2,hand direction: (-0.191306, 0.830611, -0.522961) ,Palm position: (28.6877, 277.055, 299.705) ,Palm normal: (-0.739979, -0.472093, -0.479124) ,Finger_type(tipposition) TYPE_THUMB(-42.9838, 294.545, 305.915) ,Finger_type(tipposition) TYPE_INDEX(-26.5976, 335.972, 250.27) ,Finger_type(tipposition) TYPE_MIDDLE(-1.90294, 250.26, 294.934) ,Finger_type(tipposition) TYPE_RING(12.5955, 243.157, 292.909) ,Finger_type(tipposition) TYPE_PINKY(18.5831, 250.968, 281.465)

TEXT FILE LINK:https://drive.google.com/file/d/1X1NdQNYlQWuNpmzGL6Wwi_SqFjIXnRbb/view?usp=sharing

CodePudding user response：

Not the most sophisticated way to parse a file, but seems to work.

data = []

text = []
with open('data_file_1.txt') as f:
    text = f.readlines()

text = ''.join(text) # Join all lines together, just in case
text = text.strip('\n').strip('\t') # Make whole text one line, just in case
rows = text.split('Frame.') # Separate rows of data

for row in rows[1:]: # First split is empty, start at second
    row_data = {}
    expected = 0
    store = []
    for item in row.split(','): # Split data at comma. Note: tuples will be split also
        if expected: # If in last item was a start of tuple, add rest values here
            expected -= 1
            store[1].append(float(item.strip().strip(')')))
            if not expected: # when all values are in place, save whole item to row_data
                row_data[store[0]] = store[1]
                store = []

        # In single value cases, parse and save them
        elif any(x in item for x in ['id', 'Timestamp', 'Hand_number', 'hand_Id_type', "hand_finger's_number"]):
            name, value = item.split(':')
            try:
                value = int(value)
            except ValueError:
                value = value.strip()
            row_data[name.strip()] = value

        # In cases, where there are tuple of 3 floats, parse and store for save
        elif any(x in item for x in ['Finger_type(tipposition)', 'Palm', 'direction']):
            name, value = item.strip('Finger_type(tipposition) ').split('(')
            store = [name.strip().strip(':'), [float(value)]]
            expected = 2

    data.append(row_data)

for row in data: # Final data in list of dict's
    print(row)

# Export your data in format like: .csv, JSON, XML, what is suitable for use in your application

prints rows like

{'id': 263126, 'Timestamp': 697287019071, 'Hand_number': 1, 'hand_Id_type': '238right hand', "hand_finger's_number": 2, 'hand direction': [-0.142081, 0.865413, -0.480493], 'Palm position': [35.2841, 284.522, 330.828], 'Palm normal': [-0.686854, -0.435733, -0.581694], 'TYPE_THUMB': [-36.7239, 301.602, 330.845], 'TYPE_INDEX': [-14.9321, 347.039, 280.375], 'TYPE_MIDDLE': [5.5661, 258.191, 321.318], 'TYPE_RING': [20.0886, 251.219, 320.136], 'TYPE_PINKY': [27.5919, 259.584, 310.508]}

CodePudding user response：

I did some simple wrangling and made it into a CSV. Hope this works for you :

lines = []
with open("data.txt") as f:
    lines=f.readlines()

# Columns in file    
header = "Frame.id,Timestamp,Hand_number,hand_Id_type,hand_finger's_number,hand direction,\
Palm position,Palm normal,Finger_type(tipposition) TYPE_THUMB,\
Finger_type(tipposition) TYPE_INDEX,Finger_type(tipposition) TYPE_MIDDLE,\
Finger_type(tipposition) TYPE_RING,Finger_type(tipposition) TYPE_PINKY\n"
out_lines = [header] # to store rows in output file
tmp=[] # empty list to store one row
for line in lines[1:]:
    tmp.append(line.strip()) # strip and append
    # if line has last column in row, add it to tmp, make the line
    # add the line to list of rows and clear tmp
    if line.strip().startswith(",Finger_type(tipposition) TYPE_PINKY"):
        new_line = ''.join(tmp) '\n'
        for column in header.split(','):
            # replace header keywords and :
            new_line = new_line.replace(column.strip(), '').replace(':','')
        out_lines.append(new_line)
        tmp=[]
        
print(''.join(out_lines))

# Uncomment the below block to write the csv file     
# with open("output.csv","w ") as f:
#     f.writelines(out_lines)