How to read a large number of text files in Python software and convert data into a sample file?-CodePudding

I receive information from an application. I have given a very small example as follows.

import pandas as  pd
df = [{'etelat':{'name' : 'sajjad1' , 'last_name' : 'esma1' },

       'class_id':{'id':101 , 'name_os' : 'win'}}, 

       {'etelat':{'name' : 'sajjad2' , 'last_name' : 'esma2' },

       'class_id':{'id':102 , 'name_os' : 'mac'}}]
df
type(df)
# Import pandas library
import pandas as pd
# initialize list of lists
data = [['sajjad1', 'esma1',101,'win'], ['sajjad2', 'esma2',102,'mac'] ] 
# Create the pandas DataFrame
df_new = pd.DataFrame(data, columns = ['etelat_name', 'etelat_last_name','class_id_id', 'class_id_name_os'])
# print dataframe.
df

1- In this case, it answers correctly, but if we want the data, which is in the form of text, I will have a problem.

2- If the data is in a folder and the number of text files is 50, if we want to call the desire of the files with a code. How is this code? enter image description here

CodePudding user response：

If you have your list of lists in a file like the following:

example.txt

[['sajjad1', 'esma1',101,'win'], ['sajjad2', 'esma2',102,'mac'],
['sajjad3', 'esma1',103,'win'], ['sajjad4', 'esma2',104,'mac'] ]

you can read the file and convert it to a pandas dataframe:

import pandas as pd

columns = ['etelat_name', 'etelat_last_name','class_id_id', 'class_id_name_os']
with open('example.txt', 'rb') as file:
    data = eval(file.read())
  
df = pd.DataFrame(data=data, columns=columns)
print(df)

output:

  etelat_name etelat_last_name  class_id_id class_id_name_os
0     sajjad1            esma1          101              win
1     sajjad2            esma2          102              mac
2     sajjad3            esma1          103              win
3     sajjad4            esma2          104              mac

Edit:

you can load it and convert it to utf-8 and then you get a list of dictionaries that you can deal with.

import ast
lst = []
with open('example.txt', 'rb') as file:
    content = file.readlines()
for item in content:
    temp = item.decode('UTF-8')
    lst.append(ast.literal_eval(temp))
print(last)

output:

[({'etelat': {'name': 'sajjad1', 'last_name': 'esma1'}, 'class_id': {'id': 101, 'name_os': 'win'}},), ({'etelat': {'name': 'sajjad2', 'last_name': 'esma2'}, 'class_id': {'id': 102, 'name_os': 'mac'}},)]

tested file with:

{'etelat':{'name' : 'sajjad1' , 'last_name' : 'esma1' },'class_id':{'id':101 , 'name_os' : 'win'}}, 
{'etelat':{'name' : 'sajjad2' , 'last_name' : 'esma2' },'class_id':{'id':102 , 'name_os' : 'mac'}},