Old man, what I read in the size of a 1 g in python CSV, content is a two-dimensional matrix, element of 10 decimal places, how to convert it to a list of the list? I now have pandas readcsv read and then into a dataframe, then the value tolist, this can be achieved, but don't know why, 1 g files take 10 g of memory, I can't stand, so I have to optimize the dtype appointed float32, is very big still,
Excuse me what method can read in the CSV, at the same time, smaller memory method?

CodePudding user response:

Then write their own, to see if can achieve efficiency, cut a few M for testing first,
Code like this

 

Data_file=open (' data. CSV ', 'r') 
List1=data_file. Readlines () 
Data1=[] 
For l in list1: 
List2=l.s trip (). The split (', ') 
Data1. Append (list2) 

CodePudding user response:

reference 1st floor seakingx response:

write their own, then look at efficiency can achieve, to cut a few M for testing first,
Code like this

 

Data_file=open (' data. CSV ', 'r') 
List1=data_file. Readlines () 
Data1=[] 
For l in list1: 
List2=l.s trip (). The split (', ') 
Data1. Append (list2) 

well, I'm going to try ~ (actually I also don't want to use dataframe, turn to two steps, and then a great memory, del variables and then collect also close not to come back, don't know why, though is likely to be del, but py haven't return the memory come back, anyway, recycling effect is very poor)