Now there is a large file, want to read by line, and then according to the sorting some element in each row,
Such as: the following two lines, after reading, sorted by time, 20200120031738/20200120031742, so to form new a file,
+ ACK: GTRTO, 570902359640984179204, 95-5, RSTBAT, 0183202012031, 738000 B $
+ ACK: GTRTO, 570902359640984179204, 95-5, DIF, 742000 C $0184202012031
I try to use sorted and sort, seem unable to form a satisfactory answer,
thank you

CodePudding user response:

Break up, into a table sorting pandas

CodePudding user response:

You have to change a kind of solution:
1.) will remain large files bigfile to read according to the line
2). But every read a line written to a new file according to certain conditions newfile

Write new files need to be aware of some conditions:
1). With the method of random writes, ensure newfile after each write all have formed the right sort
2.) when random write how to obtain newfile file location offset, you can use regular expressions to match lines
(see the file content format as you describe are relatively uniform, is + ACK: at the beginning, end $,)
3). In order to increase the search efficiency, in every time write newfile, record has been written to the file number of rows, and then use binary search more quickly locate newfile offset.

CodePudding user response:

And, of course, if with the help of a database table of the means to achieve, in fact more convenient!