Has 15 TXT files, each file contains 200 lines, each line has 500 is composed by the Numbers 0 or 1,
Form is as follows:
1110000001110101101001001110000000000000000000000000110001...
1110101101001001110000000000000000000000000110001101101001...
0000000000000000011000110110100100000000000000000110001101...
.
.
[is]
Want from 15 files, each file to choose a line out, the line 15 Numbers in accordance with the corresponding digits addition, want to find after superposition, the number is 0 or 1 that a combination of,
Such as the superposition of the two lines below:
1110000001110101101001001110000000000000000000000000110001...
1110101101001001110000000000000000000000000110001101101001...
Generation:
2220101102111102211001001110000000000000000110001101211002...
It is not in conformity with the requirements, what I want is after stacking Numbers are all 1 or 0,
[the experiment]
With 15 at first, the two dimensional array, 500 x200 two-dimensional arrays, but to do iterate over 15 layer, array add operation, but also to do is too time consuming, more than 4 layer circulation, basic system if not come,
Do you have what good method, do the traversal?
CodePudding user response:
Read the two files at a time, to calculate, will conform to the requirements of the results in a new file, record the data source, manual free memory, read the two files again,CodePudding user response:
Each file is read only one line? Random read a line? That as long as 15 variables recorded a line of data,According to your request, combined, will only 0 or 1 row data, two rows of data, the same position, then, not at the same time for 1
To record the location of every behavior, compare the two actions 1 data, position the same positive result is not what you want,
Use binary search algorithm, so the, there may be a promotion
CodePudding user response:
To calculate the string conversion again good, odd number is 1, an even number is 0, 15 together, however, may be 15 yo, think about how, or two lines of one plus one conversionCodePudding user response:
Calculation comparison algorithm, and I have a little idea, for reference:Look at this problem carefully, and found that number are all 1 s and 0 s, are likely to conjure up a binary,
If the character to binary calculation, can get a lot of simplified calculation,
From this point of view below:
1. The string with binary digital conversion between
# read the string, converted to a binary number:
String='1001'
Num=int (string, 2)
9 # the original string is converted to an integer or 0 b1001
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
# to digital conversion back to the original string:
STR (bin (9))
# Numbers, then back to the original string '0 b1001 (0 b need to delete the beginning two characters)
2. Analyze your requirements, only when the 1 and 1 met together, the data is not available, other 0 and 1, 0 and 0,
The bitwise with binary 'and' true value table,
So the calculation can be so:
convert strings to Numbers, bitwise operations, from
If result equals 0, then the data is available, or at least two as 1, so is not available,
Then the data is equal to zero for "bitwise or", the results continue to the next set of "and" operation, the number of
This is because: from group1 and group2 took two g1 and g2 [j], [I] if "bitwise and g1" [I] & amp; G2 [j] is not equal to zero,
Behind the 13 group takes no matter what, including g1 and g2 [j] [I] the combination of all unavailable,
For example:
G1 [0]=0 b1001
G2 [0]=0 b11000
G2 [1]=0 b110
G1 [0] & amp; G2 [0]=8 is not equal to zero, the g1 [0] and the combination of g2 [0] is unavailable
G1 [0] & amp; G2 [1]=0 so g2 and g1 [0] [1] the combination of available
Then put the g1 [0] and g2 [1] "or" operation,
The g1 [0] | g2 [1]=0 b1111, this value waiting to read the next set of iteration,
As a result, the algorithm of calculation comparison:
read two groups of data for the first time,
Cross the bitwise and & amp; ""
If is equal to zero, for "bitwise or |" operation, save the results
The else: abandon
Iteration, read the next set of data, and continue to "and" "or" operation
Continue until 15 set out, the rest is the desired result,
Computer do bitwise and or operation very fast
Traversal algorithm, may need to use the depth first search (DFS) algorithm, and study the DFS can not see
Personal humble opinion, I do not know right,
CodePudding user response:
15 per 200 lines of a file, I think the data volume is not big, can do violence directly, can you send the data to come up for everyone to have a try?CodePudding user response:
You that I'm considered as a 500 - bit binary number, number of sets of 15, 200 in each groupIf the number of 15 a1, a2,... A15 meet your request, so sure a1, a2, a2, a3,... A14, a15 both two meet the requirements of you
So you can get two files two such initialization, can screen out a lot
CodePudding user response:
Thanks for your reply, I rearrange the problems:I put the data import to sqlite files, file download: http://u.163.com/nnnnnEwm extraction code: 8 uqbfsjz
There is A, B, C... Number of 13 tables, each table for
A: 24, B: 72 C: 72, D: 72, E: 48, F: 48, G: 96, H: I: 96, 48, J: K: 24, 96, L: 96, M: 72,
Each table has the name field is to mark the uniqueness of each record, datas string field is made up of 729 0 or 1,
I use the Python for loop:
As A first step, the new AB table, put A, B to loop combination of data, remove does not conform to the requirements of record, the results are as follows:
Article 24 * 72=1728, article 1300 after screening, cycle time: 1.66 seconds, the database size: 1.0 MB,
ABC table, the second step, the newly built the AB table and C to loop combination of data, remove does not conform to the requirements of record, the results are as follows:
Article 1300 * 72=93600, article 53300 after screening, cycle time: 86.86 seconds, the database size: 41.7 MB
The third step, the newly built ABCD table, put the ABC and D to loop combination of data, remove does not conform to the requirements of record, the results are as follows:
Article 53300 * 72=3837600, article 1705600 after screening, cycle time: 3608 seconds, the database size: 1.30 GB
The third step is need 1 hour to cycle, and the size of the database has been on the GB, but I did step 4,
The fourth step, the newly built ABCDE table, put the ABCD and E to loop combination of data, remove does not conform to the requirements of record, the results are as follows:
Article 48=1705600 * 81868800, article 28995200 after screening, cycle time: 119253 seconds, the database size: 22.1 GB
Step 4 took about 33 hours, also is the time of waiting for more than a day, but can no longer do it down, so wait is for many days, and the database size is rising fast,
[environment]
Intel i5-3470 3.20 GHz, 8 gb of memory, Windows 7 SP1 64 + Jupyter Lab, Python 3.7.6, FireFox browser,
[cyclic code]
# create AB tables and fields to store A serial number, A deposit B B number, field DATAS to store the DATAS of A + B
The import sqlite3
Conn=sqlite3. Connect (' KML_AB. Db)
Print (" the Opened the database successfully ")
C=conn. Cursor ()
C.e xecute (" 'the CREATE TABLE AB (CHAR A CHAR (4), B (4), DATAS TEXT); "')
Print (" Table AB created successfully ")
MIT ()
conn.comConn. Close ()
Print (" # Datebase closed ")
# cycle to solve KML00. Db and KML01. Db content is exactly the same file
The import time
T1=time. Perf_counter ()
The import sqlite3
Conn0=sqlite3. Connect (' KML00. Db)
Conn1=sqlite3. Connect (' KML01. Db)
Conn2=sqlite3. Connect (' KML_AB. Db)
C0=conn0. Cursor ()
C1=conn1. Cursor ()
C2=conn2. Cursor ()
Print (" # the Opened the database successfully ")
Try:
For the x in the range (24) :
C0. Execute (" select rowid, name, datas from A where rowid="+ STR) (x + 1)
Result0=c0. Fetchone ()
For y in range (72) :
C1. The execute (" select rowid, name, datas from B where rowid="+ STR (y + 1))
Result1=c1. Fetchone ()
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull