Now going to put a database a table of data from the amount of data about 200 g of data read out, and do a processing, how to do the batch, if I at the time of processing data, program crash, when restart to ensure don't read repetitive data, this how to make, consult the great god,

CodePudding user response:

This is a table of data? Why not do partition,,,

CodePudding user response:

You can add a mark in the table, the processed write 1, not processed writing 0;

PS: 200 gigabytes of data, it is suggested that use stored procedures to handle, with the front desk program, or a little uncomfortable;

CodePudding user response:

@ ghx287524027 is the data table, on table is larger

CodePudding user response:

@ the net of selling fruits not only this form now, this is just one of a month, key generated a result table, the table is process is bad to change, I now idea is I can read a little at a time file, and then written to a file, circulation, speaking, reading and writing, so every time the JVM can recycle used to read from the cache, but they were afraid to efficiency couldn't keep up with,

CodePudding user response:

If so, you have to go to the space in time, using the principle of pages each time to read some data from the table, read cycles

CodePudding user response:

Suggested process, each time in a group of data, processing the records of this batch of data the largest number, the batch of data from the largest number start,
Program can also be connected to the Numbers continue to run

CodePudding user response:

The building Lord solved?