Is the fact that the database has nearly 1 million data, need to query, processing, query traversing data alone, very slow, is there any examples of concurrent want concurrent processing these data at the same time, speed up, thank you, is not on the query speed bottleneck, but on the process data, have to deal with one by one, I take out the first 1000 data, for example, in the list1, list2, list3, list4, list5 Tstringlist (define) inside each 200 different content data, and then points 5 thread processing, found than 1000 single thread processing on the processing time is not fast, but slow, and dealt with in a thread under a thread to handle, so ask Daniel to write, how to improve the speed, data take take according the above list first, how to make 5 thread processing at the same time, thank you

CodePudding user response:

Should be the problem of the code you write,

CodePudding user response:

Nobody reply, hope to Daniel to write simple, how multi-threaded concurrent, how to account for the kernel

CodePudding user response:

First, you should optimize the database, the rational use of the index, the data is too much to add the hash algorithm, and overall improve the efficiency of the query methods too much, but it is multi-threaded problem too, you the CPU efficiency, is at full speed for query is not much help,
Question is, why do you want to traverse the 1 million data, the query's purpose is to find useful, like baidu find out in the hundreds of millions of entries to recommend to you, it took him a long time??????? He in traverse to also do not know to baidu to qualifications,,,, if you input the start purposefully establish database, effective use of time, don't bother, to find something useful for a second is not something,

CodePudding user response:

If you deal with different transaction, such as a thread, you put some affairs more time-consuming another thread to handle other affairs, that may increase the processing of your ability, but a transaction to you a few threads does not improve efficiency, the CPU can only handle one thing at a time, you are a thread in the query, another thread can only wait for, you should consider how to optimize your business logic,

CodePudding user response:

Under the restriction of hardware, concurrency is not speed, because of, concurrent to the allocation of CPU, a thread, the CPU to calculate, multiple threads to each thread respectively, the allocation of CPU time, so, sometimes multithreading, rather slow,
Unless your computer CPU speed, core number, server, for example, at this point, the concurrent can improve the speed,
Image processing, you need more core, large memory, in this way, the realization of parallel, in the true sense to improve the speed,

CodePudding user response:

Third floor: man, I was on the server, there is a index, is currently the first found out on the server memory, the key to data processing, so have to traverse, I'm processing, such as article 1 million before the news events, news content, there are others posted a news, I will check before news of similarity, so all processing once,

CodePudding user response:

Fifth floor man, give a method of trouble ah, is first put into memory, then process, how the asynchronous thread processing, is planning to put 16 nuclear server as a service,

CodePudding user response:

The query to the data in the TClientDataSet cache to the local processing again, but from what you said "and dealt with in a thread under a thread to handle" look or your code has a problem,

CodePudding user response:

1, optimize your SQL and optimization index,
2, if the optimized SQL can not reach you request that you change the performance more servers and clusters

CodePudding user response:

Article 1 million the previous news events, for example, news content, there are others posted a news, I will check before news of similarity, so all processing once,
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
I think you have a problem of the algorithm, a news, will traverse the original 1 million records, this is too terror, can try to add keywords, to extract the new news keywords, and then extracting keywords related news only for processing

CodePudding user response:

As far as I know, multithreading is the use of the CPU of free time to do something else, avoid is not performed at the same time, I understand, right?

CodePudding user response:

This problem in two kinds of circumstances, a server database is the bottleneck, the second is the client the bottleneck, if you want to simple data processing, that both good do,
Dividing the database into several libraries, each library into the part of the data, then n client (note) on different computers, there is no problem as long as the network, this certainly can deal with OK,
Is actually distributed processing of large data, but the somebody else is automatic, you need to manually decorate!

CodePudding user response:

1. Database tuning
2. You can't concurrent threads, see if you have any resource is the exclusive

CodePudding user response:

Write a stored procedure, the processing of your execution on the server end, can a bit faster,

CodePudding user response:

Question is, why do you want to traverse the 1 million data, the purpose of the query is to find a useful, if you input the start purposefully establish database, effective use of time, don't bother, to find something useful for a second what is not -- -- -- -- -- - the most on this,

CodePudding user response:

So every time go to traverse the really good?
2 steps to do this news similarity,
First of all, should be put every news to save to the database, set up the keyword indexing (field)
Then: to compute the similarity of time, and then use SQL statements to screen out ~ ~

CodePudding user response:

The building Lord wham, loading 1 million every time data processing, this also is too far-fetched