Requirements are as follows:

Prerequisite:
Mysql database with 30 million data, the effective data (need to deal with the data) is about 10%, monthly growth rate of 3 million

Requirements:
1, demand is one-time query out 3 million valid data of the source data to perform a detailed, according to a specific field in the table to three sides (the three interfaces including Dubbo and Rest interface) request execution conditions, the three parties interface data returned after judge whether this data, execution conditions meet, trigger executed rules, does not meet the execution of a
2, the first after the execution, continue to implement the second time, 24 hours do not pause

Current treatment scheme:
Tomcat server cluster 2 each Tomcat ten thread combined twenty thread while loop execution, perform other thread to lock data, this article then ignore

Status:
With data volume increasing effectiveness of this approach is less and perform again the time it takes longer and longer, need to find a more reasonable solution, convenient need server extensions

Speculation can be completed with Storm or Spark cluster, there is no proper way


Please give some ideas for sure, I will here

CodePudding user response:

The frame of the project added: need to integrate Spring and Dubbo

CodePudding user response:

Preempt bits, the feeling after I encountered a similar problem