In the spark in the streaming data accessed by the kafka, I (this part is ok)

Then need and to compare the data in the database, the data needs to be read into memory, or check each data library efficiency is too low,

Then the cache every eight data statistics


There is no any ideas, please god to give directions

CodePudding user response:

Database can write a socket sender to flow at the same time interval of frequency with you to query the database, and then put the socket and kafka stream join, can compare, but can't say how many will perform a batch of article, only by time interval or sliding window

CodePudding user response:

I want to cached in memory database, if every time to query efficiency is too low, and the data of 3000 w data is very big

CodePudding user response:

Can use redis for caching, storage, query speed faster

CodePudding user response:

Can use redis for caching, storage, query speed is fast

CodePudding user response:

Can start sparkstreaming to monitor the data before one-time load in memory, again to do Kfaka reading and comparison of data,