Now I have a lot of tasks, such as statistical data for the latest day every ten seconds, which according to different dimensions, there are many kinds of statistical methods, different conditions but each task actually needed data are the same, is the last day of the data (not natural day is 24 hours before the current time), if the data is in memory of each task is a waste of space, increase after tasks are going to have a problem, so have a Shared space can read? Let all the tasks are to read the same piece of Shared data,

I have a question can be directly read in sparkstreaming mongo's data?

CodePudding user response:

If you can make the cluster itself so much homework to run at the same time, you can put the job processing methods in class, and then each job to open a thread, each thread processing, and then open dynamicAllocation, then execute multiple jobs at the same time in an application