As title, that is, multiple nodes can access to, and then can go to update this variable, after you have updated, other nodes access is also the latest value, spark have this kind of solution? Shared variables are read-only, seemingly can't

CodePudding user response:

Radio variable sc. Broadcast ()?

CodePudding user response:

Spark provides the accumulative variables, see the official made it very clear

CodePudding user response:

LZ, please find the solution

CodePudding user response:

The broadcast on radio variables from the Driver object serialization and deserialization on the Executor, therefore in the Executor of the variable changes, just modify the Executor of the local copy of a copy of the other Executor will not be affected,
The accumulator is Driver read-only, Executor writing object, only
So you can do through database or HDFS file, and thus the real-time share the data encapsulation method inside the operator to access your visits, attention to the problem of read-write lock,

CodePudding user response:

In addition if not particularly large objects, such as string or an array, not a collection of attention can be serialized on the ZooKeeper, ZK data cannot be more than 1 MB
Recommend a menagerie, realized the distributed lock, Leader election, and implements the Java. Util. Concurrent package under the concurrent collections of the distributed implementation of the interface

CodePudding user response:

Themselves can encapsulate implementation, broadcast and accumulated variables cannot be as you say it,

Put to Shared data zookeeper is a way, is to control concurrent write the problems by themselves, not the trouble, and then open a web service, every time get the value from the web service, finished the update,