can spark and hadoop deployment respectively in two different physical host cluster?
That spark deployed in a pile of hard disk is very small, but the memory of the host name great physical host spark1 - n, hadoop deployment in a pile of hard disk is bigger, other memory generally on the host, the host name hadoop1 - n, spark after deployment of the default remote access to the hadoop cluster, become an independent spark computing cluster, hadoop is it calculated the data source and data results of cases, from the physical machine level is separate, but all hosts in the same room, network communication speed there is no problem,
main want to reach the spark is not with the original hadoop cluster hive for memory, also can make full use of its advantages in calculation, the purpose of

Could you tell me whether it can be deployed, should pay attention to what specific details, the spark should pay attention to what parts when the configuration,

CodePudding user response:

It comes to spark calculation using the hadoop cluster data, will involve a lot of the server, the data transmission between the disguised prolong processing time; As to spark and hive competition memory, you can configure the slaves of the spark, it is ok to get rid of the hive is node.. Another spark calculation if a large amount of data involved, the memory is not enough to use, data will be temporary cache to disk, so disk small also is not good..

CodePudding user response:

If the spark when managed to yarn, you can configure each node performing the duty of the spark the biggest memory use space, does not necessarily seized all the memory on the server, if you can can use docker container model.

CodePudding user response:

Must be a cluster

CodePudding user response:

Can spark and hadoop deployment respectively in two different physical host on a cluster, the measured at www.jiaoyidao.net

CodePudding user response:

Can deploy completely no problem, I now is such a deployment, your spark task should not managed to yarn,