Home > other >  RDD map operation why focus on an executor implementation
RDD map operation why focus on an executor implementation

Time:10-15

Python wrote a very simple program:
The conf=Sparkconf (.) setMaster (" spark://123.43.54.12:7077 "). The setAppname (" test ")
Sc=SparkContext (conf=conf)
Def ct (p) :
Temp=0
For I in range (10000) :
Tempi=math. Sin (I) + math.h cos (I)
Temp=temp + tempi
Return temp

P_list=list (range (100000))
P_rdd=sc. Parallelize (p_list, 4)
gt_value=https://bbs.csdn.net/topics/p_rdd.map (lambda p: ct (p)). Coalesce (144)

Sc. Stop ()

Every execution, found that only run on a executor, perform at a time, and the worker may change the address, can't use all of the actuator,
Submit the grammar:
Submit - master spark://123.43.54.12:7077 - deploy - mode client - num - executors 2 - executor - cores 1

Is very strange, the operating system's logical cpu3,,
  • Related