RDD. Map reference external val error & quot; The Task not serializable"-CodePudding

 
Val Size=5 
Val fixedLogs=RDD. Map {row=& gt; {
The Size 
} 
}

Statement in the spark - shell variable val Size=5; References in the map, then an error "Task not serializable" what should I do

Thank you for the great god

CodePudding user response:

Val bcSize=sc. BroadCast (size)
Val fixedLogs=RDD. Map {row=& gt; {
BcSize. Value
}
}

CodePudding user response:

Because you RDD transform and the action of operator is performed on the executor, and you write the main method, is run on driver, so you can't cross machine across processes to visit you in the driver's size, to access, can only use radio variable, this is on each executor to create a copy of the object being broadcast,

CodePudding user response:

I can't, or an error

CodePudding user response:

I am using SQL dataFrame. RDD. Map
The following code still not serialize

Val bcSize=sc. Broadcast (5)
Val test=dayErrorDF. Select (" judge. * ", "fix. *"). The RDD. Map {row=& gt; BcSize. Value
}

reference 1st floor link0007 response:

val bcSize=sc. The broadCast (size)
Val fixedLogs=RDD. Map {row=& gt; {
BcSize. Value
}
}

CodePudding user response:

Scala> Val df=sqlContext range (1100)
Df: org. Apache. Spark. SQL. DataFrame=[id: bigint]

Scala> Val bcSize=sc. Broadcast (5)
BcSize: org. Apache. Spark. Broadcast. Broadcast (Int)=broadcast (0)

Scala> Val m=df. RDD. Map {row=& gt; BcSize. Value}
M: org. Apache. Spark. RDD. RDD (Int)=MapPartitionsRDD [6] at the map at & lt; Console> 31:

Scala> M.c mount
Res2: Long=999

No problem

CodePudding user response:

reference 4 floor baidu_20778985 response:

I am using SQL dataFrame. RDD. Map
The following code still not serialize

Val bcSize=sc. Broadcast (5)
Val test=dayErrorDF. Select (" judge. * ", "fix. *"). The RDD. Map {row=& gt; BcSize. Value
}

Quote: refer to 1st floor link0007 response:

Val bcSize=sc. BroadCast (size)
Val fixedLogs=RDD. Map {row=& gt; {
BcSize. Value
}
}

DayErrorDF. Select (" judge. * ", "fix. *)" I think it is the part of the problem, I have not seen this kind of writing,
You alone dayErrorDF. Select (" judge. * ", "fix. *"), will show to see an error

CodePudding user response:

refer to 6th floor link0007 response:

Quote: refer to 4th floor baidu_20778985 response:

I am using SQL dataFrame. RDD. Map
The following code still not serialize

Val bcSize=sc. Broadcast (5)
Val test=dayErrorDF. Select (" judge. * ", "fix. *"). The RDD. Map {row=& gt; BcSize. Value
}

Quote: refer to 1st floor link0007 response:

Val bcSize=sc. BroadCast (size)
Val fixedLogs=RDD. Map {row=& gt; {
BcSize. Value
}
}

DayErrorDF. Select (" judge. * ", "fix. *)" I think it is the part of the problem, I have not seen this kind of writing,
You alone dayErrorDF. Select (" judge. * ", "fix. *"), will show to see an error

Can display, it is a nested, remove the ". * "also not line, very strange,

CodePudding user response:

refer to 7th floor baidu_20778985 response:

Quote: refer to the sixth floor link0007 response:

Quote: refer to 4th floor baidu_20778985 response:

I am using SQL dataFrame. RDD. Map
The following code still not serialize

Val bcSize=sc. Broadcast (5)
Val test=dayErrorDF. Select (" judge. * ", "fix. *"). The RDD. Map {row=& gt; BcSize. Value
}

Quote: refer to 1st floor link0007 response:

Val bcSize=sc. BroadCast (size)
Val fixedLogs=RDD. Map {row=& gt; {
BcSize. Value
}
}

DayErrorDF. Select (" judge. * ", "fix. *)" I think it is the part of the problem, I have not seen this kind of writing,
You alone dayErrorDF. Select (" judge. * ", "fix. *"), will show to see an error

Can display, it is a nested, remove the ". * "also not line, it's strange that

You refresh my through this problem,,, spark version is how much?

CodePudding user response:

refer to the eighth floor link0007 response:

Quote: refer to 7th floor baidu_20778985 response:

Quote: refer to the sixth floor link0007 response:

Quote: refer to 4th floor baidu_20778985 response:

I am using SQL dataFrame. RDD. Map
The following code still not serialize

Val bcSize=sc. Broadcast (5)
Val test=dayErrorDF. Select (" judge. * ", "fix. *"). The RDD. Map {row=& gt; BcSize. Value
}

Quote: refer to 1st floor link0007 response:

Val bcSize=sc. BroadCast (size)
Val fixedLogs=RDD. Map {row=& gt; {
BcSize. Value
}
}

DayErrorDF. Select (" judge. * ", "fix. *)" I think it is the part of the problem, I have not seen this kind of writing,
nullnullnullnullnullnullnullnullnullnullnull