Home > other >  Each node Shared SQLContext problem.
Each node Shared SQLContext problem.

Time:09-21

 
The object Test {
Def main (array: the array [String]) : Unit={
Val conf=new SparkConf (). SetAppName (" test ")
//setMaster (" spark://192.168.2.232:7077 ")
//. SetJars (Array [String] (" file://D:/ideaWorkspace/GDBigDAtaSpark_2/out/artifacts/GDBigDAtaSpark_2_jar/GDBigDAtaSpark_1.jar "))
Val sc=new SparkContext (conf)
Val sparkSQL=new SQLContext (sc)
Val schema=StructType (
StructField (" id ", StringType, false) : :
StructField (" name ", StringType, false) : : Nil
)
Val RDD=sc. TextFile (" HDFS://192.168.2.232:8020/user/hive/warehouse/t_test "). The map (x=& gt; X.s plit (" \ u0001 ")). The map (r=& gt; The Row (r (0), r (1)))
Schema, sparkSQL. CreateDataFrame (RDD) registerTempTable (" t_test ")
Val df=sparkSQL SQL (" select * from t_test ")
//val df=sparkSQL. Value. SQL (" select * from t_test ")
Df. Foreach (row=& gt; {
Println (" s "key=${row (0). The toString}")
The import sparkSQL. Implicits. _
SparkSQL. SQL (s "select * from t_test where id=${row (0). The toString}")
})
}
}

You spark great god help have a look, I want to read in the DataFrame times took another table data, but SQLContext null Pointers, excuse me this how to solve,
sparkSQL. SQL (s "select * from t_test where id=${row (0). The toString}")
This line will be submitted to the empty refers to the abnormal,

CodePudding user response:

Have a great god gives directions, or my programming way and thinking is wrong,

CodePudding user response:

ssssss

CodePudding user response:

Cannot be quoted in the operator SparkContext (including SQLContext), change the idea, you this demand can be simple to implement with SQL join

CodePudding user response:

No reference in the RDD RDD, if another table is not big, variable by radio, otherwise use the join
  • Related