The object Test {
Def main (array: the array [String]) : Unit={
Val conf=new SparkConf (). SetAppName (" test ") 
//setMaster (" spark://192.168.2.232:7077 ") 
//. SetJars (Array [String] (" file://D:/ideaWorkspace/GDBigDAtaSpark_2/out/artifacts/GDBigDAtaSpark_2_jar/GDBigDAtaSpark_1.jar ")) 
Val sc=new SparkContext (conf) 
Val sparkSQL=new SQLContext (sc) 
Val schema=StructType (
StructField (" id ", StringType, false) : : 
StructField (" name ", StringType, false) : : Nil 
) 
Val RDD=sc. TextFile (" HDFS://192.168.2.232:8020/user/hive/warehouse/t_test "). The map (x=& gt; X.s plit (" \ u0001 ")). The map (r=& gt; The Row (r (0), r (1))) 
Schema, sparkSQL. CreateDataFrame (RDD) registerTempTable (" t_test ") 
Val df=sparkSQL SQL (" select * from t_test ") 
//val df=sparkSQL. Value. SQL (" select * from t_test ") 
Df. Foreach (row=& gt; {
Println (" s "key=${row (0). The toString}") 
The import sparkSQL. Implicits. _ 
SparkSQL. SQL (s "select * from t_test where id=${row (0). The toString}") 
}) 
} 
} 

You spark great god help have a look, I want to read in the DataFrame times took another table data, but SQLContext null Pointers, excuse me this how to solve,
sparkSQL. SQL (s "select * from t_test where id=${row (0). The toString}")
This line will be submitted to the empty refers to the abnormal,

CodePudding user response:

Have a great god gives directions, or my programming way and thinking is wrong,

CodePudding user response:

ssssss

CodePudding user response:

Cannot be quoted in the operator SparkContext (including SQLContext), change the idea, you this demand can be simple to implement with SQL join

CodePudding user response:

No reference in the RDD RDD, if another table is not big, variable by radio, otherwise use the join