Please comment:
Val sqlContext=new sqlContext (sparkContext)
Val result=RDD
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select id from the table ")
},
1)In RDD map, sqlContext output is empty? Consult the reason? Thank you very much ~ ~

CodePudding user response:

Val sqlContext=new sqlContext (sparkContext)
Val result=RDD
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select id from table
")},
1)The table you created?

CodePudding user response:

The table creation,
Overall code in general is this:
Val conf=new SparkConf ()
Val sparkContext=new sparkContext (conf)
Val sqlContext=new sqlContext (sparkContext)

Val srcTable=SRC. Table (srcTbl). The cache ()
Val iptables=IP. Table (ipTbl). The cache ()

Val src_rdd=srcTable. Select (" type ", "date", "IP"). The toDF (" type ", "date", "IP"). The cache ()
Val ip_rdd=iptables. Filter (iptables (" country ")===1). The filter (iptables (" province ")!==0). Select (" start_ip end_ip ""," province "). The cache ()
Ip_rdd. RegisterTempTable (" ip_rdd ")

Val result=src_rdd
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select province from ip_rdd where start_ip & lt;=1032301638 and end_ip & gt;=1032301638 order by province limit 1 "). Head (). The toString ()
PvMap (x). The concat (value)
PvMap (x)
}, 1))
ReduceByKey (+ _ _)
Here sqlContext, in the process of the Map on the print, I was empty, but in the val result=src_rdd. Map out call sqlContext. SQL is normal,

CodePudding user response:

reference 1st floor link0007 response:

val sqlContext=new sqlContext (sparkContext)
Val result=RDD
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select id from table
")},
1)The table you created?

The table creation,
Overall code in general is this:
Val conf=new SparkConf ()
Val sparkContext=new sparkContext (conf)
Val sqlContext=new sqlContext (sparkContext)

Val srcTable=SRC. Table (srcTbl). The cache ()
Val iptables=IP. Table (ipTbl). The cache ()

Val src_rdd=srcTable. Select (" type ", "date", "IP"). The toDF (" type ", "date", "IP"). The cache ()
Val ip_rdd=iptables. Filter (iptables (" country ")===1). The filter (iptables (" province ")!==0). Select (" start_ip end_ip ""," province "). The cache ()
Ip_rdd. RegisterTempTable (" ip_rdd ")

Val result=src_rdd
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select province from ip_rdd where start_ip & lt;=1032301638 and end_ip & gt;=1032301638 order by province limit 1 "). Head (). The toString ()
PvMap (x). The concat (value)
PvMap (x)
}, 1))
ReduceByKey (+ _ _)
Here sqlContext, in the process of the Map on the print, I was empty, but in the val result=src_rdd. Map out call sqlContext. SQL is normal,

CodePudding user response:

reference sufengxue2012 reply: 3/f

Quote: refer to 1st floor link0007 response:

Val sqlContext=new sqlContext (sparkContext)
Val result=RDD
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select id from table
")},
1)The table you created?

The table creation,
Overall code in general is this:
Val conf=new SparkConf ()
Val sparkContext=new sparkContext (conf)
Val sqlContext=new sqlContext (sparkContext)

Val srcTable=SRC. Table (srcTbl). The cache ()
Val iptables=IP. Table (ipTbl). The cache ()

Val src_rdd=srcTable. Select (" type ", "date", "IP"). The toDF (" type ", "date", "IP"). The cache ()
Val ip_rdd=iptables. Filter (iptables (" country ")===1). The filter (iptables (" province ")!==0). Select (" start_ip end_ip ""," province "). The cache ()
Ip_rdd. RegisterTempTable (" ip_rdd ")

Val result=src_rdd
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select province from ip_rdd where start_ip & lt;=1032301638 and end_ip & gt;=1032301638 order by province limit 1 "). Head (). The toString ()
PvMap (x). The concat (value)
PvMap (x)
}, 1))
ReduceByKey (+ _ _)
Here sqlContext, in the process of the Map on the print, I was empty, but in the val result=src_rdd. Map out call sqlContext. SQL is normal,

I see your SQL result should be static, broadcast went out on the map to access the value line, it is not necessary to use sqlContext

CodePudding user response:

reference 4 floor link0007 response:

Quote: reference sufengxue2012 reply: 3/f

Quote: refer to 1st floor link0007 response:

Val sqlContext=new sqlContext (sparkContext)
Val result=RDD
The map (x=& gt; ({
Var value: String=sqlContext. SQL (" select id from table
")},
1)The table you created?

The table creation,
Overall code in general is this:
Val conf=new SparkConf ()
Val sparkContext=new sparkContext (conf)
Val sqlContext=new sqlContext (sparkContext)

Val srcTable=SRC. Table (srcTbl). The cache ()
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull