Read SparkSQL Hbase table, the DataFrame times wrong operation-CodePudding

-- -- -- -- -- -- -- -- -- -- -- SparkSQL read Hbase tables, operating DataFrame times wrong -- -- -- -- -- -- -- -- -- -- -- --

-- -- -- -- -- -- -- -- -- -- -- -- -- the code is as follows -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

CodePudding user response:

Visual is a line you age is null, try the where age is not null and the age & gt; 25

CodePudding user response:

reference 1st floor link0007 response:

visual line is you have a age is null, try the where age is not null and the age & gt; 25

Table data is not null, and tried the SQL is the same mistake, feeling is not the problem of SQL, seems to be a problem with hbase integration, because no matter which query column, just use the where screening will quote this error,

CodePudding user response:

add:
The same table, if read generated from the MySQL DataFrame and perform operations without any problems,
From Hbase reads, as long as use the where filtering, or use the filter () method, which will be submitted to the above error,

Below is the table data:

CodePudding user response:

refer to the second floor w1123900645 response:

Quote: refer to 1st floor link0007 response:

Visual is a line you age is null, try the where age is not null and the age & gt; 25

Table data is not null, and tried the SQL is the same mistake, feeling is not the problem of SQL, seems to be a problem with hbase integration, because no matter which queries, just use the where screening will quote this error,

You try the Hive hanging HBase appearance form, in the Hive test, again by Spark visit Hive this table

CodePudding user response:

reference 4 floor link0007 response:

Quote: refer to the second floor w1123900645 response:

Quote: refer to 1st floor link0007 response:

Visual is a line you age is null, try the where age is not null and the age & gt; 25

Table data is not null, and tried the SQL is the same mistake, feeling is not the problem of SQL, seems to be a problem with hbase integration, because no matter which queries, just use the where screening will quote this error,

You try the Hive hanging HBase appearance form, in the Hive, test again by Spark visit Hive this table

Thank you, I have not used Hive,
Above is to use Hbase - Spark. Third party dependent jar jar package, it may not be able to solve, I guess is version compatibility problems, so I give up this way,
Then I tried to read into RDD Hbase table, then use's official website, programmatically specify Schema, converting the RDD to DataFrame way, it's no problem,
Although a lot of complex code, flexibility also is very poor, but this can be used this way for a while,

CodePudding user response:

Spark - hbase already for a long time not updated, that don't use

CodePudding user response:

reference 5 floor w1123900645 reply:

Quote: refer to 4th floor link0007 response:

Quote: refer to the second floor w1123900645 response:

Quote: refer to 1st floor link0007 response:

Visual is a line you age is null, try the where age is not null and the age & gt; 25

Table data is not null, and tried the SQL is the same mistake, feeling is not the problem of SQL, seems to be a problem with hbase integration, because no matter which queries, just use the where screening will quote this error,

You try the Hive hanging HBase appearance form, in the Hive, test again by Spark visit Hive this table

Thank you, I have not used Hive,
Above is to use Hbase - Spark. Third party dependent jar jar package, it may not be able to solve, I guess is version compatibility problems, so I give up this way,
Then I tried to read into RDD Hbase table, then use's official website, programmatically specify Schema, converting the RDD to DataFrame way, it's no problem,
Although a lot of complex code, flexibility also is very poor, but this can be used this way for a while,

Advice or a Hive as the unity of HDFS is structured appearance, our side through the Hive access hbase appearance is no problem, the spark to access