Home > other >  Spark read hbase null pointer exception, a great god guide genuflect is begged
Spark read hbase null pointer exception, a great god guide genuflect is begged

Time:09-18

The spark version: 1.2.1
Hbase version: 0.98

Import org, apache hadoop. Hbase. HBaseConfiguration
Import org, apache hadoop. Hbase. Graphs. TableInputFormat
The import org. Apache. Spark. SparkConf
The import org. Apache. Spark. SparkContext

The object HBaseTest {
Def main (args: Array [String]) {
Val sparkConf=new sparkConf (). SetAppName (" HBaseTest ")
Val sc=new SparkContext (sparkConf)
Val conf=HBaseConfiguration. The create ()

//hbase - site. XML has added to the project, and packaging to the jar package, the same configuration, the local write an application reads the hbase can be successful, with the spark will not make the
The conf. Set (" hbase. Zookeeper. Quorum ", "node1, 2, node3")
The conf. Set (" hbase. Zookeeper. Property. ClientPort ", "2181")
The conf. Set (" hbase. Master ", "node1:60000")

The conf. Set (TableInputFormat INPUT_TABLE, "student") has been created in//hbase student table

Val hBaseRDD=sc. NewAPIHadoopRDD (conf, classOf [TableInputFormat],
ClassOf [org. Apache hadoop. Hbase. IO. ImmutableBytesWritable],
ClassOf [. Org. Apache hadoop, hbase client. The Result])

Val count=hBaseRDD. The count ()
Println (" the Users RDD Coun: "+ count)
Sc. Stop ()
}
}

Submission is as follows:
the./bin/spark - submit - class HBaseTest - master the local [2] - num - executors 3 - driver - 512 MB of memory, executor - 512 MB of memory, executor - cores 1/usr/local/share/testhbase jar


the following exception, apparently didn't even on Hbase
15/07/20 20:59:45 INFO RecoverableZooKeeper: the Process identifier=hconnection - 0 x633e79e7 connecting to ZooKeeper ensemble=2:2181, node1:2181, node3:2181
15/07/20 20:59:45 INFO ClientCnxn: Opening a socket connection to the server node2.fd.h3c.com/192.38.18.102:2181. Will not attempt to authenticate using SASL (unknown error)
15/07/20 20:59:45 INFO ClientCnxn: Socket connection established to node2.fd.h3c.com/192.38.18.102:2181, initiating the session
15/07/20 20:59:45 INFO ClientCnxn: Session establishment complete on server node2.fd.h3c.com/192.38.18.102:2181, sessionid=0 x24e9fc3f322001a, negotiated a timeout=40000
15/07/20 20:59:45 INFO ZooKeeperRegistry: ClusterId read in ZooKeeper is null
15/07/20 20:59:45 INFO RegionSizeCalculator: Calculating region sizes for the table "student".
Error: application failed with exception
Java. Lang. RuntimeException: Java. Lang. NullPointerException
At org, apache hadoop. Hbase. Client. RpcRetryingCaller. CallWithoutRetries (RpcRetryingCaller. Java: 209)
At org, apache hadoop. Hbase. Client. ClientScanner. Call (288) ClientScanner. Java:
At org, apache hadoop. Hbase. Client. ClientScanner. NextScanner (ClientScanner. Java: 268)
At org, apache hadoop. Hbase. Client. ClientScanner. InitializeScannerInConstruction (ClientScanner. Java: 140)
At org, apache hadoop. Hbase. Client. ClientScanner. & lt; init> (ClientScanner. Java: 135)
At org, apache hadoop. Hbase. Client. HTable. GetScanner (HTable. Java: 802)
At org, apache hadoop. Hbase. Client. MetaScanner. MetaScan (MetaScanner. Java: 200)
At org, apache hadoop. Hbase. Client. MetaScanner. MetaScan (MetaScanner. Java: 85)
At org, apache hadoop. Hbase. Client. MetaScanner. AllTableRegions (MetaScanner. Java: 310)
At org, apache hadoop. Hbase. Client. HTable. GetRegionLocations (HTable. Java: 666)
At org, apache hadoop. Hbase. Util. RegionSizeCalculator. & lt; init> (RegionSizeCalculator. Java: 79)
At org, apache hadoop. Hbase. Util. RegionSizeCalculator. & lt; init> (RegionSizeCalculator. Java: 64)
At org, apache hadoop. Hbase. Graphs. TableInputFormatBase. GetSplits (TableInputFormatBase. Java: 160)
The at org. Apache. Spark. RDD. NewHadoopRDD. GetPartitions (NewHadoopRDD. Scala: 98)
The at org. Apache. Spark. RDD. RDD $$$partitions anonfun $2. Apply (RDD. Scala: 220)
The at org. Apache. Spark. RDD. RDD $$$partitions anonfun $2. Apply (RDD. Scala: 218)
At the scala. Option. GetOrElse (120) Option. The scala:
The at org. Apache. Spark. RDD. RDD. Partitions (RDD. Scala: 218)
The at org. Apache. Spark. SparkContext. RunJob (SparkContext. Scala: 1335)
The at org. Apache. Spark. RDD. RDD. Count (RDD. Scala: 925)
The at HBaseTest $. The main (HBaseTest. Scala: 27)
The at HBaseTest. Main (HBaseTest. Scala)
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
At sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 57)
At sun. Reflect. DelegatingMethodAccessorImpl. Invoke (43) DelegatingMethodAccessorImpl. Java:
The at Java. Lang. Reflect. Method. Invoke (606) Method. The Java:
The at org. Apache. Spark. Deploy. SparkSubmit $. Launch (SparkSubmit. Scala: 367)
The at org. Apache. Spark. Deploy. SparkSubmit $. The main (SparkSubmit. Scala: 77)
The at org. Apache. Spark. Deploy. SparkSubmit. Main (SparkSubmit. Scala)
Under Caused by: Java. Lang. NullPointerException
At org, apache hadoop. Hbase. Zookeeper. ZooKeeperWatcher. GetMetaReplicaNodes (ZooKeeperWatcher. Java: 269)
At org, apache hadoop. Hbase. Zookeeper. MetaRegionTracker. BlockUntilAvailable (MetaRegionTracker. Java: 241)
At org, apache hadoop. Hbase. Client. ZooKeeperRegistry. GetMetaRegionLocation (ZooKeeperRegistry. Java: 62)
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull
  • Related