Val result: org. Apache. Spark. RDD. RDD [(String, Int)]
Result. The foreach (res=& gt;
{var put=new put (Java. Util. UUID. RandomUUID (), toString. Reverse. GetBytes ()). The add (" lv6. "getBytes (), res. _1. ToString. GetBytes (), res) _2. ToString. GetBytes) table. The put (put)}
)
Program is above, the result is (key, value) to save Array to hbase error various no serialization
The Exception in the thread "thread - 3" Java. Lang. Reflect. InvocationTargetException
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
At sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 57)
At sun. Reflect. DelegatingMethodAccessorImpl. Invoke (43) DelegatingMethodAccessorImpl. Java:
The at Java. Lang. Reflect. Method. Invoke (606) Method. The Java:
At org. Apache. Spark. Deploy. Yarn. ApplicationMaster $$$2. -anon run (186). ApplicationMaster scala:
Under Caused by: org. Apache. Spark. SparkException: Job aborted due to stage a failure: Task not serializable: Java. IO. NotSerializableException: org.. Apache hadoop, hbase. Client. HTablePool $PooledHTable
The at org.apache.spark.scheduler.DAGScheduler.org $$$$$$failJobAndIndependentStages DAGScheduler scheduler spark apache (DAGScheduler. Scala: 1044)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$abortStage anonfun $1. Apply (DAGScheduler. Scala: 1028)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$abortStage anonfun $1. Apply (DAGScheduler. Scala: 1026)
At the scala. Collection. The mutable. ResizableArray $class. Foreach (ResizableArray. Scala: 59)
At the scala. Collection. Mutable. ArrayBuffer. Foreach (ArrayBuffer. Scala: 47)
The at org. Apache. Spark. The scheduler. DAGScheduler. AbortStage (DAGScheduler. Scala: 1026)
The at org.apache.spark.scheduler.DAGScheduler.org $$$$$$submitMissingTasks DAGScheduler scheduler spark apache (DAGScheduler. Scala: 771)
At org. Apache. Spark. The scheduler. DAGScheduler $$$$16 handleTaskCompletion anonfun $$anonfun $$1. Apply the apply $MCVI $sp (DAGScheduler. Scala: 901)
At org. Apache. Spark. The scheduler. DAGScheduler $$$$$handleTaskCompletion anonfun $16 anonfun $$1. Apply the apply (DAGScheduler. Scala: 898)
At org. Apache. Spark. The scheduler. DAGScheduler $$$$$handleTaskCompletion anonfun $16 anonfun $$1. Apply the apply (DAGScheduler. Scala: 898)
At the scala. Option. Foreach (236) Option. The scala:
The at org. Apache. Spark. The scheduler. DAGScheduler $$$handleTaskCompletion anonfun $16. The apply (DAGScheduler. Scala: 898)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$handleTaskCompletion anonfun $16. The apply (DAGScheduler. Scala: 897)
At the scala. Collection. The mutable. ResizableArray $class. Foreach (ResizableArray. Scala: 59)
At the scala. Collection. Mutable. ArrayBuffer. Foreach (ArrayBuffer. Scala: 47)
The at org. Apache. Spark. The scheduler. DAGScheduler. HandleTaskCompletion (DAGScheduler. Scala: 897)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessActor $$anonfun $receive $2. ApplyOrElse (DAGScheduler. Scala: 1226)
The at akka. Actor. ActorCell. ReceiveMessage (ActorCell. Scala: 498)
The at akka. Actor. ActorCell. Invoke (ActorCell. Scala: 456)
The at akka. Dispatch. Mailbox. ProcessMailbox (Mailbox. Scala: 237)
The at akka. Dispatch. Mailbox. The run (Mailbox. Scala: 219)
The at akka. Dispatch. ForkJoinExecutorConfigurator $AkkaForkJoinTask. Exec (AbstractDispatcher. Scala: 386)
At the scala. Concurrent. Forkjoin. ForkJoinTask. DoExec (ForkJoinTask. Java: 260)
At the scala. Concurrent. Forkjoin. ForkJoinPool $WorkQueue. RunTask (ForkJoinPool. Java: 1339)
At the scala. Concurrent. Forkjoin. ForkJoinPool. RunWorker (ForkJoinPool. Java: 1979)

CodePudding user response:

I'm just begin to contact the spark, the building Lord this problem, my personal advice, take some classes implement the serializable interface.

You appeared to be the spark is integrated with hadoop, don't know whether the hadoop internal some not serialized

CodePudding user response:

Involve the class need to deal with the serialization, if not can abandon foreach, also use other methods instead of http://blog.csdn.net/fighting_one_piece/article/details/38437647
I hope it can help you

CodePudding user response:

Attention, run into the same problem,

CodePudding user response:

Messages. The map (new Function String> () {
@ Override
Public String call (Tuple2 & lt; String, String> Tuple2) {
Return tuple2. _2 ();
}
}). Foreach (new Function2 & lt; JavaRDD , Time, Void> () {

Private HTableInterface table=null;

@ Override
Public Void call (JavaRDD Values, Time Time)
Throws the Exception {

Values. The foreach (new VoidFunction () {

@ Override
Public void call (String STR) throws the Exception {

HConnection connection=HConnectionManager. The createConnection (HBaseConfiguration. The create ());
Table=connection. GetTable (tableName);
String [] strings=SPACE. The split (STR);
String tableName=strings [0];
String type=strings [1];

//if (null==table) {
Connection. GetTable (tableName);
Table. SetAutoFlush (false);
//}

If (type. The equals (" DELETE ")) {
The Delete d=new Delete (strings. [2] getBytes ());
Try {
Table. Delete (d);
} the catch (IOException e) {
e.printStackTrace();
}
} else {
Put the p=new Put (Bytes. ToBytes (strings [2]));
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull