I custom SQL functions, a spark of intermediate value is an array, but set the middle type to ArrayType will quote red

Put the ArrayType the superclass DataType is not red, but run error

Don't you can produce in the middle of the type of Array? Great god answer, thank!!!!!!
Here are all the code
The class GameDuration (startDayLong: Long, endDayLong: Long) extends UserDefinedAggregateFunction {

//input the data type of the
Override def inputSchema: StructType=StructType (
StructField (" eventType ", IntegerType) : :
StructField (" timestamp "LongType) : : Nil
)

//intermediate results of type
Override def bufferSchema: StructType=StructType (
StructField (" list ", ArrayType) : : Nil
)

//in the end the results returned type
Override def dataType: dataType=LongType

Override def deterministic: Boolean=true

//specify the initial value
Override def initialize (buffer: MutableAggregationBuffer) : Unit={
Buffer (1)=ArrayBuffer [Long] ()
}

//there is a data involved in operation will update the intermediate results (the update operation in each partition)
Override def update (buffer: MutableAggregationBuffer, input: Row) : Unit={
Val eventType=input. Get int (0)
Val timestamp=input. GetLong (1)
Var TMP=0 l
If (eventType==3) TMP=timestamp
The else TMP=- timestamp

Buffer (1)=buffer. GetAs [ArrayBuffer [Long]] (1) +=TMP
}

//global convergence
Override def merge (buffer1: MutableAggregationBuffer, buffer2: Row) : Unit={
Buffer1 (0)=buffer1. Get int (0) + buffer2. Get int (0)
}

//final computation
Override def the evaluate (buffer: Row) : Long={
Val arr: ArrayBuffer [Long].=buffer getAs [ArrayBuffer [Long]] (1)
If (arr (0) & gt; Arr +=0) - startDayLong
If (arr. Last & lt; +=0) arr endDayLong
Arr. Sum
}
}
And this is what I call the function, do not know whether it is right

CodePudding user response:

O god, and no one

CodePudding user response:

Is anyone here? The great spirit show

CodePudding user response:

Great god save me door

CodePudding user response:

ArrayType for parameter

Def bufferSchema: StructType=StructType (StructField (" linksTemp ArrayType (StringType, true)) : : Nil)

Org. Apache. Spark. SQL. Types. The ArrayType is ArrayType under this package, don't quote wrong