Line Fitting using LinearRegression in Pyspark gives wildly different coeffecients-CodePudding

I have a dataframe like so:

 --------- ------------------ 
|rownumber|      Moving_Ratio|
 --------- ------------------ 
|     1000|105.67198820168865|
|     1001|105.65729748456914|
|     1002| 105.6426671752822|
|     1003|105.62808965618223|
|     1004|105.59623035662119|
|     1005|105.52385366516299|
|     1006|105.44762361744378|
|     1007|105.35977134665733|
|     1008|105.25685407339793|
|     1009|105.16307473993363|
|     1010|105.06600545864703|
|     1011|104.96056753478364|
|     1012|104.84525664217107|
|     1013| 104.7401615868953|
|     1014| 104.6283459710509|
|     1015|104.53484736833259|
|     1017|104.43492576734955|
|     1019|104.33599903547659|
|     1020|104.24640223269283|
|     1021|104.15275303890549|
 --------- ------------------

There are 10k rows, I've just truncated it for the sample view. The data is by no means linear and looks like this:

However, I'm not worried about a perfect fit for each and every data point. I'm basically looking to fit a line that captures the direction of the curve and find its slope. As shown by the green line in the image that was generated by a statistics software.

The feature column I'm trying to fit in a line is Moving_Ratio The min and max values of Moving_Ratio are:

 ----------------- ------------------ 
|min(Moving_Ratio)| max(Moving_Ratio)|
 ----------------- ------------------ 
|26.73629202745194|121.84100616620908|
 ----------------- ------------------

I tried creating a simple linear model with the following code:

vect_assm = VectorAssembler(inputCols =['Moving_Ratio'], outputCol='features')
df_vect=vect_assm.transform(df)\


lir = LinearRegression(featuresCol = 'features', labelCol='rownumber', maxIter=50, 
regParam=0.3, elasticNetParam=0.8)
model = lir.fit(df_vect)
Predictions = model.transform(df_vect)

coeff=model.coefficients

When I look at the predictions, I seem to be getting values nowhere near the original data corresponding to those rownumbers.

Predictions.show()

 --------- ------------------ -------------------- ----------------- 
|rownumber|      Moving_Ratio|            features|       prediction|
 --------- ------------------ -------------------- ----------------- 
|     1000|105.67198820168865|[105.67198820168865]|8935.419272488462|
|     1001|105.65729748456914|[105.65729748456914]| 8934.20373303444|
|     1002| 105.6426671752822| [105.6426671752822]|8932.993191845864|
|     1003|105.62808965618223|[105.62808965618223]|8931.787018623438|
|     1004|105.59623035662119|[105.59623035662119]|8929.150916159619|
|     1005|105.52385366516299|[105.52385366516299]|  8923.1623232745|
|     1006|105.44762361744378|[105.44762361744378]|8916.854895949407|
|     1007|105.35977134665733|[105.35977134665733]| 8909.58582253401|
|     1008|105.25685407339793|[105.25685407339793]|8901.070240542358|
|     1009|105.16307473993363|[105.16307473993363]|8893.310750051145|
|     1010|105.06600545864703|[105.06600545864703]|8885.279042666287|
|     1011|104.96056753478364|[104.96056753478364]| 8876.55489697866|
|     1012|104.84525664217107|[104.84525664217107]|8867.013842017961|
|     1013| 104.7401615868953| [104.7401615868953]|8858.318065966234|
|     1014| 104.6283459710509| [104.6283459710509]|8849.066217228752|
|     1015|104.53484736833259|[104.53484736833259]|8841.329954963563|
|     1017|104.43492576734955|[104.43492576734955]|8833.062240915566|
|     1019|104.33599903547659|[104.33599903547659]|8824.876844336828|
|     1020|104.24640223269283|[104.24640223269283]|8817.463424838508|
|     1021|104.15275303890549|[104.15275303890549]| 8809.71470236567|
 --------- ------------------ -------------------- ----------------- 

Predictions.select(min('prediction'),max('prediction')).show()
 ----------------- ------------------ 
|  min(prediction)|   max(prediction)|
 ----------------- ------------------ 
|2404.121157489531|10273.276308929268|
 ----------------- ------------------ 

coeff[0]
82.74200940195973

The min and max of the predictions are completely outside the input data. What am I doing wrong? Any help will be greatly appreciated

CodePudding user response：

When you initialize LinearRegression object, featuresCol should list all features (independent variable) and labelCol should list the label (dependent variable). Since you are predicting 'Moving_Ratio', set up featuresCol='rownumber' and labelCol='Moving_Ratio' to specify LinearRegression correctly.