I'm trying to code a Decision Tree method for the data in an exoplanet catalogue. It's a worskhop for one of the courses of my Master's studies. I have writen this in an Jupyter Notebook
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn
data = pd.read_csv('exoplanet.eu_catalog_2021.12.15.csv')
data_new = data.select_dtypes(include=['float64'])#Select only dtype float64 data
data_new[~data_new.isin([np.nan, np.inf, -np.inf]).any(1)]
data_new_2 = data_new.loc[:,('mass', 'mass_error_min')]
data_new_2.dropna(subset =["mass_error_min"], inplace = True)
data_new_2.info()
print(data_new_2)
with this result
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1425 entries, 1 to 4892
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 mass 1425 non-null float64
1 mass_error_min 1425 non-null float64
dtypes: float64(2)
memory usage: 33.4 KB
As you can see, there is no empty cells. Besides, I wrote this for converting all the numbers to float64 (just in case!)
data_new_2['mass'] = data_new_2['mass'].astype(float)
data_new_2['mass_error_min'] = data_new_2['mass_error_min'].astype(float)
Then, I splitted the data into the traininig and test subsets
from sklearn.model_selection import train_test_split
X = data_new_2.drop(["mass"], axis = 1)
y = data_new_2["mass"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .30, random_state = 42)
And there is no problem... until this part
from sklearn.tree import DecisionTreeClassifier
classifier = DecisionTreeClassifier()
classifier.fit(X_train, y_train_2)
because I get this error message
ValueError Traceback (most recent call last)
<ipython-input-327-7b81afce3234> in <module>
1 from sklearn.tree import DecisionTreeClassifier
2 classifier = DecisionTreeClassifier()
----> 3 classifier.fit(X_train, y_train_2)
.
.
.
~/.local/lib/python3.6/site-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
104 msg_err.format
105 (type_err,
--> 106 msg_dtype if msg_dtype is not None else X.dtype)
107 )
108 # for object dtype data, we only check for NaNs (GH-13254)
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
I don't understand why this error message appears because I have no Nan, infitnity or "too large" data in X_train nor y_train data.
What can I do?
CodePudding user response:
There are some infinite values in your mass_error_min
column:
data_new_2.describe()
mass mass_error_min
count 1425.000000 1425.0000
mean 6.060956 inf
std 13.568726 NaN
min 0.000002 0.0000
25% 0.054750 0.0116
50% 0.725000 0.0700
75% 3.213000 0.5300
max 135.300000 inf
So, you have to fill those inf with some value, use this code:
value = data_new_2['mass_error_min'].quantile(0.98)
data_new_2 = data_new_2.replace(np.inf, value)