Trying to remove item from a list of pd.Series using pop but not working properly-CodePudding

I'm trying to make a class to find outliers in a dataset using statistics methods like interquantile range and zscore. I would like to know why some outliers get removed from my list of pd.Series and some not given the empty condition.

class OutliersDetector:
  def __init__(self, X):
    self.outliers = []
    self.X = X

  def detect_range(self):
    self.reset_outliers()
    # to implement
    self.remove_empty_items()

  def detect_zscore(self):
    self.reset_outliers()
    zscore = np.abs(stats.zscore(self.X))
    threshold_std = 3
    for index, col_name in enumerate(self.X.columns): # X and zscore always have the same shape
      col = zscore[:, index]
      self.outliers.append( pd.Series(col[col >= threshold_std], name=col_name) )
    self.remove_empty_items()

  # none of the if statements i tried worked
  def remove_empty_items(self):
    for index, item in enumerate(self.outliers):
      #if item.size == 0:
      #if len(item.index) == 0:
      if item.empty:
        print("[no outliers] {}".format(item.name))
        self.outliers.pop(index)

  def reset_outliers(self):
    self.outliers = []

  def show_outliers(self):
    for item in self.outliers:
      print("[name]: {}\n[outliers]: {}\n".format(item.name, item.size))

outliers_detector = OutliersDetector(X_train_transformed)
outliers_detector.detect_zscore()
print("\noutliers found: ")
outliers_detector.show_outliers()

Output: Rainfall, Month, Location, WindDir9a should no be printed below "outliers found" because has 0 size but...

[no outliers] RainToday
[no outliers] Year
[no outliers] Day
[no outliers] WindGustDir
[no outliers] WindDir3pm
[no outliers] Sunshine
[no outliers] Humidity3pm
[no outliers] Cloud9am

outliers found:
[name]: Rainfall
[outliers]: 0

[name]: Evaporation
[outliers]: 289

[name]: Month
[outliers]: 0

[name]: Location
[outliers]: 0

[name]: WindDir9am
[outliers]: 0

How can I fix this?

CodePudding user response：

In remove_empty_items you are modifying the self.outliers list while iterating over it. This causes undefined behaviour. Your code should create a new list instead of modifying the current one in place:

  def remove_empty_items(self):
    non_empty_outliers = []
    for item in self.outliers:
      if item.empty:
        print("[no outliers] {}".format(item.name))
      else:
        non_empty_outliers.append(item)
    self.outliers = non_empty_outliers