Isolation Forest

Isolation Forest is a novel anomaly detection approach designed to locate outliers or abnormalities in a dataset. Unlike previous approaches that profile normal data points, Isolation Forest offers a new approach, identifying abnormalities directly. The core premise of Isolation Forest is founded on the notion that anomalies are usually uncommon and distinct from regular cases, making them simpler to separate.

The Isolation Forest workflow entails creating a set of isolation trees, with each tree constructed by randomly choosing characteristics and splitting data points until each point is isolated in its own leaf node. Anomalies are intended to need fewer partitions to isolate than typical instances, making them easier to identify based on lower average path lengths across all trees.

Code:

Now, we will try to find and eliminate outliers (anomalies) elegantly with the help of Isolation Forest.

Importing Libraries

import pandas as pd
import os
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import warnings

warnings.filterwarnings('ignore')
pd.reset_option('^display.', silent=True)

Reading the Dataset

# Load the two datasets
X_train = pd.read_csv("../input/house-prices-advanced-regression-techniques/train.csv")
X_test = pd.read_csv("../input/house-prices-advanced-regression-techniques/test.csv")

# Seperate independent and dependent variable
num_train = len(X_train)
num_test = len(X_test)
y_train = X_train.SalePrice
X_train.drop(['SalePrice'], axis=1, inplace=True)

# Merge train and test data to simplify processing
df = pd.concat([X_train, X_test], ignore_index=True)

# Rename odd-named columns
df = df.rename(columns={"1stFlrSF": "FirstFlrSF",
                        "2ndFlrSF": "SecondFlrSF",
                       "3SsnPorch": "ThirdSsnPorch"})

# Shopw 5 samples
df.head()

Output:

The next step is some minor pre-processing before we can run IsolationForest. We remove columns with a high number of NaNs (>1000) and fill in missing values for all features. Finally, we validate that there is no missing data.

# Find columns with more than 1000 NaN's and drop them (see above)
columns = [col for col in df.columns if df[col].isnull().sum() > 1000]
df = df.drop(columns, axis=1)

# Fill LotFrontage with the median
df['LotFrontage'].fillna((df['LotFrontage'].mean()), inplace=True)

# No garage values means no year, area, or cars
for col in ['GarageYrBlt', 'GarageArea', 'GarageCars']:
    df[col] = df[col].fillna(0)
    
# No garage info means you don't have one
for col in ['GarageType', 'GarageFinish', 'GarageQual', 'GarageCond']:
    df[col] = df[col].fillna('None')

# Fill no basement
for col in ['BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2']:
    df[col] = df[col].fillna('None')

# Fill remaining categorical and numerical cols with None and 0
cat_columns = df.select_dtypes('object').columns
num_columns = [i for i in list(df.columns) if i not in cat_columns]
df.update(df[cat_columns].fillna('None'))
df.update(df[num_columns].fillna(0))

# Check for missing values 
print(df.isnull().values.any())

Output:

Isolation Algorithm

The Isolation Forest (IF) approach performs best when trees are generated from a subset of the data set rather than the complete data set. This is quite different from virtually all other strategies, which rely on data and require more of it for increased accuracy. Sub-sampling works wonders in this method since regular instances might disrupt the isolation process by being closer to outliers. In this example, we set max_samples=100, causing Isolation Forest to generate 100 samples to train the base estimator for each feature.

fit: fits the base estimator with the max_samples count for the specific feature.
predict: returns -1 if the observation is an outlier and 1 otherwise.
decision_function: computes the measured outlier score using the fitted model.

The stats dataframe just contains the original sample values, their scores, whether IF considers them an outlier or not, and some basic feature statistics such as min, max, and median values.

from sklearn.ensemble import IsolationForest
rng = np.random.RandomState(0)

# Helper function to train and predict IF model for a feature
def train_and_predict_if(df, feature):
    clf = IsolationForest(max_samples=100, random_state=rng)
    clf.fit(df[[feature]])
    pred = clf.predict(df[[feature]])
    scores = clf.decision_function(df[[feature]])
    stats = pd.DataFrame()
    stats['val'] = df[feature]
    stats['score'] = scores
    stats['outlier'] = pred 
    stats['min'] = df[feature].min()
    stats['max'] = df[feature].max()
    stats['mean'] = df[feature].mean()
    stats['feature'] = [feature] * len(df)
    return stats

# Helper function to print outliers
def print_outliers(df, feature, n):
    print(feature)
    print(df[feature].head(n).to_string(), "\n")

# Run through all features and save the outlier scores for each feature
num_columns = [i for i in list(df.columns) if i not in list(df.select_dtypes('object').columns) and i not in ['Id']]
result = pd.DataFrame()
for feature in num_columns:
    stats = train_and_predict_if(df, feature)
    result = pd.concat([result, stats])
    
# Gather top outliers for each feature
outliers = {team:grp.drop('feature', axis=1) 
       for team, grp in result.sort_values(by='score').groupby('feature')}

# Print the top 10 outlier samples for a few selected features
n_outliers = 10
print_outliers(outliers, "LotArea", n_outliers)
print_outliers(outliers, "YearBuilt", n_outliers)
print_outliers(outliers, "BsmtUnfSF", n_outliers)
print_outliers(outliers, "GarageYrBlt", n_outliers)

Output:

Let's have a look at the results.

LotArea has four major outliers (-1) with an outlier score of around -0.33 and values more than 100,000. These are distant from the average value of 10168 for that attribute. We can see that LotArea ranges from 1300 to 215245, thus lowering the influence that these four observations (out of a total of) have on the variance of this feature may aid our modeling approach later on.
YearBuilt is less varied than LotArea, with a maximum outlier score of around -0.25. This demonstrates that values are not too distant from the mean. The future's lowest values (~1880) were identified as outliers using IF.
BsmtUnfSF resembles YearBuilt but has a significantly larger variance.
GarageYrBlt plainly contains outliers at 0 val according to IF, but this is understandable given that these observations were made without any garage. The majority of residences have a GarageYrBlt, and they depart significantly from the average.

Next, we utilize pandas' clipping capability to remove outliers at input levels. It operates by setting a minimum and maximum value for that particular characteristic. All observations with a value less than the minimum will be allocated the min, while all observations with a value more than the max will be assigned the maximum. These are simply examples; you can change the settings as you see fit.

# Use clipping to level out the observed outliers in the data
df.LotArea = df.LotArea.clip(1300,50000)
df.YearBuilt = df.YearBuilt.clip(1880,2010)
df.BsmtUnfSF = df.BsmtUnfSF.clip(100,1900)
df.GarageYrBlt = df.GarageYrBlt.clip(0,2020)

We now retrain the IsolationForest classifier to test if clipping the values improves the outlier ratings. Notice how the outlier scores generated by IsolationForest for the example features have decreased after we clipped them.

# Train IsolationForest again, this time after with clipped outliers for 4 features
result = pd.DataFrame()
for feature in num_columns:
    stats = train_and_predict_if(df, feature)
    result = pd.concat([result, stats])

# Gather top outliers for each feature
outliers = {team:grp.drop('feature', axis=1) 
       for team, grp in result.sort_values(by='score').groupby('feature')}


# Print the top 10 outlier samples for the features we chose to clip
n_outliers = 10
print_outliers(outliers, "LotArea", n_outliers)
print_outliers(outliers, "YearBuilt", n_outliers)
print_outliers(outliers, "BsmtUnfSF", n_outliers)
print_outliers(outliers, "GarageYrBlt", n_outliers)

Output:

We demonstrated how Isolation Forest may be used to detect outliers in a dataset. We utilized the housing price data set as an example. Isolation Forest thrives on subsampled data and does not require building the tree from the complete dataset. It performs well with subsampled data. The technique runs very quickly since it does not rely on computationally expensive operations such as distance or density calculation. The training step has a linear time complexity and a low constant, making it suitable for any large-scale data processing application.

Next TopicMcNemar Test

← prev next →