Strategies for Missing Data: Imputation vs Deletion

Okay, you've identified the missing values (NaNs) lurking in your dataset using methods from the previous section. Now comes the decision point: what should you do about them? Ignoring missing data is rarely an option, as most analytical algorithms and visualization tools don't handle NaN values gracefully. Broadly, you have two main approaches: removing the missing data (deletion) or filling it in with estimated values (imputation). The choice isn't always obvious and depends heavily on the context, the amount of missing data, and your analytical goals.

Deletion Strategies: Removing Missing Data

The most straightforward approach is simply to remove rows or columns containing missing values.

Row Deletion (Listwise Deletion)

This involves removing entire rows (observations) that contain any missing value in any column.

• When it's useful: This strategy is most justifiable when the number of rows with missing data is very small compared to the total dataset size (e.g., less than 5%). If the data is missing completely at random (MCAR), meaning the fact that data is missing doesn't depend on either the missing value itself or other observed values, then deletion might not introduce significant bias.
• Potential downsides: The primary disadvantage is the loss of potentially valuable information contained in the non-missing columns of the deleted rows. If a large percentage of rows have missing values, this can drastically reduce your dataset size and statistical power. Furthermore, if the data is not missing completely at random, deleting rows can introduce bias into your analysis, leading to skewed results.

In Pandas, you can easily drop rows with any NaN values using the .dropna() method:

# Assuming 'df' is your DataFrame
df_cleaned_rows = df.dropna()

# Check the shape before and after
print(f"Original shape: {df.shape}")
print(f"Shape after dropping rows with NaN: {df_cleaned_rows.shape}")

Column Deletion

Alternatively, if a particular column (feature) has a very high proportion of missing values, it might provide little useful information for your analysis. In such cases, deleting the entire column might be a reasonable option.

• When it's useful: Consider this if a column is missing a substantial percentage of its values (e.g., > 50-70%). Keeping such a column might require extensive imputation, which could introduce more noise or bias than simply removing it.
• Potential downsides: You lose all information associated with that feature, which might be relevant even if incomplete. The threshold for deletion is subjective and depends on the feature's perceived importance.

To drop columns with missing values in Pandas, specify axis=1 in the .dropna() method. You can also use the thresh parameter to keep columns that have at least a certain number of non-NaN values.

# Drop columns where *all* values are NaN
df_cleaned_cols_all_nan = df.dropna(axis=1, how='all')

# Drop columns with more than 30% missing values
threshold = len(df) * 0.7 # Keep columns with at least 70% non-NaN values
df_cleaned_cols_thresh = df.dropna(axis=1, thresh=threshold)

print(f"Original shape: {df.shape}")
# print(f"Shape after dropping columns with all NaN: {df_cleaned_cols_all_nan.shape}")
print(f"Shape after dropping columns with >30% NaN: {df_cleaned_cols_thresh.shape}")

Imputation Strategies: Filling Missing Data

Imputation involves replacing missing values with substitute values. This preserves your sample size but introduces artificial data points, potentially affecting the data's original distribution and relationships. Simple imputation techniques are common during initial EDA.

Mean Imputation

Replace missing values in a numerical column with the mean of the observed values in that column.

• Pros: Simple to implement. Preserves the mean of the column.
• Cons: Reduces the variance of the column. Distorts relationships between variables (correlation). Highly sensitive to outliers, which can skew the mean. Only applicable to numerical data.

# Impute missing values in 'numerical_col' with the mean
mean_value = df['numerical_col'].mean()
df['numerical_col_mean_imputed'] = df['numerical_col'].fillna(mean_value)

Median Imputation

Replace missing values in a numerical column with the median of the observed values.

• Pros: Simple to implement. More effective against outliers than mean imputation.
• Cons: Like mean imputation, it reduces variance and distorts correlations. Only applicable to numerical data.

# Impute missing values in 'numerical_col' with the median
median_value = df['numerical_col'].median()
df['numerical_col_median_imputed'] = df['numerical_col'].fillna(median_value)

Mode Imputation

Replace missing values in a column with the mode (the most frequent value) of the observed values.

• Pros: Applicable to both numerical and categorical data. Simple.
• Cons: Can introduce bias, especially if one value is overwhelmingly dominant. Reduces variance. May arbitrarily choose one mode if multiple exist. Distorts relationships.

Pandas' .mode() method returns a Series (because there might be multiple modes). Typically, you'll use the first mode ([0]).

# Impute missing values in 'categorical_col' with the mode
mode_value = df['categorical_col'].mode()[0]
df['categorical_col_mode_imputed'] = df['categorical_col'].fillna(mode_value)

# Can also be applied to numerical columns, though less common
# numerical_mode_value = df['numerical_col'].mode()[0]
# df['numerical_col_mode_imputed'] = df['numerical_col'].fillna(numerical_mode_value)

Constant Value Imputation

Replace missing values with a predefined constant, such as 0, -1, "Unknown," or "Missing."

• Pros: Simple. Can sometimes signal that the value was originally missing (e.g., using "Unknown").
• Cons: Introduces an arbitrary value that can significantly distort statistical measures (mean, variance, correlation) and distributions, unless the chosen constant has a meaningful interpretation in the context (e.g., 0 might mean 'absence' for certain features).

# Impute numerical NaNs with 0
df['numerical_col_zero_imputed'] = df['numerical_col'].fillna(0)

# Impute categorical NaNs with 'Unknown'
df['categorical_col_unknown_imputed'] = df['categorical_col'].fillna('Unknown')

Choosing the Right Strategy: Deletion vs. Imputation

So, should you delete or impute? There's no universal answer. Consider these factors:

1. Amount of Missing Data:
   • Few missing values (<5%): Deletion (listwise) might be acceptable if you have a large dataset. Simple imputation is also unlikely to cause major distortions.
   • Moderate missing values (5-30%): Deletion becomes riskier due to information loss. Imputation is often preferred, but the choice of method matters more. Consider the potential impact on variance and correlations.
   • Many missing values (>30%): Listwise deletion is usually inappropriate. Column deletion might be considered if a specific feature is mostly missing. Imputation becomes necessary but potentially problematic; simple methods might heavily distort the data. More sophisticated imputation techniques might be needed later.
2. Nature of the Data and Variable:
   • Numerical: Mean or median imputation are common choices. Median is preferred for skewed distributions or data with outliers.
   • Categorical: Mode imputation or filling with a constant like "Unknown" are typical.
3. Mechanism of Missingness (Why is data missing?): While an exploration is for later stages, consider if the missingness seems random or if it might be related to other variables. If missingness is systematic (e.g., high-income earners not reporting income), simple imputation or deletion could lead to biased conclusions.
4. Analysis Goal: For quick exploration, simple imputation or even deletion might suffice. For building predictive models, the handling of missing data becomes more critical, often involving strategies integrated with model training (like imputing based only on training data).

Practical Recommendation:

Start by quantifying the extent of missing data per column and overall. If missing data is minimal, deletion might be the simplest path. If it's more prevalent, simple imputation (median for numerical, mode for categorical) is a reasonable starting point for initial EDA, allowing you to proceed with analysis while being mindful of the potential distortions introduced. Always document the strategy you choose, as it's an important step in your data preparation process. More complex imputation methods can be explored later if simple techniques prove inadequate for downstream tasks like modeling.