Strategy 2: Deleting Columns

Sometimes, missing data isn't just scattered across a few rows; it can heavily affect entire columns (features) in your dataset. In the previous section, we looked at removing rows with missing values. Now, we'll consider a different, sometimes necessary, approach: removing entire columns.

This strategy is usually reserved for situations where a specific column has a very high percentage of missing entries. Why? Because if most of the data in a column is missing, that feature might not provide much useful information for your analysis or machine learning model. Trying to fill in a vast number of missing values (imputation, which we'll cover next) in such a column could introduce significant bias or noise, potentially doing more harm than good.

When to Consider Deleting a Column

There isn't a strict rule, but a common practice is to consider dropping a column if a large fraction of its values are missing. What constitutes "large"? This often depends on the context, the size of your dataset, and the importance of the feature. Some analysts use thresholds like 50%, 60%, or even 70% missing values as a starting point for considering column deletion.

Imagine you have a dataset with information about customers, including a column for 'Fax Number'. Today, very few customers might provide this, leading to perhaps 95% missing values in that column. For most analyses (like predicting purchase behavior), this column is unlikely to be helpful and could be a good candidate for removal.

Let's visualize this. Suppose we calculate the percentage of missing values for several columns in a dataset:

In this example, the 'Last_Login_Device' column has 85% missing data. This high percentage makes it a strong candidate for removal.

How to Delete Columns

Most data analysis libraries provide straightforward functions to drop columns. For instance, using the popular pandas library in Python, you would typically identify the names of the columns you wish to remove and use a function like drop() specifying the column names and indicating that you are dropping columns (as opposed to rows).

# Example using pandas (Python)
# Assume 'df' is your DataFrame and you want to drop 'Column_X'

# Calculate missing percentages
missing_percentages = df.isnull().mean() * 100

# Identify columns to drop (e.g., threshold of 70%)
columns_to_drop = missing_percentages[missing_percentages > 70].index

# Drop the identified columns
df_cleaned = df.drop(columns=columns_to_drop)

print(f"Original columns: {df.columns.tolist()}")
print(f"Columns after dropping: {df_cleaned.columns.tolist()}")

Advantages and Disadvantages

• Advantage: Simplicity. Removing the column entirely eliminates the missing data problem for that feature without needing complex imputation. It can also speed up subsequent processing by reducing the dataset's dimensionality.
• Advantage: Avoids introducing potentially biased data through imputation, especially when the missingness is very high.
• Disadvantage: Information Loss. This is the most significant drawback. Even if a column has many missing values, the few values that are present might hold valuable information. Deleting the column means losing that information entirely.
• Disadvantage: Potential Impact on Other Features. Sometimes, the pattern of missingness itself can be informative, or the feature might be important when considered alongside others, even if sparse.

Making the Decision

Deleting columns is a trade-off. You simplify the dataset but risk losing potentially useful information. Before dropping a column:

1. Assess the Percentage: Quantify how much data is actually missing.
2. Evaluate Feature Importance: Consider how relevant this feature might be to your analysis goals. Is it theoretically important? Could it be useful even if sparse?
3. Consider Alternatives: Think about whether imputation techniques (discussed next) might be feasible and less damaging, even if the missing percentage is somewhat high.

Choosing to delete a column is a more drastic step than deleting individual rows. It's often employed when the proportion of missing data in a feature is so substantial that the feature itself is unlikely to contribute meaningfully to the analysis, or when imputation seems too unreliable.