Stop Guessing: Pick the Right Categorical Encoding in Interviews

Categorical features must be converted to numbers before feeding them to models. The encoding you pick can significantly affect model quality, runtime, and overfitting risk. Below is a compact guide to the three most common encodings — when to use each one, their trade-offs, and quick best-practices you can cite in interviews.

1) One‑Hot Encoding

• What it does: Creates a binary column for each category (category_i → 0/1).
• When to use: Nominal categories with low cardinality (e.g., color: red/blue/green).
• Pros: No implied order; model sees categories as independent.
• Cons: Explodes dimensionality with many categories → sparsity, memory blow-up, higher risk of overfitting.
• Tip: Use for low-cardinality features. For medium/high cardinality consider hashing, embeddings, or target encoding instead.

2) Label Encoding (Ordinal Integers)

• What it does: Maps categories to compact integers (A→0, B→1, C→2).
• When to use: Features that are truly ordered (small/medium/large) or some tree-based models where integer labels are often acceptable.
• Pros: Very compact, no extra columns.
• Cons: Introduces an artificial ordinal relationship if none exists — many linear models and distance-based algorithms will be misled.
• Tip: If the categories are not ordinal, avoid for linear/logistic/KNN/SVM. If using trees, label encoding is commonly used but still be cautious — consider frequency encoding or one-hot for low cardinality.

3) Target (Mean) Encoding

• What it does: Replaces each category with the mean of the target variable for that category (e.g., average purchase rate).
• When to use: High-cardinality categories where the category carries predictive signal.
• Pros: Very informative and compact (one column). Works well when category prevalence correlates with target.
• Cons: Major risk of target leakage and overfitting — the category mean on the training set can leak information about the target.
• How to mitigate leakage:
  • Use out-of-fold (K-fold) encoding: compute category means using only the training folds and apply to the holdout fold.
  • Smooth the estimates toward a global prior to reduce variance for rare categories.

Example smoothing formula:

Encoded_value = (count category_mean + alpha global_mean) / (count + alpha)

Where alpha controls how strongly rare categories are pulled toward the global mean (common choices: alpha between 5 and 50). Larger alpha → more smoothing.

Quick implementation pattern for models:

• During CV training: for each fold, compute smoothed means on the training portion and transform the validation portion with those means.
• For final model: compute smoothed means on the full training data and apply to test/unseen data.

Rule of Thumb

• One‑Hot: use for low-cardinality nominal features.
• Label: use for truly ordered features or (cautiously) with tree models.
• Target: use for high-cardinality features only with strict CV and smoothing to avoid leakage.

Extra tips to mention in interviews

• Always think about the downstream model family (linear vs tree vs neural net) when choosing an encoding.
• For very high cardinality, consider hashing trick or learned embeddings (deep learning) instead of one-hot.
• Validate using proper CV and check for leakage — target encoding without CV is a common source of exaggerated validation scores.

Keep this mental checklist in interviews: cardinality, order, model type, and leakage risk. Name the risks and the mitigations — interviewers like concise, practical answers.