# Same Data, Same Model… Different Results? The One Detail Interviewers Want You to Say

# Same Data, Same Model… Different Results? The One Detail Interviewers Want You to Say

<img src="https://bugfree-s3.s3.amazonaws.com/mermaid_diagrams/image_1769883670451.png" alt="Reproducible ML diagram" width="700" />

In interviews, don’t lead with "noise" as your first explanation. The single most important point to call out is non-determinism arising from randomness and compute. In short: even with the same dataset and algorithm, tiny sources of randomness and floating‑point differences can steer training down different optimization paths.

Why identical data + model can produce different results

- Random initialization and SGD ordering: random weight seeds and the order of mini-batches change the optimization trajectory; small early differences can amplify.
- Data shuffling and augmentation: what the model "sees" first affects learning, especially with non-convex objectives.
- Hardware, drivers and libraries: different GPUs/CPUs, CUDA/cuDNN versions or BLAS implementations cause tiny floating‑point differences that can accumulate.
- Framework non-determinism: some operations are inherently non-deterministic (atomic adds, parallel reductions), and different library versions may change implementation details.

What interviewers want to hear (concise phrasing)

"I’d point out non-determinism from randomness and compute. To address it, I’d fix and document seeds, log environment and library versions, and run repeated trials (or k‑fold CV) to report mean ± std. If strict determinism is required, enable deterministic ops and lock versions — noting this can incur a performance cost." 

Practical, actionable steps to improve reproducibility

1. Fix random seeds consistently

- Python, NumPy, framework and GPU seeds (examples):

```python
# PyTorch
import random, os
import numpy as np
import torch

seed = 42
random.seed(seed)
np.random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
```

```python
# TensorFlow
tf.random.set_seed(42)
```

2. Make data pipeline deterministic where possible

- Use deterministic augmentations or fixed random states for augmentation libraries.
- Set dataloader workers to 0 or use deterministic worker initialization where supported.

3. Log environment and versions

- Record OS, Python, framework (PyTorch/TensorFlow) versions, CUDA, cuDNN, driver and GPU model.
- Helpful commands: `pip freeze`, `nvidia-smi`, `nvcc --version`.
- Save the seed and environment info alongside model checkpoints and experiment logs.

4. Use deterministic/backward flags and environment variables when needed

- PyTorch: `torch.use_deterministic_algorithms(True)` and `torch.backends.cudnn.deterministic = True` (may slow training).
- Newer NVIDIA setups: set `CUBLAS_WORKSPACE_CONFIG` and related env vars if using cuBLAS deterministic features.
- Be aware: enforcing determinism can change numerical behavior and slow performance.

5. Containerize and pin dependencies

- Use Docker containers or conda environments with exact package versions to make experiments portable and reproducible.

6. Run repeated trials and report uncertainty

- Perform multiple independent runs (e.g., 5–10) or k‑fold cross‑validation.
- Report mean ± standard deviation (or CI) for key metrics rather than a single run.

7. Automation and logging

- Automate experiments with scripts and log metadata (seed, commit hash, command-line args) to experiment trackers (MLflow, Weights & Biases, etc.).
- Include the seed in model filenames and experiment metadata.

Quick checklist to state in interviews

- "I’d lock seeds, log environment and package versions, run multiple trials (or k‑fold CV) and report mean±std. If necessary, enable framework deterministic flags and pin dependencies in a container." 

Notes and tradeoffs

- Full determinism is sometimes impossible or too costly; focus on reducing sources of variance and transparently reporting uncertainty.
- Small differences can be acceptable if you quantify them and they don’t change your model selection.

Bottom line: interviewers want to hear that you understand the true causes (randomness + compute) and that you have concrete steps—seeds, versioning, repeated trials—to diagnose and mitigate them.

#MachineLearning #Reproducibility #MLOps #DataScience