# Win Analytics Interviews: Validate Footfall Ideas with A/B Tests (Not Opinions) # Win Analytics Interviews: Validate Footfall Ideas with A/B Tests (Not Opinions) A/B testing diagram TL;DR: Don’t assert that a loyalty program or promotion “works” — prove it. Define a clear KPI (daily footfall, retention, or average ticket), choose a test unit (store/day/customer), run randomized control vs treatment with identical timing, guard against confounders, report lift and statistical confidence, then decide to scale, iterate, or kill. ## Why A/B tests matter for footfall Business stakeholders often make decisions based on intuition or anecdotes. For analytics interviews (and real product work) the highest-signal evidence is a well-designed A/B test. It replaces opinions with measurable, actionable results you can defend. ## Step-by-step: validate a footfall idea 1. Define the KPI - Pick the metric that matches the business goal: daily footfall, retention rate, conversion-to-purchase, or average ticket value. Be explicit (e.g., "daily unique customer visits per store"). 2. Choose the test unit and randomization level - Common units: store, store-day, or customer. Unit choice depends on interference risk and operational constraints. Randomize at the unit level to avoid spillovers. 3. Design control vs treatment - Ensure identical timing and context for both groups (same weekdays, marketing calendar, opening hours). - Use equal exposure windows and balance pre-period metrics if possible. 4. Precompute sample size and duration - Run a power calculation for the minimum detectable effect (MDE). Don’t stop early unless pre-registered sequential rules are in place. 5. Guard against confounders - Watch for holidays, staffing changes, store refurbishments, local events, and weather. - Strategies: stratified randomization, blocking (match similar stores), run parallel experiments, or exclude anomalous dates. 6. Run the test and collect data - Monitor data quality, missing telemetry, and any operational deviations (e.g., cashier errors). 7. Analyze and report - Report absolute and relative lift, with confidence intervals and p-values. Prefer CI + effect size over p-value-only reporting. - Segment results (by store type, weekday, customer cohort) and run pre-specified heterogeneity checks. 8. Decide: scale, iterate, or kill - Scale if lift is business-relevant and robust. Iterate (tweak treatment) if signal exists but not enough ROI. Kill if no lift or negative impact. ## Common pitfalls and how to avoid them - Stopping early: inflates false positives. Predefine stopping rules or use sequential testing methods. - Operational confounders: coordinate with ops to avoid staffing or promo changes during the test. - Spillover effects: randomize at a higher level (store instead of customer) if customers cross stores. - Multiple comparisons: correct for multiplicity when testing many outcomes or segments. ## How to present results in an interview - Lead with the business question and KPI. - Describe unit choice and why you randomized that way. - Summarize power calculation and test duration. - Show lift, confidence intervals, and a decision threshold (e.g., MDE and required ROI). - Mention confounders you controlled for and any sensitivity checks. ## Quick checklist for validating footfall ideas - [ ] KPI defined and measurable - [ ] Test unit and randomization plan - [ ] Power calculation and pre-registered duration - [ ] Confounder mitigation (blocking/stratification) - [ ] Data quality monitoring plan - [ ] Effect size + CI reported, not just p-value - [ ] Clear decision rule: scale / iterate / kill A/B tests are the difference between convincing and conjecture. When you can say "we tested, we measured X% lift with Y% confidence," you move from debate to decision — exactly the kind of thinking that wins analytics interviews and drives better product outcomes.