# System Design Interview: Build a Push Notification Service That Scales
System Design Interview: Build a Push Notification Service That Scales
Push notifications are short server→client messages used to drive engagement, re-engage users, and surface time-sensitive information (e.g., order updates, security alerts). In system design interviews you must show how you'd design a solution that scales to millions of messages per second while staying reliable, accurate, and secure.
Below is a concise, interview-friendly design: clear requirements, high-level architecture, critical components, scaling strategies, and trade-offs.
## 1. Core requirements (must-haves)
- Scalability: support millions of pushes/sec and large fan-out (one message → millions of devices).
- Reliability & correctness: no silent drops; once-and-only-once or at-least-once guarantees depending on use case.
- Targeting: segment by user preferences, behavior, region, device type, and A/B experiments.
- Analytics: capture delivery, open/click, and latency metrics for each notification.
- Security & authenticity: prevent spoofing and unauthorized sends; data in transit must be protected.
- Low latency for time-sensitive notifications; scheduling and rate-limiting for non-urgent messages.
## 2. High-level architecture
- Clients register device tokens with the service.
- API layer accepts "send" and "schedule" requests.
- Message queue (e.g., Kafka) buffers bursts and preserves ordering for topics where needed.
- Worker/fan-out layer expands a message to target device tokens.
- External push services (FCM for Android, APNs for iOS) deliver to devices.
- Datastore for user preferences, device tokens, and logs.
- Cache (Redis) for hot preferences and recent device lists.
- Monitoring, metrics, and analytics pipeline to collect delivery/open events.
## 3. Component details
- API Layer
- Authenticated REST/gRPC endpoints for producers (apps, backend services).
- Validation, quota checks, and basic targeting filtering.
- Persist send requests to persistent queue or primary DB (if scheduled).
- Message Queue
- Durable topics for spikes and retry logic.
- Partitioning by target shard (user id or topic) to allow parallelism and ordering.
- Fan-out Workers
- Read from queue, resolve target device tokens (from cache/DB), apply preferences and rate limits, then batch requests to FCM/APNs.
- Implement retry/backoff and idempotency keys for at-least-once delivery semantics.
- Device Token Store
- Stores mapping: user_id → [device_token, platform, last_active, locale, prefs].
- TTL on stale tokens; periodic cleanup for invalid tokens.
- Cache
- Redis caching frequently used device lists and user prefs to reduce DB load.
- Delivery Adapters
- Thin adapters for FCM/APNs with connection pooling and exponential backoff.
- Track feedback (invalid tokens) and update token store.
- Analytics Pipeline
- Event ingestion (delivery, click/open) → stream processing (Kafka/Fluentd) → OLAP storage for dashboards and reports.
- Monitoring & Alerting
- Track success rate, latency percentiles, queue depth, error rates, and downstream service limits.
## 4. Scaling strategies
- Horizontal scaling
- Partition work by user ID, topic, or geographic shard. Scale API, workers, and adapters independently.
- Batching and connection pooling
- Batch tokens per device platform to reduce number of requests and maximize throughput.
- Backpressure and throttling
- If downstream (FCM/APNs) quota reached, apply graceful degradation, queueing, or rate limiting.
- Hot key mitigation
- Very popular topics (e.g., global announcement) should be handled with a publish/subscribe topic and parallel delivery shards.
- Caching
- Cache audience resolution to avoid repeated DB lookups for frequent sends.
## 5. Reliability and correctness
- Delivery guarantees
- For critical notifications (security), aim for at-most-once with strong idempotency and confirmation flows, or at-least-once with deduplication on the client.
- Retries & DLQ
- Retries with exponential backoff and a Dead Letter Queue (DLQ) for messages that repeatedly fail.
- Idempotency
- Use message IDs to detect and ignore duplicates in workers and delivery adapters.
- Validation of delivery
- Correlate downstream acknowledgements and device events to verify delivery; mark tokens invalid when FCM/APNs reports them.
## 6. Targeting & personalization
- Audience selection
- Support arbitrary predicates (segments by event, attributes, cohorts) computed via precomputed audience lists (offline) or real-time filtering.
- Preferences and suppression
- Respect user opt-outs, Do Not Disturb windows, and platform-specific rate limits.
- A/B testing
- Include campaign metadata and track variants in analytics.
## 7. Analytics & observability
- Events to capture: enqueue, dispatch, delivery ack, failure, open/click, bounce.
- Stream these to a metrics pipeline to compute KPIs in near real-time.
- Expose dashboards for queue depth, success rate, latency P95/P99, and platform error breakdowns.
## 8. Security
- Authenticate producers with short-lived tokens and RBAC.
- Sign messages and use HTTPS/TLS for all transport.
- Audit logs for sends and admin actions.
- Rate-limit to prevent abuse and spoofing.
## 9. Data model (simplified)
- DeviceToken { user_id, token, platform, created_at, last_active, status }
- Notification { id, payload, audience, scheduled_at, priority, sender_id }
- DeliveryLog { notification_id, device_token, status, attempt_count, last_error, timestamp }
## 10. Capacity planning (simple example)
- If average payload + headers per request is 1 KB and you need 1M messages/sec, bandwidth ~1 GB/sec (≈8 Gbps). Design network and adapter pooling accordingly.
- Use batching (e.g., 500 tokens/batch) to reduce concurrent connections and control outbound throughput.
## 11. Trade-offs and interview talking points
- At-least-once vs at-most-once: choose based on business needs (e.g., marketing can tolerate duplicates; security alerts cannot miss).
- Strong ordering is expensive: only enforce per-user or per-topic when required.
- Cost vs freshness: precompute audiences for large campaigns vs compute them on demand for targeted real-time sends.
## 12. Quick interview checklist
- State requirements and SLAs first.
- Draw the high-level flow: Client → API → Queue → Workers → FCM/APNs → Device.
- Discuss partitioning, batching, backpressure, retries, and idempotency.
- Cover analytics, monitoring, and security.
- Mention trade-offs and optimizations for cost and latency.
Designing a push notification service in an interview is about clarity: show how components interact, explain your scaling levers, and justify trade-offs based on correctness, latency, and cost.