Feature store with online, offline parity

Problem class

ML practitioners spend up to 60% of their time on feature engineering (Airbnb data). The same feature — "customer's average purchase value over the last 30 days" — is independently computed by three different teams with slightly different SQL, producing divergent values. The offline (batch training) computation uses a different code path than the online (real-time inference) computation, causing silent training-serving skew that is "extremely hard to debug." Features are not discoverable: GoJek found 10 different versions of the same feature being independently maintained. There is no point-in-time correctness guarantee, allowing data leakage from future information during training.

Mechanism

A feature store has two stores: an offline store (typically a columnar format on the lakehouse — Delta Lake, Parquet) for batch training data retrieval, and an online store (Redis, DynamoDB, Cassandra) for low-latency real-time inference serving. A single feature definition generates materialization pipelines for both stores, guaranteeing consistency. Point-in-time joins ensure training data reflects only what was known at prediction time. A feature registry enables discovery and reuse across teams. Transformation pipelines can be Python, Spark, or SQL — the feature store handles scheduling and backfill.

Required inputs

• Lakehouse or data warehouse as the offline store backing layer
• Low-latency key-value store for online serving (Redis, DynamoDB, Cassandra)
• Feature definition API (Python SDK or declarative YAML)
• ML training infrastructure that reads from the offline store
• Model serving infrastructure that reads from the online store
• Feature governance and discovery UI

Produced outputs

• Consistent feature values for training and inference (zero training-serving skew)
• Reusable feature library shared across ML teams (reducing redundant computation)
• Point-in-time correct training datasets
• Sub-millisecond online feature retrieval for real-time models
• Feature usage tracking and discovery

Industries where this is standard

• Ride-hailing and mobility (Uber, Lyft, GoJek) with dynamic pricing, ETA prediction, and fraud detection at millions of QPS
• Food delivery and marketplace platforms (DoorDash, Instacart) with real-time store ranking and delivery time prediction
• Content streaming (Spotify, Netflix) with personalization across hundreds of millions of users
• Fintech (Better.com, Clicklease) with real-time credit scoring and fraud detection
• Hyperscale e-commerce (Amazon, Pinterest) with product recommendations and ad targeting

Counterexamples

• Fewer than 5 ML models in production: The engineering investment in a feature store is not justified. Atlassian spent 3 engineers for 1 year building an internal feature store that wasn't scalable; small teams with fewer than 5 models rarely need a full feature store.
• "Kitchen sink" feature bloat: Data scientists add 200–300+ features because the store makes it easy, ignoring that 80% may be redundant. Feature governance requires active pruning.
• Treating the feature store as "just a database": Without feature discovery, governance, or distribution-drift monitoring, a feature store becomes an expensive key-value cache.

Representative implementations

• Uber (Michelangelo/Palette) manages 10,000+ features serving 10,000+ production models. Palette onboarding deployment time was reduced by >95%, and Cassandra cluster migration time dropped by 90%. 80% of Uber's ML workload is feature engineering.
• DoorDash (Gigascale) handles 10+ million queries/second, powers ~200 ML models processing 500+ billion predictions/week, and stores billions of feature-value pairs. Metrics layer optimization reduced pipeline latency and cloud costs each by ~50%.
• Airbnb (Zipline/Chronon) reported ML practitioners spent ~60% of their time on feature engineering before Zipline. After deployment, feature engineering tasks dropped from months to approximately 1 day. Only 5% of production ML is actual model code; 95% is data plumbing that Zipline addresses.
• Meta/Facebook achieves massive reuse — top 100 features are reused across 100+ models. GoJek found up to 10 different versions of the same feature being independently maintained before adopting Feast.

Common tooling categories

Feature store platform (Feast / Tecton / Hopsworks / Vertex AI Feature Store / SageMaker Feature Store / Databricks Feature Store) + offline store (Delta Lake / Parquet on S3) + online store (Redis / DynamoDB / Cassandra) + feature pipeline orchestration + feature discovery UI + drift monitoring.