Feature stores

Features and the MLOps loop

Start with the usage example of p. 738, a credit-scoring application. To decide on a loan, a model reads a few numbers about each customer: "Customer demographics (age, income, etc.)", "Historical credit behavior (loan repayments, defaults)" and "Real-time transaction data (recent purchases, account balance changes)". Each of these numbers is a feature.

In the notation of statistics a feature is a column of the design matrix \(X\); a feature store keeps those columns — named, dated and shared — outside any single model.

The same slide says where the store belongs: "It's a key asset in ML-Ops". MLOps (Machine Learning Operations) is "a set of practices and tools that aim to streamline the development, deployment, and maintenance of machine learning (ML) models in production environments"; it "extends the principles of DevOps (Development and Operations) to the ML domain" and "brings together data scientists, machine learning engineers, and operations teams" (p. 722). Page 723 draws it as a loop with no end.

Page 724 tells the same loop in five phases, each with what MLOps promises for it:

1. Development — experiments with "models, algorithms, and feature sets"; code, models and datasets are versioned.
2. Training and Validation — the model is trained and "validated using a test dataset", in a way that "can be reproduced".
3. Deployment — automated with CI/CD practices.
4. Monitoring — "real-time insights into how the model is performing, and whether data or concept drift is affecting its accuracy".
5. Retraining — "When the model's performance drops or when new data becomes available".

Page 725, titled "ML-Ops – focus on data and features", draws the same lifecycle as four interlocking rings — Data, ML, Dev and Ops — and the first ring is pure data work: collect, curate, transform, validate. That ring is where a feature store works.

Fix the two drifts of phase 4 now; the slides use them without defining them. Data drift: the distribution of the inputs changes — this year's applicants are younger. Concept drift: the relation between inputs and target changes — the same spending pattern no longer means the same risk. In statistical terms \(P(X)\) moves, or \(P(Y \mid X)\) moves.

Pages 726–728 list eight components of MLOps: version control for models and data ("datasets, features, models, and hyperparameters"); CI/CD for ML; model training and retraining; monitoring and model performance tracking ("accuracy, precision, recall, and business-specific KPIs"); data and feature management; automation and orchestration ("Kubernetes, Airflow, or Kubeflow"); collaboration between teams; model governance and compliance. Page 729 adds five benefits: faster time to market, improved collaboration, increased model reliability, reusability "of code, pipelines, and features", better model governance.

The fifth component is the door to this lesson: "MLOps integrates with feature stores to ensure that the same data transformations applied during training are applied in production, avoiding issues like 'training-serving skew.'" (p. 727).

MLOps is a loop, and the features are needed at two of its stations: training and inference.

The biggest problem: one feature, computed twice

Page 743 is titled "The biggest problem to face": "Features for a machine learning model are required in both the modelling and production (deployment) phases of the lifecycle. This implies duplication of code and risk of inconsistency".

Here is what that looks like. A data scientist writes spend_l30d, a customer's spending over the last 30 days, in a notebook: purchases minus refunds, computed over the history to build the training set. The model goes live, and an engineer rewrites the same feature inside the production service — another language, a deadline, and no refunds. On 14 June customer C2, who returned an item of 220.00 the day before, has 275.00 in the training data and 495.00 in production. Both programs run without a single error.

The fix is to stop having two programs. On p. 742 the feature store is "An interface between the raw data and the models. It takes raw data and then transforms it into features subsequently used for model training and inference", and it "ensures that the features used across both models (and possibly many models) are consistent". The same slide calls it "a centralised repository that stores curated features" and "a data management layer that allows data scientists, machine learning engineers, and data engineers to collaborate, share, and discover features".

What a feature store "grants", in the words of p. 743: "Consistency between model training and inference leads to better model accuracy", "eliminating online/offline skew", "Share and reuse features across models and teams, ensuring faster model development", "Provide feature versioning, lineage, provenance, and regulatory compliance". Page 737 sums it up in four benefits: Consistency (the same transformations in training and production), Efficiency (no redundant feature engineering), Scalability (batch and real-time features at large scale) and Version Control (models trained on older versions of features "can still be reproduced").

Page 733 shows who talks to the store, and when. Data engineers, data scientists and ML engineers "Define and iterate on new features" and "Load feature data for training" and "for testing"; the prediction service in production receives the feature values computed for inference and writes back to the store ("Log served features, prediction, outcomes", "Track feature stats to identify issues"); the engineers read those back to "Monitor for drift & quality issues". Every phase of the loop reads from the store or writes to it.

Is it always worth it? Feature stores "are a must-have when working on a project where the intention is to deploy models at scale and in production"; "However, if you are working within a small team and on a small project or a POC, you may not need it" (p. 744). A proof of concept has one model and no production: nothing is computed twice.

A feature store replaces two programs that compute the same feature with one definition that feeds both training and serving.

Two stores: the history for training, the latest value for serving

One definition does not mean one database. Back to the credit score of p. 738: "For training, the offline store would provide historical data, while for real-time predictions, the online store would supply the latest transaction data." Two stores, because training and serving ask two different questions.

The offline store "Contains batch features used for model training. These are typically updated at regular intervals (e.g., daily or weekly) and stored in large-scale data storage systems like Hadoop or cloud-based data lakes" (p. 735). It keeps "months' or years' worth of feature data", "often stored in data warehouses or data lakes like S3, BigQuery, Snowflake, Redshift" (p. 765). Its question is a scan: every customer's value on every day of the last two years. That is the workload of a warehouse (lesson 23).

The online store "Contains real-time features used for serving live predictions. These features need to be served with low latency for real-time inference and are typically stored in low-latency databases like Redis" (p. 735). It persists "feature values for low-latency lookup during inference", and "They typically only store the latest feature values for each entity" (p. 765). Its question is a lookup: the current values of customer C2, now, in milliseconds — a GET on a key, exactly what Redis is built for (lesson 17).

Page 765 adds: "Online stores are usually eventually consistent." Why is that acceptable? The article the slides follow explains that most ML uses have no strict consistency requirement: a fraud check computed on a balance a few seconds old is still a sensible check, while a check that arrives after the payment has gone through is useless. Latency and availability win — the BASE side of the trade-off of lesson 5.

Serving: one API, two answers

Reading from the two stores is the task the slides call Serving: "An easy and canonical way to access all features in a company consistently across all environments" (p. 764). Offline, "feature values are commonly accessed through notebook-friendly feature store SDKs"; online, "a feature store delivers a single vector of features at a time made up of the freshest feature values, through a high-performance API backed by a low-latency database". The figure of p. 764 asks for the same three features both ways (l7d and l30d mean "last 7 days" and "last 30 days"):

.get_feature_vector (
    features=["item_clicks_l7d",
              "item_price",
              "user_spend_l30d"],
    entities={"user_id":3,
              "item_id":62}
)

{
  "item_clicks_l7d":6,
  "item_price":"$25.12",
  "user_spend_l30d":"$12.10"
}

.get_training_data (
    features=["item_clicks_l7d",
              "item_price",
              "user_spend_l30d"],
    entities=entities_df
)

user_id  item_id  time   item_clicks_l7d  item_price  user_spend_l30d
1        54       7485   6                $23.2       $0
2        35       46854  5                $41.54      $8.41

The same three names, two very different answers: one vector of the latest values for one key, or one row per (entity, time) with the values as they were then. That time column comes back in section 5.

How do values reach the online store? Page 757, about Feast: "For offline use cases that only rely on batch data, Feast does not need to ingest data and can query your existing data", while "For online use cases, Feast supports ingesting features from batch sources to make them available online (through a process called materialization), and pushing streaming features to make them available both offline / online". The offline store can be the warehouse you already have, queried in place; materialization copies the latest value of each key into the online store; a stream pushes to both.

The widget puts the two stores, and the two programs of section 2, side by side. Its one insight: skew is silent — it appears, hides and reappears with the data and the calendar, and only one shared definition removes it.

The offline store answers "what was the value then?" for training; the online store answers "what is the value now?" for one key, in milliseconds.

Inside the box: five tasks

Page 763 opens the box (Fig. 3). Data enter from a stream source and a batch source and leave towards model serving and model training. Three tasks sit on the data path — transform, storage, serving, with an on-demand transform inside serving — and two span all of them: operational monitoring above, the registry below. The slides number them 1 Serving, 2 Storage, 3 Transform, 4 Monitoring, 5 Registry; you have met the first two.

Transform: where feature engineering runs

Page 741 defines what the transform task runs. Feature engineering: "DEF: a machine learning approach that extracts features from a raw dataset, resulting in new features". It "ranges from basic transformations such as aggregations to more advanced feature transformations such as word embeddings produced by machine learning algorithms", and its goal is "to essentially create a better dataset to improve the performance of machine learning algorithms". Page 734 names the everyday steps: "aggregations, normalization, one-hot encoding".

The jobs "are executed on data processing engines (e.g. Spark or Pandas) to which the feature store is connected" (p. 766): the store orchestrates them, it does not replace Spark (lesson 38). Page 767 sorts them into three kinds, by when their input exists:

The vector served on p. 766 mixes all three, colour-coded on the slide: "item_clicks_last_5_mins":6 comes from a stream transform, "user_spend_last_30_days":"$1.10" from a batch transform, and "user_current_location":"US_NY" from an on-demand transform fed with the request's application data, {user_ip: "x.x.x.x"}. Why can't the third be precomputed? Its input — where the user is right now — exists only when the request arrives.

Monitoring and the registry

Monitoring (p. 768) tracks "operational metrics relating to core functionality": for storage "availability, capacity, utilization, staleness", for serving "throughput, latency, error rates". Keep it apart from the key component "Monitoring and Validation" (p. 736), which checks the feature values themselves so that "data drifts or anomalies are detected early". One watches the machine, the other the data.

The registry "is a central interface for user interactions with the feature store. Teams use the registry as a common catalog to explore, develop, collaborate on, and publish new definitions. The registry allows for important metadata to be attached to feature definitions" (p. 769). It holds definitions, not values. It is what makes reuse real: the "catalog or search functionality" of p. 736 lets the next data scientist find spend_l30d instead of writing it a third time.

Pages 734–736 list eight "Key Components". Each lives in one of the five tasks (the mapping is ours):

Three tasks on the data path — transform, storage, serving — and two over all of them: monitoring and the registry.

Feast: entities, feature views and the point-in-time join

The deck ends with one technology in detail: Feast, "Feature Store for Machine Learning", "Type: Open-source" (p. 746). It was born at the ride-hailing company Gojek with Google Cloud and announced in January 2019 (Google Cloud blog) — the "Go-jek 2019" dot of p. 739. Its objects form a small tree (Fig. 4).

A feature view maps to "0 or more entities" (p. 758): zero for a global feature ("num_daily_global_transactions"), one for a user feature ("user_age or last_5_bought_items"), several — "aka a composite key" — for a feature of a pair ("num_user_purchases_in_merchant_category", a user and a merchant category). "A collection of entities for a feature view is called an entity key", and "Entities should be reused across feature views", so that features about the same thing are found, and joined, on the same key.

Then the sentence the rest of this section unpacks: "Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or materialize features into an online store" (p. 759). Pages 760–761 show what its rows look like:

The slides use "entity key" in two senses: the set of entities of a feature view (p. 758), and the pair of values that identifies one row, (customer_A, driver_1001) in the red frame of p. 761. Both are Feast's usage. In Feast's Python SDK the driver view of p. 760 is declared like this (adapted from the Feast documentation; not on the slides):

# a feature view: two features of the entity "driver",
# read from a BigQuery table
from datetime import timedelta
from feast import BigQuerySource, Entity, FeatureView, Field
from feast.types import Float32, Int64

driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats = FeatureView(
    name="driver_stats",
    entities=[driver],
    ttl=timedelta(days=1),
    schema=[
        Field(name="trips_today", dtype=Int64),
        Field(name="rating", dtype=Float32),
    ],
    source=BigQuerySource(table="my-project.rides.driver_stats"),
)

Why the timestamp matters: the point-in-time join

Why insist on time? Build a training set for "will this trip end in a complaint?". The labels arrive as rows — driver, time of the trip, complaint yes or no — called the entity dataframe. To each row you must attach the driver's rating. A plain join on driver_id with today's table gives every row today's rating, including ratings computed after the trip, perhaps because of that very complaint. The model trains on information it will never have in production: data leakage. Its symptom is an offline score far better than anything production delivers.

Feast's documentation names the cure — the slides do not: "Feast performs a point-in-time join (also called a 'last known good value' temporal join): for each entity row, it retrieves the latest feature values with a timestamp at or before the entity row's event_timestamp." Without that rule, "features generated after the prediction time could leak into training data — effectively letting the model 'see the future'". In symbols:

1. Take one row of the entity dataframe: key driver_1001, timestamp 5/22 10:00.

2. Keep the feature rows with the same key: 4.6 (5/20), 4.7 (5/21), 4.8 (5/22), 4.9 (5/23), 4.9 (5/24), all at 0:00.

3. Drop every row dated after 10:00 on 5/22: the two values 4.9 did not exist yet when the trip took place.

4. Take the latest of what remains: \(\tau^{*}\) = 5/22 0:00, value 4.8 — the rating p. 760 prints for that day.

5. Check its age against the ttl: 10 hours, inside a ttl of one day, so 4.8 is joined. Older than the ttl, the feature would be null — and null too if no rating were dated at or before 10:00.

The same rule explains the online store: materialization is the point-in-time join evaluated at "now" for every key — "the latest feature values for each entity" (p. 765). The widget builds the training set both ways. Its one insight: a training row may only see the past of its own timestamp, and a plain join quietly hands it the future.

In Feast the whole join is one call, and the online lookup another (Feast documentation; store is the FeatureStore of your project):

# the training set: one row per labelled trip,
# each row joined point-in-time
import pandas as pd

entity_df = pd.DataFrame({
    "driver_id": ["driver_1001", "driver_1002"],
    "event_timestamp": pd.to_datetime(
        ["2021-05-22 10:00", "2021-05-22 20:00"]),
    "complaint": [0, 1],
})
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=["driver_stats:rating", "driver_stats:trips_today"],
).to_df()

# one vector of the latest values for one key, for a prediction now
features = store.get_online_features(
    features=["driver_stats:rating", "driver_stats:trips_today"],
    entity_rows=[{"driver_id": "driver_1001"}],
).to_dict()

A training row may only see feature values dated at or before its own timestamp; the online store is the same join, taken at "now".

Nine products, one history, five dimensions

Page 739 says where it all began: "First Experience in Uber, with Michelangelo platform." Fig. 5 redraws its timeline.

Pages 746–754 review nine products in one template, "Type" plus "Key Characteristics". Most traits repeat on every slide ("Online and Offline Stores", versioning, "real-time and batch"); what tells them apart is the type and one or two traits:

The course covers every family of systems "along 5 dimensions" (intro deck, slide 6). The feature-store slides do not fill them in; here is the profile their pages imply (ours):

So where does a feature store sit among the systems of this course? Above them:

Back to Uber. Michelangelo's shared features lived in HDFS/Hive for training and in Cassandra for online predictions, and the transformations each model applied to them were written in a small language whose "expressions are part of the model configuration and the same expressions are applied at training time and at prediction time" (Uber Engineering blog, 2017). One definition, so a feature is not computed twice; two stores, one for the history and one for the latest values, read by key. The second danger of the hook, reading a value at the wrong moment, is what the timestamps of the time-series model and the point-in-time join are for. That is why the "store" was never a new database: it is a layer of definitions over a data lake and a low-latency store. The deck's last sentence says the same: "a feature store acts as the backbone of the feature lifecycle in machine learning systems, enabling teams to efficiently manage and serve high-quality features across different stages of the ML pipeline, from model development to deployment" (p. 771).

A feature store is a layer — a registry of definitions over a warehouse for the history and a key-value store for the latest values — not a new kind of database.