Not every AI system uses a language model, and reaching for one is often the wrong call.
Fraud detection, recommendation ranking, churn prediction, price estimation, ad targeting — these are classical machine learning problems, and a trained model on tabular features beats a language model on every axis that matters: it is milliseconds rather than seconds, costs a fraction, and is far easier to evaluate.
Interviews increasingly combine both — a recommendation system with a language-model ranking layer on top — so the classical lifecycle remains essential rather than superseded.
The lifecycle
Diagram, as text
- Raw data leads to Feature pipeline
- Feature pipeline leads to Feature store
- Feature store leads to Training
- Training leads to Model registry
- Model registry leads to Serving
- Feature store leads to Serving
- Serving leads to Monitoring
- Monitoring leads to Training (retrain trigger)
Two properties distinguish this from ordinary software delivery.
The artefact is derived from data, not written. You cannot review a model in a pull request. Reproducing it requires the same data, code and configuration — so all three must be versioned.
It decays without being touched. Code does not rot; models do, because the world their training data described keeps moving.
Features, and why they get their own infrastructure
A feature is an input signal: a customer's average order value over 30 days, the time since their last login, the click-through rate of a product.
Computing these is most of the work in a classical ML system, and it creates a specific problem that defines this lesson.
Training-serving skew
The defect that quietly ruins ML systems.
Your training pipeline computes "average order value over 30 days" with a batch SQL query over the warehouse. Your serving path computes the same feature in application code from a live database.
Two implementations of one definition. They will diverge — a different boundary condition, one excludes cancelled orders, one uses a rolling window and the other calendar days.
The model was trained on one distribution and served another. Accuracy degrades and nothing errors. Offline metrics stay excellent; production quietly underperforms.
This is genuinely hard to detect, because it produces no failure — only a model that is worse than it tested.
What a feature store does
It gives features one definition with two access paths.
┌── offline store (warehouse) -> training, batch scoring
feature definition ──┤
└── online store (low latency) -> real-time inferenceYou define the feature once. The store materialises it into a warehouse for training and a fast key-value store for serving, guaranteeing the same logic produced both.
It also provides three things that are painful to build:
Point-in-time correctness. When building a training set, each row must use feature values as they were at that moment, not as they are now. Otherwise you leak future information into training — the model learns from data it will not have at inference, scores brilliantly offline, and fails in production. This is the second classic ML bug, and getting it right by hand is difficult.
Reuse. Features defined by one team serve another rather than being recomputed slightly differently.
Freshness management. Some features update daily, some in seconds. The store handles both.
Training pipelines
Training must be reproducible: the same data version, code version and configuration produce the same model. Without that you cannot debug a regression or satisfy an audit.
Version all three. Data especially — "the customers table" is not a version, and a rerun a week later trains on different rows.
Automate retraining, triggered by schedule, by data volume, or by monitored drift. Manual retraining means it happens when someone remembers, which is after quality has already fallen.
Validate before promotion. A pipeline that trains and deploys without a gate will eventually ship a model degraded by a bad data day. Compare against the current production model on a held-out set and refuse to promote a regression.
The registry
The catalogue of trained models: versions, metrics, the data and code that produced them, and current stage — staging, production, archived.
It answers the questions that arise during an incident: which model is serving right now, what was it trained on, how did it score, and what do we roll back to. Without it, those answers live in somebody's memory.
Serving
Two shapes, chosen by how fresh predictions must be.
Batch scoring precomputes predictions on a schedule and stores them for lookup. Cheap and simple. Right when inputs change slowly — daily churn scores, weekly recommendations.
Real-time inference computes on request. Necessary when the prediction depends on immediate context: this session, this cart, this transaction.
A common hybrid precomputes candidates in batch and ranks them in real time — the standard recommendation architecture, and worth knowing as a pattern.
Monitoring, and why models decay
The reason ML needs continuous operation rather than delivery.
Data drift. The input distribution shifts. New customer segments, seasonal change, a marketing campaign bringing different traffic.
Concept drift. The relationship between inputs and outcome changes. Fraud patterns evolve specifically because fraudsters adapt to your model.
Feedback loops. Your model influences the data it will next be trained on. A recommender that surfaces certain items generates clicks on those items, which trains the next model to prefer them — narrowing over time in ways offline metrics will not reveal.
Monitor input distributions and prediction distributions, not only accuracy — because ground-truth labels often arrive late, and by the time you can measure accuracy the decay is weeks old. Distribution shift is the early signal.
Check your understanding
Common mistakes
Two implementations of one feature. The defining ML bug.
Leaking future information. Point-in-time correctness is subtle and produces impossibly good offline scores.
Unversioned training data. Reruns are not reproducible and regressions are undebuggable.
Manual retraining. Happens after quality has already fallen.
No promotion gate. A bad data day ships a bad model.
Monitoring accuracy only. Labels arrive late; watch distributions.
Using a language model for tabular prediction. Slower, costlier and worse at it.
Practice
A fraud model uses 'number of transactions in the last hour'. Describe how this feature can go wrong across training and serving, and how you would prevent each.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The same sentence describes several different computations.
Show a strong answer
The phrase is ambiguous, and each reading is a different feature.
- Which hour? The hour before the transaction being scored, or the hour before now? At training time these differ by however long ago the historical event was. Scoring uses the first; a naive training query frequently computes the second, which is leakage.
- Inclusive of the current transaction? A one-off difference that shifts the whole distribution, and the model will have fitted to whichever the training pipeline chose.
- What about late-arriving data? At serving time some of the last hour's transactions may not have landed yet, so the live value is systematically lower than the historical value computed after everything settled. The model then sees a distribution at serving time it never saw in training.
- What if there are none? Zero, or missing? If training used zero and serving sends null, the model receives something it has never seen.
How I would prevent each:
- One definition, one implementation. Define the feature once in the feature store with an explicit window and boundary semantics, and have both training and serving read from that definition. This removes the first two problems by construction.
- Point-in-time correct joins for training data, so each row gets the value as of the event's timestamp — not as of today.
- Simulate serving-time freshness in training. If, at serving time, only data older than two minutes is reliably present, compute the training feature with the same two-minute lag. Training on a completeness you will not have in production guarantees skew.
- Make missing explicit and consistent. Choose zero or null deliberately, encode it identically in both paths, and monitor the null rate — a rising null rate is one of the earliest signals of an upstream pipeline break.
The monitoring that catches what design misses: compare the live feature distribution against the training distribution continuously. A silent divergence here is the most common cause of a model that degrades with no deploy.
Your team wants a feature store. What would you need to see before agreeing it is worth the infrastructure?
Write your own answer first, on paper or in a note. Then compare.
Show a hint
It solves specific problems. Check you have them.
Show a strong answer
A feature store is real infrastructure — a serving path, a training path, a registry and an operational burden. Check the problems it solves are the problems you have.
What I would want to see:
- More than one model, or more than one team. A feature store's value is sharing definitions. One model owned by one team can define its features in one place in its own repository and get most of the benefit for none of the cost.
- Evidence of training–serving skew. Has a model actually degraded because two pipelines disagreed? If not, the primary justification is theoretical.
- Features that are genuinely expensive to compute — aggregations over long windows, joins across systems — where precomputing and reusing pays. Features that are a field lookup do not need a store.
- A real low-latency serving requirement. If the model runs in a batch job overnight, the online serving half of the store is unused.
- Reuse in practice. Are the same features wanted by several models? If every model has its own bespoke features, the registry holds a list rather than a shared asset.
If most of those are absent, the honest recommendation is a lighter version: define features as shared, versioned code used by both paths, with point-in-time joins done carefully in the training query. That addresses skew — the most damaging problem — without a new platform.
What would change my mind immediately: several teams already computing "customer lifetime value" or "transactions in the last hour" independently, with slightly different definitions. That duplication is the problem a feature store exists for, and it is worth the infrastructure the moment it appears.
Key takeaways
- Classical ML still wins for tabular prediction — orders of magnitude faster and cheaper than a language model, and easier to evaluate.
- The artefact is derived from data, so data, code and configuration must all be versioned for reproducibility.
- Training-serving skew — two implementations of one feature definition — is the defining silent failure.
- A feature store gives one definition with an offline and an online path, plus point-in-time correctness that prevents leaking future data into training.
- Adopt one when several models share features; a shared library suffices for a single model.
- Automate retraining and gate promotion on a held-out comparison against the current production model.
- Models decay through data drift, concept drift and feedback loops — monitor input and prediction distributions, since labels arrive late.
- Offline metrics do not predict business outcomes; validate with online experiments.
Further reading
- D. Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NIPS 2015).
- Martin Zinkevich, "Rules of Machine Learning: Best Practices for ML Engineering" (Google, 2017).
- Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022), chapters 5–6 — feature engineering and training–serving skew.
- Nick Handel et al., "Feature Store: A Data Management Layer for Machine Learning" (Feast documentation and design notes).
- Jimmy Lin and Dmitriy Ryaboy, "Scaling Big Data Mining Infrastructure: The Twitter Experience" (SIGKDD Explorations, 2013).