A Practical ML Workshop at Google: What Actually Ships

A workshop, not a keynote

Some rooms are for applause and some are for work. The hands-on machine learning workshop I spoke at in Mountain View in 2019, hosted on Google's campus, was firmly the second kind. It was a smallish room with long tables, power strips taped down the middle, and roughly forty practitioners with laptops open. There was a whiteboard, and by the end of the day it was completely full. Nobody was there to be impressed. They were there because they had trained models that worked beautifully in a notebook and then watched those same models struggle, or quietly rot, once they were exposed to real traffic.

That was the exact subject I had been asked to speak on, and it's a subject close to my heart: the gap between a trained model and a model that ships. I told the room at the start that I wasn't going to teach them how to train a deep neural network — they already knew, and better than me in some cases. I was going to talk about everything that happens after model.fit() returns, because that is where enterprise machine learning actually lives or dies.

The 10% you keep talking about, and the 90% you don't

I drew a rectangle on the whiteboard and shaded one small corner of it. "This," I said, "is the model." The corner was maybe a tenth of the box. The rest — the unshaded ninety percent — was data collection, feature pipelines, validation, serving infrastructure, monitoring, and the machinery to retrain and redeploy safely. Everybody in that room recognized the picture, because everybody had underestimated it at least once.

The uncomfortable truth of 2019 was that most of the pain in production machine learning had nothing to do with the algorithm. The algorithm was the part with tutorials and Kaggle leaderboards. The part with no glamour and all the risk was the pipeline — the plumbing that fed the model at training time and, separately, at serving time. And those two plumbing systems being different was, I argued, the single most common way models died in production.

Training-serving skew: the bug that hides

I asked how many people had ever computed a feature one way in their training script — say, a rolling average over the last thirty days computed in a batch job over a data warehouse — and computed the "same" feature a different way in the live service, where the data arrived one event at a time. Most hands went up, some sheepishly.

That mismatch has a name — training-serving skew — and it is vicious precisely because nothing crashes. The model receives features at serving time that are subtly different from the ones it was trained on, so its predictions quietly degrade. Your offline evaluation looks great. Your production metrics slide, and you have no error message to chase. I told the room that if they took one thing from the day, it should be this: compute your features once, and share that computation between training and serving. The industry was just then converging on the idea of a shared feature layer for exactly this reason.

Drift: the model is fine; the world moved

The second failure mode I put on the board was drift, and I split it in two because the fixes are different.

Data drift is when the inputs change. The distribution of what your model sees in production moves away from the distribution it was trained on — a new customer segment, a new device, a seasonal shift, an upstream system that started formatting a field differently. The model isn't wrong; it's just being asked about a world it never saw.

Concept drift is nastier: the relationship between inputs and the right answer changes. The same input should now produce a different prediction. Fraud is the classic case — adversaries adapt specifically to defeat your model, so yesterday's honest-looking pattern is today's attack. No amount of clever architecture saves you from concept drift. Only retraining on fresh data does, which is why online and incremental learning were such active topics that year.

The lesson I pressed on the workshop was that drift is not a failure of the model — it is the expected, permanent condition of any model exposed to a live world. You don't fix it once. You build a system that expects it.

Why this mattered to enterprises then

For a startup, a model going stale is embarrassing. For an enterprise, it can be a compliance event, a safety event, or a very large amount of money. That raised the stakes on everything in the unshaded ninety percent of my rectangle.

This was the year "MLOps" stopped being a curiosity and started being a budget line. The teams that were succeeding had borrowed the discipline of software operations and applied it to models: version the data and the model together, not just the code; test the pipeline, not only the notebook; monitor production predictions the way you'd monitor a service's latency; and be able to roll a model back as fast as you'd roll back a bad deploy. None of that was exotic. It was ordinary engineering rigor applied to a new kind of artifact — one whose behavior depended on data it had seen, not only on the code someone wrote.

A concrete example: the churn model that quietly failed

I gave the workshop a case from the wild, sanitized. A team had a customer-churn model — a solid deep network, well-validated offline, live in production and trusted by the business. Six months in, its predictions had drifted from useless-in-one-direction to actively-misleading, and nobody noticed until a quarterly review.

The autopsy found two of the exact problems from the whiteboard. First, training-serving skew: a key feature — days since last login — was computed in the training warehouse in the customer's local time zone, but in the live service in UTC. Small, invisible, and enough to shift the feature for a large slice of users. Second, concept drift: the company had launched a new product tier, and the meaning of engagement had changed underneath the model. The fix wasn't a better architecture. It was a shared feature computation to kill the skew, a monitoring dashboard that watched the input distributions, and a scheduled retraining job so the model relearned as the product evolved. Boring. Decisive. That's what shipping looks like.

The short bridge to now

I won't pretend 2019 had the tooling we take for granted today — it didn't, and the teams that shipped did it with a lot of hand-built plumbing. But the principles were already settled: keep models in production honest with monitoring, let them learn continuously as the world moves, and treat the pipeline as the product. Those are the same principles StudioX now operationalizes as autonomous AI workers and Missions, carried well past the churn-model workshops of that Mountain View afternoon.

---

Related on StudioX: Enterprise AI Platform · AI Workers · AI Missions