AIFrom Enthusiast to Practitioner
In 2015 I spent an afternoon at Kepler Labs watching people train a model, and came away with an uncomfortable realisation: I had been writing confidently about a practice I had never performed.
I said at the time I was going to fix that. It took longer than I intended, as these things do, but I’ve now been through Berkeley’s data science programme properly, and I want to write down what changed. Not as a course review, but because the things that turned out to matter were not the things I expected.
What I Thought I’d Learn
I expected algorithms. I’d been reading papers for years and I assumed the gap between me and a practitioner was a catalogue of methods I hadn’t studied.
That was wrong in an instructive way. The algorithms are the most accessible part of the material. You can learn what a random forest does in a morning. Understanding when its answer means anything took me considerably longer, and that turned out to be the actual subject.
“I thought I was missing techniques. I was missing the discipline that tells you whether a technique’s output is real, which nobody writes papers about because it isn’t novel.” — Sameer Gupta
The Three Things That Actually Landed
NumPy changed how I think, not just how I code.
I expected this to be a library to learn. It’s a way of thinking. Once you stop writing loops and start expressing operations over whole arrays, problems reshape themselves. Broadcasting in particular took a while to become intuitive and then became the thing I reach for constantly.
The speed matters, obviously, since vectorised operations run in compiled code rather than interpreted loops. But the real gain is that vectorised code says what it means. A loop describes how you’re doing something. An array operation describes what you’re computing.
Pandas is where the actual work lives, and nobody warns you.
Every course, every tutorial, every paper starts from a clean dataset. Real data is not clean. It has missing values that mean four different things, dates in three formats, duplicate records that aren’t quite duplicates, and a categorical column with eleven spellings of the same category.
I’d estimate eighty per cent of the effort in anything I’ve built since is joining, reshaping, and cleaning. Merges that silently duplicate rows. Group-bys that drop nulls when you didn’t want them to. Indexes that don’t align. This is the unglamorous majority of the job and I had genuinely not understood that from the outside.
Scikit-learn made the concepts concrete.
Reading about regularisation is one thing. Watching a model’s coefficients shrink as you turn up the penalty, and seeing the validation score improve then get worse, is another. The library is well designed enough that the API teaches you the shape of the problem: everything fits, everything transforms, everything can go in a pipeline.
The pipeline abstraction in particular is worth more than it looks, and I’ll come back to why.
The Lesson I Didn’t Expect
Here’s the thing I’d tell my 2015 self, and it has almost nothing to do with modelling.
Cross-validation will lie to you, and it lies in a specific, learnable way.
The idea is straightforward. Hold out part of your data, train on the rest, evaluate on the held-out part, repeat. You get an honest estimate of performance on data the model hasn’t seen.
Except it isn’t honest if you did anything to the data before splitting it. Scaled your features using the mean of the whole dataset? Your training set has now seen information from your test set. Filled missing values using an overall median? Same problem. Selected which features to use based on their correlation with the target, across all the data? You’ve badly contaminated the result.
Every one of those produces a validation score that is too good, and the model then underperforms in production and nobody can work out why.
“The most expensive mistake in this field isn’t building a bad model. It’s building a bad model that your own evaluation says is excellent, and finding out from a customer.” — Sameer Gupta
This is what the pipeline abstraction is really for. It isn’t tidiness. It’s that a pipeline applies every transformation inside each fold, so the contamination becomes structurally impossible rather than something you have to remember.
I had read about leakage. I had not understood it, because I had never produced a beautiful validation score and then had it explained to me why it was worthless.
What Changed in How I Write
Three things, concretely.
I’m more sceptical of reported numbers, including my own. When I read a paper or a vendor claim now, my first question is how the evaluation was constructed, not what the score was. That question is answerable surprisingly often and the answer is frequently uncomfortable.
I have more respect for boring methods. A well-validated logistic regression with clean features beats a badly-validated gradient boosting model, every time, and it can be explained to a regulator. I’d been quietly assuming sophistication and quality were correlated. They’re weakly correlated at best.
I understand where project budgets actually go. I’d been writing about the economics of these projects for six years while imagining the money went into modelling. It goes into data preparation and into the repeated cycle of building something, discovering the evaluation was flawed, and doing it again.
Was It Worth It
Yes, and not for the credential.
I am not an expert in the sense that the people at Kepler Labs are experts. Three years of coursework doesn’t substitute for a decade of doing this daily, and I’d be embarrassed to claim otherwise.
What I have is enough practice to know what I don’t know, and enough experience of things going wrong to recognise the shape of it when someone describes a project to me. That’s a smaller claim than “expert” and it’s the one that’s actually useful when a client describes a plan and something about it feels wrong.
Final Thoughts
If you’re in a similar position, writing or deciding about this technology without having built any of it, I’d recommend the detour. Not to change careers. To stop having opinions about a craft you’ve only read about.
The specific thing I’d tell you to focus on isn’t the algorithms. It’s evaluation. Learn to be suspicious of your own results, and learn the specific mechanisms by which a validation score can be wrong.
Everything else is available in a library. That part isn’t.