Machine learning in a hurry: what I’ve learned from the SLICED ML competition

This article is originally published at http://varianceexplained.org

This summer I’ve been competing in the SLICED machine learning competition, where contestants have two hours to open a new dataset, build a predictive model, and be scored as a Kaggle submission. Contestants are graded primarily on model performance, but also get points for visualization and storytelling, and from audience votes. Before SLICED I had almost no experience with competitive ML, so I learned a lot!

As of today I’m 5th in the standings, short of the cutoff for the playoffs, so if you want to see me continue you can vote for me as an “Audience Choice” here! (Though one of my favorite things about the competition is that audience members can compete along with Kaggle submissions, so I expect I’ll join in in any case!)

For four of the SLICED episodes (including the two weeks I was competing) I shared a screencast of my process. Episodes 4 and 7 were the ones I was competing live (you can also find the code here):

• Episode 1: Board Game Ratings
• Episode 4: Rain in Australia
• Episode 5: AirBnb Prices
• Episode 7: Credit churn risk

If you watch just one, I recommend AirBnb, which was a blast! (It was easily my favorite dataset of the competition, with a mix of text, geographic, and categorical data- though episode 1 and episode 8 were also delightful).

In this post, I’ll share some of what I’ve learned about using R, and the tidymodels collection of packages, for competitive Kaggle modeling.

The tidymodels packages

For years Python has had a solid advantage in tooling for machine learning (scikit-learn is just dynamite), but R has been catching up fast. The tidyverse team at RStudio has been developing the tidymodels suite of packages to play well with the tidyverse tools. This was the first time I’d really gotten in-depth experience with them, and I’ll share what I liked (and didn’t).

Pit of success

The “pit of success”, in contrast to a “summit of success” or a “pit of doom”, is the philosophy that users shouldn’t have to strive and struggle to do the right thing. They should be able to fall into good habits almost by accident, because it’s easier than using them incorrectly.

There’s a lot of traps you can fall into when doing machine learning, and the tidymodels packages are designed to avoid those “without thinking”. I think the single most impressive one is avoiding data leakage during your data cleaning.

Recipes and data leakage

Think of a few of the many feature engineering steps we might apply to a dataset before fitting a model:

• Imputing missing values (e.g. with mean)
• Dimensionality reduction (e.g. with PCA)
• Bucketing rare categories (e.g. replacing rare levels of a categorical variable with “Other”)

The recipes package offers functions for performing these operations (like step_impute_mean, step_pca, and step_other), but that’s only part of its value. The brilliant insight of the recipes packages is that each of those feature engineering steps is itself a model that needs to be trained! If you include the test set when you’re imputing values with their mean, or when you’re applying PCA, you’re accidentally letting in information from the test set (which makes you more likely to overfit and not realize it).

The recipes package makes this explicit: a recipe has to be trained on data (with prep()) before it is applied to a dataset (with bake()). By combining this recipe with a model specification into a workflow object, you encapsulate the entire start-to-end process of feature engineering and modeling. Other tidymodels tools then take advantage of this encapsulated workflow.

Recipes and data leakage

I’ve found each of the other packages also plays a role in the tidymodels “pit of success”. For instance:

• Cross validation (rsamples): Cross validation is one of the key ways to avoid overfitting, especially when selecting models and choosing hyperparameters. vfold_cv and fit_resamples make it easy.
• Tuning hyperparameters (tune): tune_grid makes it trivial to specify a combination of hyperparameters see how the performance varies. autoplot() in particular has a real “does exactly what I want out of the box” feel.
• Trying different models (parsnip): I really like not having to deal with each modeling package’s idiosyncrasies of inputs and outputs; less time reading docs means more time focusing on my model and data.
• Text classification (textrecipes): This has a special place in my heart. I helped develop the tidytext package, but I’ve never been completely thrilled with how it fit into a machine learning workflow. By providing tools for tokenization and filtering, the textrecipes package turns text into just another type of predictor.

These conveniences aren’t just a matter of saving effort or time; together they add up to a more robust and effective process. Prior to tidymodels, I might have just skipped tuning a parameter, or stopped with the first model type I tried, because iterating would be too difficult (a phenomenon known as shlep blindness).

Some of the resistance I’ve seen to tidymodels comes from a place of “This makes it too easy- you’re not thinking carefully about what the code is doing!” But I think this is getting it backwards. By removing the burden of writing procedural logic, I get to focus on scientific and statistical questions about my data and model.

Silly complaint: "this tool keeps you from thinking." No one stops thinking when they use good tools. They think about more important things

— David Robinson (@drob) August 26, 2016