Two years as a Data Scientist at Stack Overflow

There are about half a dozen developers who regularly take part, and they all do great work. But I especially appreciate Ian Allen and Jisoo Shin for coming up with the idea of these sessions back in September, and for following through in the months since. Ian and Jisoo joined the company last summer, and were interested in learning R to complement their development of product features. Their curiosity, and that of others in the team, has helped prove that data analysis can be a part of every engineer’s workflow.

Writing production code

My relationship to production code (the C# that runs the actual Stack Overflow website) has also changed. In my first year I wrote much more R code than C#, but in the second I’ve stopped writing C# entirely. (My last commit to production was more than a year ago, and I often go weeks without touching my Windows partition). This wasn’t really a conscious decision; it came from a gradual shift in my role on the engineering team. I’d usually rather be analyzing data than shipping features, and focusing entirely on R rather than splitting attention across languages has been helpful for my productivity.

Instead, I work with engineers to implement product changes based on analyses and push models into production. One skill I’ve had to work on is writing technical specifications, both for data sources that I need to query or models that I’m proposing for production. One developer I’d like to acknowledge specifically Nick Larsen, who works with me on the Data Team. Many of the blog posts I mention above answer questions like “What tags are visited in New York vs San Francisco”, or “What tags are visited at what hour of the day”, and these wouldn’t have been possible without Nick. Until recently, this kind of traffic data was very hard to extract and analyze, but he developed processes that extract and transform the data into more readily queryable tables. This has many important analyses possible besides the blog posts, and I can’t appreciate this work enough.

(Nick also recently wrote an awesome post, How to talk about yourself in a developer interview, that’s worth checking out).

Working with other teams

Last year I mentioned that one of my projects was developing targeting algorithms for Job Ads, which match Stack Overflow visitors with jobs they may be interested in (such as, for example, matching people who visit Python and Javascript questions with Python web developer jobs). These are an important part of our business and still make up part of my data science work. But I learned in the last year about a lot of components of the business that data could help more with.

One team that I’ve worked with that I hadn’t in the first year is Display Ads. Display Ads are separate from job ads, and are purchased by companies with developer-focused products and services.

For example, I’ve been excited to work closer with Steve Feldman on the Display Ad Operations team. If you’re wondering why I’m not ashamed to work on ads, please read Steve’s blog post on how we sell display ads at Stack Overflow- he explains it better than I could. We’ve worked on several new methods for display ad targeting and evaluation, and I think there’s a lot of potential for data to have a postive impact for the company.

Changes in the rest of my career

There’ve been other changes in my second year out of academia. In my first year, I attended only one conference (NYR 2016) but I’ve since had more of a chance to travel, including to useR and JSM 2017, PLOTCON, rstudio::conf 2017, and NYR 2017. I spoke at a few of these, about my broom package, about gganimate and about the history of R as seen by Stack Overflow.

Julia and I wrote and published an O’Reilly book, Text Mining with R (now available on Amazon and free online here). I also self-published an e-book, Introduction to Empirical Bayes: Examples from Baseball Statistics, based on a series of blog posts. I really enjoyed the experience of turning blog posts into a larger narrative, and I’d like to continue doing so this next year.

There are some goals I didn’t achieve. I’ve had a longstanding interest in getting R into production (and we’ve idly investigated some approaches like Microsoft R Server), but as of now we’re still productionizing models by rewriting them in C#. And there are many teams at Stack Overflow that I’d like to give better support to- prioritizing the Data Team’s time has been a challenge, though having a second data scientist has helped greatly. But I’m still happy with how my work has gone, and excited about the future.

In any case, this made the whole year worthwhile:

Easily my favorite thing to come out of the Trump Twitter analysis was @nypost calling @StackOverflow "a Q&A site for egghead programmers" pic.twitter.com/0xrYkM2OOU

— David Robinson (@drob) November 4, 2016