Cats are great and so is the forcats R package

This article is originally published at https://sw23993.wordpress.com

Cats are great. Perhaps Hadley Wickham and Lionel Henry think so too given the wonderful choice of name for their purrr package. Hadley Wickham has also created a smartly named package called forcats which is an anagram of factors. All R users will know what factors are and how useful they can be when analysing categorical data.

In the data science profession, one spends a vast amount of time getting data into a clean, tidy format (a.k.a preprocessing) for subsequent data visualisation, statistical analysis and modelling. If I were to estimate how much of my work life is spent doing this, I would say 70-80% at least. But I enjoy it!

This is often the major difference between university and real world application of data science. The datasets used in learning are nice and clean whereas those you encounter in the real world are far from it. I really enjoy data preprocessing and tackling the problems that arise during it. I have found a handful of functions from the forcats package to be incredibly useful when working with categorical data, or factors, in R.

Prepare some demonstration data 

A simple dataset is created using the following code. Note that there are 7 categories (factor levels) in the sales variable.

library(dplyr) # Load dplyr so we can use the pipe operator
library(forcats)

set.seed(55)
df <- data_frame(sales = factor(rep(c("Online",
 "Post",
 "Web",
 "Call Centre",
 "Inbound Phone",
 "Outbound Phone",
 "Field Sales",
 NA), 50)),
 buy = sample(c(0, 1), 400, replace = T)) %>%
 mutate(sales = sample(sales, size = length(sales), replace = T))

summary(df$sales) # Get counts

Dealing with missing data (NAs)

Missing values are a common occurrence in real world data. In an R dataframe, they need to be considered when performing statistical computations on continuous data such as calculations of the mean, trimmed mean, median, variance, standard deviation, etc. In addition, when you have a dataset from which you want to build a model you may need to treat NAs in order to retain particular variables and/or prevent the loss of data due to incomplete cases.

Common strategies involve imputing the mean or median (continuous) or, if categorical, the mode. However, in some cases this is not appropriate and you may need to set NAs as an explicit factor level. With forcats::fct_explicit_na() we can achieve this in a one-liner. Let’s try it out on the sales variable in our test data…

df$sales <- fct_explicit_na(df$sales) # Make NAs a level

summary(df$sales) # Get counts

The NAs are now represented by an explicit level and NA values are simply replaced by the string (Missing). The new level can be named whatever you like by passing a name to the na_level argument…

df$sales <- fct_explicit_na(df$sales, na_level = "My New Level")

Synonymous factor levels

Sometimes a categorical variable may have two or more factor levels that refer to the same group. There may be subtle differences in syntax such as upper case leading letter versus lower case leading letter (GroupA vs. groupA), for example. In this situation, one can use forcats::fct_collapse() to collapse the synonymous levels into one. In our test data, let’s assume that Web and Online refer to the same sales channel and we want to combine both into a single factor level called Online…

df$sales <- fct_collapse(df$sales, Online = c("Online", "Web"))

Lumping infrequent factor levels into the level Other

Another situation can occur whereby one wants to analyse or model using groups of data where large sample sizes are necessary for statistical significance. Imagine a factor variable with 20 levels but just 5 of these account for over 90% of the observations in a dataset. You might drop these rare occurrences entirely but any loss of data should be avoided if possible. Alternatively, you could lump the infrequent levels into one level which covers them all, preserving the rest of the attributes associated with those observations and enabling you to tidy up the group levels.

Using forcats::fct_lump() lumps all infrequent levels into the default level Other while ensuring that the count associated with this level remains the lowest of all levels.

df$sales <- fct_lump(df$sales)

Reordering bars in a ggplot2 barplot

The tidyverse is a suite of packages which contains dplyr, ggplot2 and forcats among others. The packages make up a wonderful ecosystem of tools for a data scientist and I would like to demonstrate how one more function, forcats::fct_infreq(), can be used with ggplot2 during the exploratory data analyis/data presentation stage. Reordering factors in a barplot can be somewhat tricky but is made easy using the forcats package…

library(ggplot2)

ggplot(df, aes(x = fct_infreq(sales))) + geom_bar()

The factor levels are now sorted by decreasing frequency and our new levels (Missing) and Other are included. We have lost none of the original data by carrying out some quick preprocessing using forcats.

Thanks for visiting r-craft.org
This article is originally published at https://sw23993.wordpress.com
Please visit source website for post related comments.