sample_n_of(): a useful helper function
This article is originally published at https://tjmahr.github.io/
Here’s the problem: I have some data with nested time series. Lots of them. It’s like there’s many, many little datasets inside my data. There are too many groups to plot all of the time series at once, so I just want to preview a handful of them.
For a working example, suppose we want to visualize the top 50 American female baby names over time. I start by adding up the total number of births for each name, finding the overall top 50 most populous names, and then keeping just the time series from those top names.
library(ggplot2)
library(dplyr, warn.conflicts = FALSE)
babynames <- babynames::babynames %>%
filter(sex == "F")
top50 <- babynames %>%
group_by(name) %>%
summarise(total = sum(n)) %>%
top_n(50, total)
# keep just rows in babynames that match a row in top50
top_names <- babynames %>%
semi_join(top50, by = "name")
Hmm, so what does this look like?
ggplot(top_names) +
aes(x = year, y = n) +
geom_line() +
facet_wrap("name")
Aaack, I can’t read anything! Can’t I just see a few of them?
This is a problem I face frequently, so frequently that I wrote a helper
function to handle this problem: sample_n_of(). This is not a very clever
name, but it works. Below I call the function from my personal R package
and plot just the data from four names.
# For reproducible blogging
set.seed(20180524)
top_names %>%
tjmisc::sample_n_of(4, name) %>%
ggplot() +
aes(x = year, y = n) +
geom_line() +
facet_wrap("name")
In this post, I walk through how this function works. It’s not very complicated: It relies on some light tidy evaluation plus one obscure dplyr function.
Working through the function
As usual, let’s start by sketching out the function we want to write:
sample_n_of <- function(data, size, ...) {
# quote the dots
dots <- quos(...)
# ...now make things happen...
}
where size are the number of groups to sample and ... are the columns names
that define the groups. We use quos(...) to capture and quote those column
names. (As I wrote before,
quotation is how we bottle up R code so we can deploy it for later.)
For interactive testing, suppose our dataset are the time series from the top 50 names and we want data from a sample of 5 names. In this case, the values for the arguments would be:
data <- top_names
size <- 5
dots <- quos(name)
A natural way to think about this problem is that we want to sample subgroups of
the dataframe. First, we create a grouped version of the dataframe using
group_by(). The function group_by() also takes a ... argument where the
dots are typically names of columns in the dataframe. We want to take the
names inside of our dots, unquote them and plug them in to where the ...
goes in group_by(). This is what the tidy evaluation world calls
splicing.
Think of splicing as doing this:
# Demo function that counts the number of arguments in the dots
count_args <- function(...) length(quos(...))
example_dots <- quos(var1, var2, var2)
# Splicing turns the first form into the second one
count_args(!!! example_dots)
#> [1] 3
count_args(var1, var2, var2)
#> [1] 3
So, we create a grouped dataframe by splicing our dots into the group_by()
function.
grouped <- data %>%
group_by(!!! dots)
There is a helper function buried in dplyr called group_indices() which
returns the grouping index for each row in a grouped dataframe.
grouped %>%
tibble::add_column(group_index = group_indices(grouped))
#> # A tibble: 6,407 x 6
#> # Groups: name [50]
#> year sex name n prop group_index
#> <dbl> <chr> <chr> <int> <dbl> <int>
#> 1 1880 F Mary 7065 0.0724 33
#> 2 1880 F Anna 2604 0.0267 4
#> 3 1880 F Emma 2003 0.0205 19
#> 4 1880 F Elizabeth 1939 0.0199 17
#> 5 1880 F Margaret 1578 0.0162 32
#> 6 1880 F Sarah 1288 0.0132 45
#> 7 1880 F Laura 1012 0.0104 29
#> 8 1880 F Catherine 688 0.00705 11
#> 9 1880 F Helen 636 0.00652 21
#> 10 1880 F Frances 605 0.00620 20
#> # ... with 6,397 more rows
We can randomly sample five of the group indices and keep the rows for just those groups.
unique_groups <- unique(group_indices(grouped))
sampled_groups <- sample(unique_groups, size)
sampled_groups
#> [1] 4 25 43 20 21
subset_of_the_data <- data %>%
filter(group_indices(grouped) %in% sampled_groups)
subset_of_the_data
#> # A tibble: 674 x 5
#> year sex name n prop
#> <dbl> <chr> <chr> <int> <dbl>
#> 1 1880 F Anna 2604 0.0267
#> 2 1880 F Helen 636 0.00652
#> 3 1880 F Frances 605 0.00620
#> 4 1880 F Samantha 21 0.000215
#> 5 1881 F Anna 2698 0.0273
#> 6 1881 F Helen 612 0.00619
#> 7 1881 F Frances 586 0.00593
#> 8 1881 F Samantha 12 0.000121
#> 9 1881 F Karen 6 0.0000607
#> 10 1882 F Anna 3143 0.0272
#> # ... with 664 more rows
# Confirm that only five names are in the dataset
subset_of_the_data %>%
distinct(name)
#> # A tibble: 5 x 1
#> name
#> <chr>
#> 1 Anna
#> 2 Helen
#> 3 Frances
#> 4 Samantha
#> 5 Karen
Putting these steps together, we get:
sample_n_of <- function(data, size, ...) {
dots <- quos(...)
group_ids <- data %>%
group_by(!!! dots) %>%
group_indices()
sampled_groups <- sample(unique(group_ids), size)
data %>%
filter(group_ids %in% sampled_groups)
}
We can test that the function works as we might expect. Sampling 10 names returns the data for 10 names.
ten_names <- top_names %>%
sample_n_of(10, name) %>%
print()
#> # A tibble: 1,326 x 5
#> year sex name n prop
#> <dbl> <chr> <chr> <int> <dbl>
#> 1 1880 F Sarah 1288 0.0132
#> 2 1880 F Frances 605 0.00620
#> 3 1880 F Rachel 166 0.00170
#> 4 1880 F Samantha 21 0.000215
#> 5 1880 F Deborah 12 0.000123
#> 6 1880 F Shirley 8 0.0000820
#> 7 1880 F Carol 7 0.0000717
#> 8 1880 F Jessica 7 0.0000717
#> 9 1881 F Sarah 1226 0.0124
#> 10 1881 F Frances 586 0.00593
#> # ... with 1,316 more rows
ten_names %>%
distinct(name)
#> # A tibble: 10 x 1
#> name
#> <chr>
#> 1 Sarah
#> 2 Frances
#> 3 Rachel
#> 4 Samantha
#> 5 Deborah
#> 6 Shirley
#> 7 Carol
#> 8 Jessica
#> 9 Patricia
#> 10 Sharon
We can sample based on multiple columns too. Ten combinations of names and years should return just ten rows.
top_names %>%
sample_n_of(10, name, year)
#> # A tibble: 10 x 5
#> year sex name n prop
#> <dbl> <chr> <chr> <int> <dbl>
#> 1 1907 F Jessica 17 0.0000504
#> 2 1932 F Catherine 5446 0.00492
#> 3 1951 F Nicole 94 0.0000509
#> 4 1953 F Janet 17761 0.00921
#> 5 1970 F Sharon 9174 0.00501
#> 6 1983 F Melissa 23473 0.0131
#> 7 1989 F Brenda 2270 0.00114
#> 8 1989 F Pamela 1334 0.000670
#> 9 1994 F Samantha 22817 0.0117
#> 10 2014 F Kimberly 2891 0.00148
Next steps
There are a few tweaks we could make to this function. For example, in my package’s version, I warn the user when the number of groups is too large.
too_many <- top_names %>%
tjmisc::sample_n_of(100, name)
#> Warning: Sample size (100) is larger than number of groups (50). Using size
#> = 50.
My version also randomly samples n of the rows when there are no grouping variables provided.
top_names %>%
tjmisc::sample_n_of(2)
#> # A tibble: 2 x 5
#> year sex name n prop
#> <dbl> <chr> <chr> <int> <dbl>
#> 1 1934 F Stephanie 128 0.000118
#> 2 2007 F Mary 3674 0.00174
One open question is how to handle data that’s already grouped. The function we wrote above fails.
top_names %>%
group_by(name) %>%
sample_n_of(2, year)
#> Error in filter_impl(.data, quo): Result must have length 136, not 6407
Is this a problem?
Here I think failure is okay because what do we think should happen? It’s not obvious. It should randomly choose 2 of the years for each name. Should it be the same two years? Then this should be fine.
top_names %>%
sample_n_of(2, year)
#> # A tibble: 100 x 5
#> year sex name n prop
#> <dbl> <chr> <chr> <int> <dbl>
#> 1 1970 F Jennifer 46160 0.0252
#> 2 1970 F Lisa 38965 0.0213
#> 3 1970 F Kimberly 34141 0.0186
#> 4 1970 F Michelle 34053 0.0186
#> 5 1970 F Amy 25212 0.0138
#> 6 1970 F Angela 24926 0.0136
#> 7 1970 F Melissa 23742 0.0130
#> 8 1970 F Mary 19204 0.0105
#> 9 1970 F Karen 16701 0.00912
#> 10 1970 F Laura 16497 0.00901
#> # ... with 90 more rows
Or, should those two years be randomly selected for each name? Then, we should
let do() handle that. do() takes some code that returns a dataframe, applies
it to each group, and returns the combined result.
top_names %>%
group_by(name) %>%
do(sample_n_of(., 2, year))
#> # A tibble: 100 x 5
#> # Groups: name [50]
#> year sex name n prop
#> <dbl> <chr> <chr> <int> <dbl>
#> 1 1913 F Amanda 346 0.000528
#> 2 1953 F Amanda 428 0.000222
#> 3 1899 F Amy 281 0.00114
#> 4 1964 F Amy 9579 0.00489
#> 5 1916 F Angela 715 0.000659
#> 6 2005 F Angela 2893 0.00143
#> 7 1999 F Anna 9092 0.00467
#> 8 2011 F Anna 5649 0.00292
#> 9 1952 F Ashley 24 0.0000126
#> 10 2006 F Ashley 12340 0.00591
#> # ... with 90 more rows
I think raising an error and forcing the user to clarify their code is a better than choosing one of these options and not doing what the user expects.
Thanks for visiting r-craft.org
This article is originally published at https://tjmahr.github.io/
Please visit source website for post related comments.
