Check ‘Developer Tools’ First To Avoid Heavy-ish Dependencies
This article is originally published at https://rud.is/b
Guillaume Pressiat (@GuillaumePressiat) did a solid post & video on using Selenium to scrape a paginated table from understat[.]com/league/EPL/2020 (I just cannot bring myself to provide an active link to any SportsBall site). He does a great job walking folks through acquiring & orchestrating the heavy dependency that is Selenium.
I did a quick “look at browser Developer Tools” tweet a few weeks back that included the entire code for retrieving the Forbes billionaires list via the JSON file the Forbes’ site loads via an XHR request responding to a similar fine article by another R user on using Selenium to do the same thing.
If you find yourself thwarted by rvest::read_html() not returning “nodes that are clearly there” it is likely due to the page rendering nodes dynamically via javascript. Selenium orchestrates full or headless browsers and lets you scrape the dynamically rendered DOM. You can see this yourself if you first view the source of an HTML page (via the browser’s “view source” menu) and then use Developer Tools to inspect the browser session. The “view source” view (in Blink-based browsers, at least) will be the raw, unrendered source HTML from the site and the DevTools “Elements” tab will have the rendered DOM elements.
The “Nework” tab of DevTools has an “XHR” tab of its own, but if you try to use it on this SportsBall site to see the JSON it loads, you’ll be bitterly disappointed because — while it does indeed render JSON into HTML DOM nodes dynamically — that JSON is embedded in the web page:
We can work in two different ways without the use of Selenium.
First, we’ll “cheat” and use the {V8} package, which is an R interface to a javascript virtual machine, the type of which browsers use to run javascript on web pages. I say “cheat” because we’re still depending on a chunk of a browser engine.
Let’s get some boilerplate out of the way:
library(V8) # V8 engine
library(rvest) # Scraping
library(stringi) # String manipulation which we'll use later
library(tidyverse) # Duh
ctx <- v8() # create a new instance of the javascript VM
pg <- read_html("https://understat.com/league/EPL/2020") # read sportsball page
If you examine the SportsBall page you’ll see that JSON.parse in a few different locations, let’s target them all:
html_nodes(pg, xpath = ".//script[contains(., 'JSON.parse')]")
## {xml_nodeset (4)}
## [1] <script>\n\tvar datesData \t= JSON.parse('\\x5B\\x7B\\x22id\\x22\\x3A\\x2214086 ...
## [2] <script>\n\tvar teamsData = JSON.parse('\\x7B\\x2271\\x22\\x3A\\x7B\\x22id\\x22 ...
## [3] <script>\n\tvar playersData\t= JSON.parse('\\x5B\\x7B\\x22id\\x22\\x3A\\x22647\ ...
## [4] <script>\n\t\tWebFont.load({\n\t\t\tgoogle: {\n\t\t\t\tfamilies: ['Barlow:500', ...
We don’t need that last one, so the first three contain all the data we need.
Turning that into data is pretty straightforward work:
html_nodes(pg, xpath = ".//script[contains(., 'JSON.parse')]") %>%
.[1:3] %>% # only want the first three nodes
html_text() %>% # turn the nodes into text
walk(ctx$eval) # tell V8 to evaluate the javascript
The VM we created now has those three variables:
ctx$get(JS("Object.keys(global)"))
## [1] "print" "console" "global" "datesData" "_week"
## [6] "_year" "teamsData" "playersData"
and, we can retrieve them like this:
as_tibble(ctx$get("datesData"))
## A tibble: 380 x 8
## id isResult h$id $title $short_title a$id $title $short_title goals$h $a
## <chr> <lgl> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 14086 TRUE 228 Fulham FLH 83 Arsen… ARS 0 3
## 2 14087 TRUE 78 Cryst… CRY 74 South… SOU 1 0
## 3 14090 TRUE 87 Liver… LIV 245 Leeds LED 4 3
## 4 14091 TRUE 81 West … WHU 86 Newca… NEW 0 2
## 5 14092 TRUE 76 West … WBA 75 Leice… LEI 0 3
## 6 14093 TRUE 82 Totte… TOT 72 Evert… EVE 0 1
## 7 14094 TRUE 238 Sheff… SHE 229 Wolve… WOL 0 2
## 8 14095 TRUE 220 Brigh… BRI 80 Chels… CHE 1 3
## 9 14096 TRUE 72 Evert… EVE 76 West … WBA 5 2
## 10 14097 TRUE 245 Leeds LED 228 Fulham FLH 4 3
## # … with 370 more rows, and 6 more variables: xG$h <chr>, $a <chr>, datetime <chr>,
## # forecast$w <chr>, $d <chr>, $l <chr>
Note that we need to do some extra processing of the second one to make it a bit tidier:
ctx$get("teamsData") %>%
map_df(~{
.x$history$id <- .x$id
.x$history$title <- .x$title
.x$history
}) %>%
as_tibble()
## # A tibble: 616 x 21
## h_a xG xGA npxG npxGA ppda$att $def ppda_allowed$att $def deep
## <chr> <dbl> <dbl> <dbl> <dbl> <int> <int> <int> <int> <int>
## 1 h 0.805 0.850 0.805 0.0885 89 20 247 14 17
## 2 a 2.03 0.535 2.03 0.535 307 33 143 24 10
## 3 h 3.08 1.66 3.08 1.66 365 25 119 25 7
## 4 a 0.874 0.672 0.874 0.672 212 23 210 24 7
## 5 h 1.50 2.38 1.50 2.38 225 17 124 34 7
## 6 h 2.45 1.00 1.69 1.00 161 23 164 22 5
## 7 a 1.99 1.39 1.99 1.39 331 24 169 15 16
## 8 h 1.77 1.50 1.77 1.50 257 14 208 17 6
## 9 a 2.39 0.572 1.63 0.572 144 11 289 23 8
## 10 a 1.27 1.14 0.508 1.14 162 28 166 20 5
## # … with 606 more rows, and 13 more variables: deep_allowed <int>, scored <int>,
## # missed <int>, xpts <dbl>, result <chr>, date <chr>, wins <int>, draws <int>,
## # loses <int>, pts <int>, npxGD <dbl>, id <chr>, title <chr>
The last one does not need any extra help:
as_tibble(ctx$get("playersData"))
## # A tibble: 505 x 18
## id player_name games time goals xG assists xA shots key_passes
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 647 Harry Kane 29 2557 19 17.6… 13 6.73… 113 39
## 2 1250 Mohamed Sa… 30 2529 19 16.1… 3 4.55… 99 40
## 3 1228 Bruno Fern… 31 2659 16 13.4… 11 10.8… 95 87
## 4 453 Son Heung-… 30 2509 14 9.35… 9 8.03… 55 56
## 5 822 Patrick Ba… 31 2572 14 14.8… 7 3.44… 93 24
## 6 5555 Dominic Ca… 26 2248 14 15.7… 0 0.95… 65 13
## 7 3277 Alexandre … 27 1818 13 11.8… 2 1.77… 43 21
## 8 314 Ilkay Günd… 24 1776 12 8.64… 1 3.29… 45 35
## 9 755 Jamie Vardy 27 2230 12 16.0… 7 4.26… 64 22
## 10 8865 Ollie Watk… 30 2700 12 13.7… 3 4.15… 81 36
## # … with 495 more rows, and 8 more variables: yellow_cards <chr>, red_cards <chr>,
## # position <chr>, team_title <chr>, npg <chr>, npxG <chr>, xGChain <chr>,
## # xGBuildup <chr>
We don’t really need {V8} for this, though, if we’re willing to use some regular expressions. We have to be a bit careful since some extra, non-JSON data comes along for the ride with that first <script> tag (see the embedded image above).
We perform the same initial setup (get the text of the first three <script> tags), then we erase everthing that isn’t JSON data (so all the var and javascript punctuation). By using comments = TRUE in call to stri_replace_all_regex we can provide documentation along with the (ugly) regex.
The creator of the SportsBall site did some encoding to make the string easier to shove into a <script> tag, so we need to undo that by converting the hex-escapes to HTML entity escapes (replace \x with %) and then decoding them with curl::curl_unescape()
We could have made the regular expression uglier to avoid the other javascript cruft in the first <script> tag, but it’s just as easy to split them all into lines and pull the first line out.
Then, it’s just a matter of running each one through jsonlite::fromJSON(). I kept it a list and just set the names as the names of the variables above.
html_nodes(pg, xpath = ".//script[contains(., 'JSON.parse')]") %>%
.[1:3] %>%
html_text() %>%
stri_replace_all_regex("
^[^\\(]+\\(' # remove everything from the beginning of the line to the first ('
| # OR
'\\)[;,][[:space:]]* # remove the last ') and everything after it
$
", "", comments = TRUE, multiline = TRUE) %>%
stri_replace_all_fixed("\\x", "%") %>%
curl::curl_unescape() %>%
stri_split_lines() %>%
map_chr(1) %>%
map(jsonlite::fromJSON) %>%
map(as_tibble) %>%
set_names(c("datesData", "teamsData", "playersData")) %>%
str(3)
## List of 3
## $ datesData : tibble [380 × 8] (S3: tbl_df/tbl/data.frame)
## ..$ id : chr [1:380] "14086" "14087" "14090" "14091" ...
## ..$ isResult: logi [1:380] TRUE TRUE TRUE TRUE TRUE TRUE ...
## ..$ h :'data.frame': 380 obs. of 3 variables:
## ..$ a :'data.frame': 380 obs. of 3 variables:
## ..$ goals :'data.frame': 380 obs. of 2 variables:
## ..$ xG :'data.frame': 380 obs. of 2 variables:
## ..$ datetime: chr [1:380] "2020-09-12 11:30:00" "2020-09-12 14:00:00" "2020-09-12 16:30:00" "2020-09-12 19:00:00" ...
## ..$ forecast:'data.frame': 380 obs. of 3 variables:
## $ teamsData : tibble [3 × 20] (S3: tbl_df/tbl/data.frame)
## ..$ 71 :List of 3
## ..$ 72 :List of 3
## ..$ 74 :List of 3
## ..$ 75 :List of 3
## ..$ 76 :List of 3
## ..$ 78 :List of 3
## ..$ 80 :List of 3
## ..$ 81 :List of 3
## ..$ 82 :List of 3
## ..$ 83 :List of 3
## ..$ 86 :List of 3
## ..$ 87 :List of 3
## ..$ 88 :List of 3
## ..$ 89 :List of 3
## ..$ 92 :List of 3
## ..$ 220:List of 3
## ..$ 228:List of 3
## ..$ 229:List of 3
## ..$ 238:List of 3
## ..$ 245:List of 3
## $ playersData: tibble [505 × 18] (S3: tbl_df/tbl/data.frame)
## ..$ id : chr [1:505] "647" "1250" "1228" "453" ...
## ..$ player_name : chr [1:505] "Harry Kane" "Mohamed Salah" "Bruno Fernandes" "Son Heung-Min" ...
## ..$ games : chr [1:505] "29" "30" "31" "30" ...
## ..$ time : chr [1:505] "2557" "2529" "2659" "2509" ...
## ..$ goals : chr [1:505] "19" "19" "16" "14" ...
## ..$ xG : chr [1:505] "17.650331255048513" "16.19410896115005" "13.438796618022025" "9.352356541901827" ...
## ..$ assists : chr [1:505] "13" "3" "11" "9" ...
## ..$ xA : chr [1:505] "6.7384555246680975" "4.557050030678511" "10.812157344073057" "8.036493374034762" ...
## ..$ shots : chr [1:505] "113" "99" "95" "55" ...
## ..$ key_passes : chr [1:505] "39" "40" "87" "56" ...
## ..$ yellow_cards: chr [1:505] "1" "0" "5" "0" ...
## ..$ red_cards : chr [1:505] "0" "0" "0" "0" ...
## ..$ position : chr [1:505] "F" "F S" "M S" "F M S" ...
## ..$ team_title : chr [1:505] "Tottenham" "Liverpool" "Manchester United" "Tottenham" ...
## ..$ npg : chr [1:505] "15" "13" "8" "14" ...
## ..$ npxG : chr [1:505] "14.605655785650015" "11.627095961943269" "6.5883138151839375" "9.352356541901827" ...
## ..$ xGChain : chr [1:505] "20.556765687651932" "21.694580920040607" "22.04182725213468" "17.928756553679705" ...
## ..$ xGBuildup : chr [1:505] "3.99019683804363" "8.287332298234105" "8.843060294166207" "5.881684513762593" ...
You can use the cleanup code from the {V8} example to reshape that second element, and readr::type_convert() can help you turn the character vectors into something more useful.
FIN
I really always pays to at take the DevTools look at page before introducing heavy dependencies. More sites are using very straightforward idioms that make the dynamically rendered page JSON source data readily available. Further, sites often add extra fields that you don’t see rendered, but may be useful to have around as you work with the resulting data.
Thanks for visiting r-craft.org
This article is originally published at https://rud.is/b
Please visit source website for post related comments.
