Iteration

# Iteration
## <br><br> Introduction to Data Science
### <a href="https://introds.org/">introds.org</a>
### <br> Dr. Mine Çetinkaya-Rundel

---

layout: true
  
<div class="my-footer">
<span>
<a href="https://introds.org" target="_blank">introds.org</a>
</span>
</div>

---

# First Minister's COVID speeches

---

## 🏁 Start with

---

## End with 🛑

```
## # A tibble: 141 x 6
##    title      date       location   abstract      text  url      
##    <chr>      <date>     <chr>      <chr>         <lis> <chr>    
##  1 Coronavir… 2020-10-26 St Andrew… Statement gi… <chr… https://…
##  2 Coronavir… 2020-10-23 St Andrew… Statement gi… <chr… https://…
##  3 Coronavir… 2020-10-22 St Andrew… Statement gi… <chr… https://…
##  4 Coronavir… 2020-10-21 St Andrew… Statement gi… <chr… https://…
##  5 Coronavir… 2020-10-20 St Andrew… Statement gi… <chr… https://…
##  6 Coronavir… 2020-10-19 St Andrew… Statement gi… <chr… https://…
##  7 Coronavir… 2020-10-16 St Andrew… Statement gi… <chr… https://…
##  8 Coronavir… 2020-10-15 Scottish … Statement gi… <chr… https://…
##  9 Coronavir… 2020-10-14 St Andrew… Statement gi… <chr… https://…
## 10 Coronavir… 2020-10-13 St Andrew… Statement gi… <chr… https://…
## 11 Coronavir… 2020-10-12 St Andrew… Statement gi… <chr… https://…
## 12 Coronavir… 2020-10-09 St Andrew… Statement gi… <chr… https://…
## 13 Coronavir… 2020-10-07 Scottish … Statement gi… <chr… https://…
## 14 Coronavir… 2020-10-06 St Andrew… Statement gi… <chr… https://…
## 15 Coronavir… 2020-10-05 St Andrew… Statement gi… <chr… https://…
## # … with 126 more rows
```

---

## Define `scrape_speech()`

```r
scrape_speech <- function(url) {
  
  speech_page <- read_html(url)

title <- speech_page %>%
    html_node(".article-header__title") %>%
    html_text()

date <- speech_page %>%
    html_node(".content-data__list:nth-child(1) strong") %>%
    html_text() %>%
    dmy()

location <- speech_page %>%
    html_node(".content-data__list+ .content-data__list strong") %>%
    html_text()

abstract <- speech_page %>%
    html_node(".leader--first-para p") %>%
    html_text()

text <- speech_page %>%
    html_nodes("#preamble p") %>%
    html_text() %>%
    list()

tibble(
    title = title, date = date, location = location,
    abstract = abstract, text = text, url = url
  )
}
```
]
]

---

## Use `scrape_speech()`

```r
url_26_oct <- "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-26-october/"
scrape_speech(url = url_26_oct)
```

```
## # A tibble: 1 x 6
##   title       date       location   abstract     text  url       
##   <chr>       <date>     <chr>      <chr>        <lis> <chr>     
## 1 Coronaviru… 2020-10-26 St Andrew… Statement g… <chr… https://w…
```

```r
url_23_oct <- "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-23-october/"
scrape_speech(url = url_23_oct)
```

```
## # A tibble: 1 x 6
##   title       date       location   abstract     text  url       
##   <chr>       <date>     <chr>      <chr>        <lis> <chr>     
## 1 Coronaviru… 2020-10-23 St Andrew… Statement g… <chr… https://w…
```

---

# Inputs

---

## Inputs

.question[
You now have a function that will scrape the relevant info on speeches given the URL of the page of the speech. Where can we get a list of URLs of each of the speeches?
]

---

## All URLs

```r
all_speeches_page <- read_html("https://www.gov.scot/collections/first-ministers-speeches/")

all_speeches_page %>%
  html_nodes(".collections-list a") %>%
  html_attr("href")
```

```
##   [1] "/publications/coronavirus-covid-19-update-first-ministers-speech-26-october/"                        
##   [2] "/publications/coronavirus-covid-19-update-first-ministers-speech-23-october/"                        
##   [3] "/publications/coronavirus-covid-19-update-first-ministers-speech-22-october/"                        
##   [4] "/publications/coronavirus-covid-19-update-first-ministers-speech-21-october/"                        
##   [5] "/publications/coronavirus-covid-19-update-first-ministers-speech-20-october/"                        
##   [6] "/publications/coronavirus-covid-19-update-first-ministers-speech-19-october/"                        
##   [7] "/publications/coronavirus-covid-19-update-first-ministers-speech-16-october-2020/"                   
##   [8] "/publications/coronavirus-covid-19-update-first-ministers-speech-15-october-2020/"                   
##   [9] "/publications/coronavirus-covid-19-update-first-ministers-speech-14-october-2020/"                   
##  [10] "/publications/coronavirus-covid-19-update-first-ministers-speech-13-october-2020/"                   
...
```

---

## COVID-19 URLs *fragments*

```r
all_speeches_page %>%
  html_nodes(".collections-list a") %>%
  html_attr("href") %>%
  str_subset("covid-19")
```

```
##   [1] "/publications/coronavirus-covid-19-update-first-ministers-speech-26-october/"       
##   [2] "/publications/coronavirus-covid-19-update-first-ministers-speech-23-october/"       
##   [3] "/publications/coronavirus-covid-19-update-first-ministers-speech-22-october/"       
##   [4] "/publications/coronavirus-covid-19-update-first-ministers-speech-21-october/"       
##   [5] "/publications/coronavirus-covid-19-update-first-ministers-speech-20-october/"       
##   [6] "/publications/coronavirus-covid-19-update-first-ministers-speech-19-october/"       
##   [7] "/publications/coronavirus-covid-19-update-first-ministers-speech-16-october-2020/"  
##   [8] "/publications/coronavirus-covid-19-update-first-ministers-speech-15-october-2020/"  
##   [9] "/publications/coronavirus-covid-19-update-first-ministers-speech-14-october-2020/"  
##  [10] "/publications/coronavirus-covid-19-update-first-ministers-speech-13-october-2020/"  
...
```

---

## COVID-19 URLs

```r
all_speeches_page %>%
  html_nodes(".collections-list a") %>%
  html_attr("href") %>%
  str_subset("covid-19") %>%
  str_c("https://www.gov.scot", .)
```

```
##   [1] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-26-october/"       
##   [2] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-23-october/"       
##   [3] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-22-october/"       
##   [4] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-21-october/"       
##   [5] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-20-october/"       
##   [6] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-19-october/"       
##   [7] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-16-october-2020/"  
##   [8] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-15-october-2020/"  
##   [9] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-14-october-2020/"  
##  [10] "https://www.gov.scot/publications/coronavirus-covid-19-update-first-ministers-speech-13-october-2020/"  
...
```

---

## Save COVID-19 URLs

```r
covid_speech_urls <- all_speeches_page %>%
  html_nodes(".collections-list a") %>%
  html_attr("href") %>%
  str_subset("covid-19") %>%
  str_c("https://www.gov.scot", .)

covid_speech_urls
```

---

# Iteration

---

## Define the task

- Goal: Scrape info on all COVID-19 speeches of the First Minister
- So far:

```r
scrape_speech(covid_speech_urls[1])
scrape_speech(covid_speech_urls[2])
scrape_speech(covid_speech_urls[3])
```
- What else do we need to do?
  - Run the `scrape_speech()` function on all COVID-19 speech links
  - Combine the resulting data frames from each run into one giant data frame

---

## Iteration

.question[
How can we tell R to apply the `scrape_speech()` function to each link in `covid_speech_urls`?
]

- Option 1: Write a **for loop**, i.e. explicitly tell R to visit a link, apply the function, store the result, then visit the next link, apply the function, append the result to the stored result from the previous link, and so on and so forth.
--

- Option 2: **Map** the function to each element in the list of links, and let R take care of the storing and appending of results.
--

- We'll go with Option 2!

---

## How does mapping work?

Suppose we have exam 1 and exam 2 scores of 4 students stored in a list...

```r
exam_scores <- list(
  exam1 <- c(80, 90, 70, 50),
  exam2 <- c(85, 83, 45, 60)
)
```

...and we find the mean score in each exam

```r
map(exam_scores, mean)
```

```
## [[1]]
## [1] 72.5
## 
## [[2]]
## [1] 68.25
```

---

...and suppose we want the results as a numeric (double) vector

```r
map_dbl(exam_scores, mean)
```

```
## [1] 72.50 68.25
```

...or as a character string

```r
map_chr(exam_scores, mean)
```

```
## [1] "72.500000" "68.250000"
```

---

## `map_something`

Functions for looping over an object and returning a value (of a specific type):

* `map()` - returns a list
* `map_lgl()` - returns a logical vector
* `map_int()` - returns a integer vector
* `map_dbl()` - returns a double vector
* `map_chr()` - returns a character vector
* `map_df()` / `map_dfr()` - returns a data frame by row binding
* `map_dfc()` - returns a data frame by column binding
* ...

---

## Go to each page, scrape speech

- Map the `scrape_speech()` function
- to each element of `covid_speech_urls`
- and return a data frame by row binding

```r
covid_speeches <- map_dfr(covid_speech_urls, scrape_speech)
```

---

```r
covid_speeches %>%
  print(n = 15)
```

---

## What could go wrong?

```r
covid_speeches <- map_dfr(covid_speech_urls, scrape_speech)
```

- This will take a while to run
- If you get `HTTP Error 429 (Too many requests)` you might want to slow down your hits by modifying your function to slow it down by adding a random wait (sleep) time between hitting each link

```r
scrape_speech <- function(url){
  
  # Sleep for randomly generated number of seconds
  # Generated from a uniform distribution between 0 and 1
* Sys.sleep(runif(1))
  
  # Rest of your function code goes here...
}
```