Click-and-Download

In the simplest case, the data you need is already on the internet in a tabular format. There are a couple of strategies here:

• Use read.csv or readr::read_csv to read the data straight into R
• Use the downloader package or curl from the shell to download the file and store a local copy, then use read_csv or something similar to read the data into R
  • Even if the file disappears from the internet, you have a local copy cached

Even in this instance, files may need cleaning and transformation when you bring them into R.

Data supplied on the web

Many times, the data that you want is not already organized into one or a few tables that you can read directly into R. More frequently, you find this data is given in the form of an API. Application Programming Interfaces (APIs) are descriptions of the kind of requests that can be made of a certain piece of software, and descriptions of the kind of answers that are returned. Many sources of data - databases, websites, services - have made all (or part) of their data available via APIs over the internet. Computer programs (“clients”) can make requests of the server, and the server will respond by sending data (or an error message). This client can be many kinds of other programs or websites, including R running from your laptop.

Install and play packages

Many common web services and APIs have been “wrapped”, i.e. R and Python functions have been written around them which send your query to the server and format the response.

Why do we want this?

• provenance
• reproducible
• updating
• ease
• scaling

Search birds by geography

The ebird website categorizes some popular locations as “Hotspots”. These are areas where there are both lots of birds and lots of birders. Once such location is at Lincoln Park Zoo in Chicago. You can see data for this site at http://ebird.org/ebird/hotspot/L1573785

At that link, you can see a page like this:

Lincoln Park Zoo

The data already look to be organized in a data frame! rebird allows us to read these data directly into R. (The ID code for Lincoln Park Zoo is **“L1573785**)

ebirdhotspot(locID = "L1573785") %>%
  tbl_df()

## Warning: `rbind_all()` is deprecated. Please use `bind_rows()` instead.

## # A tibble: 23 × 11
##          lng                   locName howMany                sciName
##        <dbl>                     <chr>   <int>                  <chr>
## 1  -87.63272 Lincoln Park Zoo, Chicago      11             Aix sponsa
## 2  -87.63272 Lincoln Park Zoo, Chicago       2          Anas rubripes
## 3  -87.63272 Lincoln Park Zoo, Chicago     160     Anas platyrhynchos
## 4  -87.63272 Lincoln Park Zoo, Chicago       1     Larus delawarensis
## 5  -87.63272 Lincoln Park Zoo, Chicago       2  Corvus brachyrhynchos
## 6  -87.63272 Lincoln Park Zoo, Chicago      14       Sturnus vulgaris
## 7  -87.63272 Lincoln Park Zoo, Chicago       3  Cardinalis cardinalis
## 8  -87.63272 Lincoln Park Zoo, Chicago       3      Passer domesticus
## 9  -87.63272 Lincoln Park Zoo, Chicago      12      Branta canadensis
## 10 -87.63272 Lincoln Park Zoo, Chicago       1 Zonotrichia albicollis
## # ... with 13 more rows, and 7 more variables: obsValid <lgl>,
## #   locationPrivate <lgl>, obsDt <chr>, obsReviewed <lgl>, comName <chr>,
## #   lat <dbl>, locID <chr>

We can use the function ebirdgeo to get a list for an area. (Note that South and West are negative):

chibirds <- ebirdgeo(lat = 41.8781, lng = -87.6298)

## Warning: `rbind_all()` is deprecated. Please use `bind_rows()` instead.

chibirds %>%
  tbl_df() %>%
  head() %>%
    kable()

Note: Check the defaults on this function. e.g. radius of circle, time of year.

We can also search by “region”, which refers to short codes which serve as common shorthands for different political units. For example, France is represented by the letters FR

frenchbirds <- ebirdregion("FR")

## Warning: `rbind_all()` is deprecated. Please use `bind_rows()` instead.

frenchbirds %>%
  tbl_df() %>%
  head() %>%
  kable()

Find out WHEN a bird has been seen in a certain place! Choosing a name from chibirds above (the Bald Eagle):

warbler <- ebirdgeo(species = 'Setophaga coronata', lat = 41.8781, lng = -87.6298)

## Warning: `rbind_all()` is deprecated. Please use `bind_rows()` instead.

warbler %>%
  tbl_df() %>%
  head() %>%
  kable()

rebird knows where you are:

ebirdgeo(species = 'Setophaga coronata') %>%
  tbl_df() %>%
  head() %>%
  kable()

## Warning: As a complete lat/long pair was not provided, your location was
## determined using your computer's public-facing IP address. This will likely
## not reflect your physical location if you are using a remote server or
## proxy.

## Warning: `rbind_all()` is deprecated. Please use `bind_rows()` instead.

API authentication

Many APIs require you to register for access. This allows them to track which users are submitting queries and manage demand - if you submit too many queries too quickly, you might be rate-limited and your requests de-prioritized or blocked. Always check the API access policy of the web site to determine what these limits are.

There are a few things we need to do to be able to use this package to access the geonames API:

1. go to the geonames site and register an account.
2. click here to enable the free web service
3. Tell R your geonames username. You could run the line

options(geonamesUsername = "my_user_name")

in R. However this is insecure. We don’t want to risk committing this line and pushing it to our public GitHub page! Instead, you should create a file in the same place as your .Rproj file. Name this file .Rprofile, and add

options(geonamesUsername = "my_user_name")

to that file.

# in .Rprofile
options("this_is_my_key" = XXXX)

# later, in the R script:
key <- getOption("this_is_my_key")

Using Geonames

What can we do? Get access to lots of geographical information via the various “web services”

countryInfo <- GNcountryInfo()

countryInfo %>%
  tbl_df() %>%
  head() %>%
  kable()

This country info dataset is very helpful for accessing the rest of the data, because it gives us the standardized codes for country and language.

Relative frequency plot

out <- plosword(list("alcohol", "heroin", "marijuana"),
    vis = "TRUE")
out$table

##   No_Articles      Term
## 1       22873   alcohol
## 2         747    heroin
## 3         518 marijuana

out$plot

Here is a list of options for the search.

Plots over time

plot_throughtime(terms = c("alcohol", "heroin", "marijuana"), limit = 200)

## Warning: Removed 5 rows containing missing values (geom_path).

Packages

• twitteR is the most popular package for R, but it only allows you to access the REST API.
• streamR is more complicated, but allows you to query the Streaming API from R.
• tweepy is a package for Python that allows you to access both the REST and Streaming APIs.

Here, we are going to practice using the twitteR package to search Twitter.

OAuth authentication

OAuth is an open standard for authorization, commonly used as a way for Internet users to authorize websites or applications to access their information on other websites but without giving them the passwords. This still requires an API key, but is a bit more complicated to setup.

1. The first step is to create a Twitter application for yourself. In order to do this, you do need to have a registered Twitter account. Go to (https://apps.twitter.com/app/new) and log in. After filling in the basic info (make sure to set the “Callback URL” to (http://127.0.0.1:1410)), go to the Permissions tab and select “Read, Write and Access direct messages”. Make sure to click on the save button after doing this. In the Details tab, take note of your consumer key and consumer secret.

2. Store your API key and token using the .Rprofile method. Edit .Rprofile and add

   options(twitter_api_key = "Your API key")
   options(twitter_api_token = "Your API secret")

   Restart RStudio to get the keys loaded into your environment.

3. Run from the console:

   setup_twitter_oauth(consumer_key = getOption("twitter_api_key"),
                       consumer_secret = getOption("twitter_api_token"))

   This will open your browser and take you to a page to authorize your app to access your Twitter account. You need this in order to search any content on Twitter.

4. At this point you should get a message back in RStudio “Authentication complete.” You’re done setting up the authentication for twitteR.

You must do this in order to query Twitter from within a non-interactive session (i.e. an R Markdown document or using source() to run a script). If you do not, you will get an error because R will not be able to finish the authentication process without your input.

Searching tweets

## [1] "Using browser based authentication"

tweets <- searchTwitter('#rstats', n = 5)
tweets

## [[1]]
## [1] "LovedeepSG: RT @Rbloggers: R Jobs for R users – 10 jobs from around the world (2016-11-17) https://t.co/w7xBuwQFf3 #rstats #DataScience"
## 
## [[2]]
## [1] "tarakc02: RT @tarakc02: Currently in development: rmapzen, an #rstats package to access the @mapzen search API https://t.co/L3QVVNEtZM -- feedback ap…"
## 
## [[3]]
## [1] "G_Devailly: New #rstats coding style makes it as readable as #perl : https://t.co/pRxUmAsh9f"
## 
## [[4]]
## [1] "atiretoo: Definition of existential pain = looking at code I wrote 2 years ago #rstats"
## 
## [[5]]
## [1] "groundwalkergmb: @stefanbache @hspter @carroll_jono Maybe, though that does not play nice with OOP in #rstats (of the S3 OR S4 variety)"

Searching users

Use getUser() which returns a user object that you can perform additional functions on. This only works for users with public profiles or those that have authorized your app.

clinton <- getUser("hillaryclinton")
clinton$getDescription()

## [1] "Wife, mom, grandma, women+kids advocate, FLOTUS, Senator, SecState, hair icon, pantsuit aficionado, 2016 presidential candidate. Tweets from Hillary signed –H"

clinton$getFriends(n = 5)

## $`3004231`
## [1] "SnoopDogg"
## 
## $`3153892631`
## [1] "RobbyMook"
## 
## $`18730233`
## [1] "daniellekantor"
## 
## $`4732338444`
## [1] "RyanForRecovery"

Tidying tweets

By default twitteR returns tweets in complex lists.

str(tweets)

## List of 5
##  $ :Reference class 'status' [package "twitteR"] with 17 fields
##   ..$ text         : chr "RT @Rbloggers: R Jobs for R users – 10 jobs from around the world (2016-11-17) https://t.co/w7xBuwQFf3 #rstats #DataScience"
##   ..$ favorited    : logi FALSE
##   ..$ favoriteCount: num 0
##   ..$ replyToSN    : chr(0) 
##   ..$ created      : POSIXct[1:1], format: "2016-11-17 16:07:11"
##   ..$ truncated    : logi FALSE
##   ..$ replyToSID   : chr(0) 
##   ..$ id           : chr "799282748812763137"
##   ..$ replyToUID   : chr(0) 
##   ..$ statusSource : chr "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>"
##   ..$ screenName   : chr "LovedeepSG"
##   ..$ retweetCount : num 4
##   ..$ isRetweet    : logi TRUE
##   ..$ retweeted    : logi FALSE
##   ..$ longitude    : chr(0) 
##   ..$ latitude     : chr(0) 
##   ..$ urls         :'data.frame':    1 obs. of  5 variables:
##   .. ..$ url         : chr "https://t.co/w7xBuwQFf3"
##   .. ..$ expanded_url: chr "https://wp.me/pMm6L-Bd5"
##   .. ..$ display_url : chr "wp.me/pMm6L-Bd5"
##   .. ..$ start_index : num 79
##   .. ..$ stop_index  : num 102
##   ..and 53 methods, of which 39 are  possibly relevant:
##   ..  getCreated, getFavoriteCount, getFavorited, getId, getIsRetweet,
##   ..  getLatitude, getLongitude, getReplyToSID, getReplyToSN,
##   ..  getReplyToUID, getRetweetCount, getRetweeted, getRetweeters,
##   ..  getRetweets, getScreenName, getStatusSource, getText, getTruncated,
##   ..  getUrls, initialize, setCreated, setFavoriteCount, setFavorited,
##   ..  setId, setIsRetweet, setLatitude, setLongitude, setReplyToSID,
##   ..  setReplyToSN, setReplyToUID, setRetweetCount, setRetweeted,
##   ..  setScreenName, setStatusSource, setText, setTruncated, setUrls,
##   ..  toDataFrame, toDataFrame#twitterObj
##  $ :Reference class 'status' [package "twitteR"] with 17 fields
##   ..$ text         : chr "RT @tarakc02: Currently in development: rmapzen, an #rstats package to access the @mapzen search API https://t.co/L3QVVNEtZM --"| __truncated__
##   ..$ favorited    : logi FALSE
##   ..$ favoriteCount: num 0
##   ..$ replyToSN    : chr(0) 
##   ..$ created      : POSIXct[1:1], format: "2016-11-17 16:07:07"
##   ..$ truncated    : logi FALSE
##   ..$ replyToSID   : chr(0) 
##   ..$ id           : chr "799282731012083712"
##   ..$ replyToUID   : chr(0) 
##   ..$ statusSource : chr "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>"
##   ..$ screenName   : chr "tarakc02"
##   ..$ retweetCount : num 3
##   ..$ isRetweet    : logi TRUE
##   ..$ retweeted    : logi FALSE
##   ..$ longitude    : chr(0) 
##   ..$ latitude     : chr(0) 
##   ..$ urls         :'data.frame':    1 obs. of  5 variables:
##   .. ..$ url         : chr "https://t.co/L3QVVNEtZM"
##   .. ..$ expanded_url: chr "https://github.com/tarakc02/rmapzen"
##   .. ..$ display_url : chr "github.com/tarakc02/rmapz…"
##   .. ..$ start_index : num 101
##   .. ..$ stop_index  : num 124
##   ..and 53 methods, of which 39 are  possibly relevant:
##   ..  getCreated, getFavoriteCount, getFavorited, getId, getIsRetweet,
##   ..  getLatitude, getLongitude, getReplyToSID, getReplyToSN,
##   ..  getReplyToUID, getRetweetCount, getRetweeted, getRetweeters,
##   ..  getRetweets, getScreenName, getStatusSource, getText, getTruncated,
##   ..  getUrls, initialize, setCreated, setFavoriteCount, setFavorited,
##   ..  setId, setIsRetweet, setLatitude, setLongitude, setReplyToSID,
##   ..  setReplyToSN, setReplyToUID, setRetweetCount, setRetweeted,
##   ..  setScreenName, setStatusSource, setText, setTruncated, setUrls,
##   ..  toDataFrame, toDataFrame#twitterObj
##  $ :Reference class 'status' [package "twitteR"] with 17 fields
##   ..$ text         : chr "New #rstats coding style makes it as readable as #perl : https://t.co/pRxUmAsh9f"
##   ..$ favorited    : logi FALSE
##   ..$ favoriteCount: num 0
##   ..$ replyToSN    : chr(0) 
##   ..$ created      : POSIXct[1:1], format: "2016-11-17 16:05:24"
##   ..$ truncated    : logi FALSE
##   ..$ replyToSID   : chr(0) 
##   ..$ id           : chr "799282297006592000"
##   ..$ replyToUID   : chr(0) 
##   ..$ statusSource : chr "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>"
##   ..$ screenName   : chr "G_Devailly"
##   ..$ retweetCount : num 0
##   ..$ isRetweet    : logi FALSE
##   ..$ retweeted    : logi FALSE
##   ..$ longitude    : chr(0) 
##   ..$ latitude     : chr(0) 
##   ..$ urls         :'data.frame':    0 obs. of  4 variables:
##   .. ..$ url         : chr(0) 
##   .. ..$ expanded_url: chr(0) 
##   .. ..$ dispaly_url : chr(0) 
##   .. ..$ indices     : num(0) 
##   ..and 53 methods, of which 39 are  possibly relevant:
##   ..  getCreated, getFavoriteCount, getFavorited, getId, getIsRetweet,
##   ..  getLatitude, getLongitude, getReplyToSID, getReplyToSN,
##   ..  getReplyToUID, getRetweetCount, getRetweeted, getRetweeters,
##   ..  getRetweets, getScreenName, getStatusSource, getText, getTruncated,
##   ..  getUrls, initialize, setCreated, setFavoriteCount, setFavorited,
##   ..  setId, setIsRetweet, setLatitude, setLongitude, setReplyToSID,
##   ..  setReplyToSN, setReplyToUID, setRetweetCount, setRetweeted,
##   ..  setScreenName, setStatusSource, setText, setTruncated, setUrls,
##   ..  toDataFrame, toDataFrame#twitterObj
##  $ :Reference class 'status' [package "twitteR"] with 17 fields
##   ..$ text         : chr "Definition of existential pain = looking at code I wrote 2 years ago #rstats"
##   ..$ favorited    : logi FALSE
##   ..$ favoriteCount: num 0
##   ..$ replyToSN    : chr(0) 
##   ..$ created      : POSIXct[1:1], format: "2016-11-17 15:59:59"
##   ..$ truncated    : logi FALSE
##   ..$ replyToSID   : chr(0) 
##   ..$ id           : chr "799280933819949056"
##   ..$ replyToUID   : chr(0) 
##   ..$ statusSource : chr "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>"
##   ..$ screenName   : chr "atiretoo"
##   ..$ retweetCount : num 0
##   ..$ isRetweet    : logi FALSE
##   ..$ retweeted    : logi FALSE
##   ..$ longitude    : chr(0) 
##   ..$ latitude     : chr(0) 
##   ..$ urls         :'data.frame':    0 obs. of  4 variables:
##   .. ..$ url         : chr(0) 
##   .. ..$ expanded_url: chr(0) 
##   .. ..$ dispaly_url : chr(0) 
##   .. ..$ indices     : num(0) 
##   ..and 53 methods, of which 39 are  possibly relevant:
##   ..  getCreated, getFavoriteCount, getFavorited, getId, getIsRetweet,
##   ..  getLatitude, getLongitude, getReplyToSID, getReplyToSN,
##   ..  getReplyToUID, getRetweetCount, getRetweeted, getRetweeters,
##   ..  getRetweets, getScreenName, getStatusSource, getText, getTruncated,
##   ..  getUrls, initialize, setCreated, setFavoriteCount, setFavorited,
##   ..  setId, setIsRetweet, setLatitude, setLongitude, setReplyToSID,
##   ..  setReplyToSN, setReplyToUID, setRetweetCount, setRetweeted,
##   ..  setScreenName, setStatusSource, setText, setTruncated, setUrls,
##   ..  toDataFrame, toDataFrame#twitterObj
##  $ :Reference class 'status' [package "twitteR"] with 17 fields
##   ..$ text         : chr "@stefanbache @hspter @carroll_jono Maybe, though that does not play nice with OOP in #rstats (of the S3 OR S4 variety)"
##   ..$ favorited    : logi FALSE
##   ..$ favoriteCount: num 1
##   ..$ replyToSN    : chr "stefanbache"
##   ..$ created      : POSIXct[1:1], format: "2016-11-17 15:59:23"
##   ..$ truncated    : logi FALSE
##   ..$ replyToSID   : chr "799273758242508800"
##   ..$ id           : chr "799280782963421185"
##   ..$ replyToUID   : chr "314708384"
##   ..$ statusSource : chr "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>"
##   ..$ screenName   : chr "groundwalkergmb"
##   ..$ retweetCount : num 0
##   ..$ isRetweet    : logi FALSE
##   ..$ retweeted    : logi FALSE
##   ..$ longitude    : chr(0) 
##   ..$ latitude     : chr(0) 
##   ..$ urls         :'data.frame':    0 obs. of  4 variables:
##   .. ..$ url         : chr(0) 
##   .. ..$ expanded_url: chr(0) 
##   .. ..$ dispaly_url : chr(0) 
##   .. ..$ indices     : num(0) 
##   ..and 53 methods, of which 39 are  possibly relevant:
##   ..  getCreated, getFavoriteCount, getFavorited, getId, getIsRetweet,
##   ..  getLatitude, getLongitude, getReplyToSID, getReplyToSN,
##   ..  getReplyToUID, getRetweetCount, getRetweeted, getRetweeters,
##   ..  getRetweets, getScreenName, getStatusSource, getText, getTruncated,
##   ..  getUrls, initialize, setCreated, setFavoriteCount, setFavorited,
##   ..  setId, setIsRetweet, setLatitude, setLongitude, setReplyToSID,
##   ..  setReplyToSN, setReplyToUID, setRetweetCount, setRetweeted,
##   ..  setScreenName, setStatusSource, setText, setTruncated, setUrls,
##   ..  toDataFrame, toDataFrame#twitterObj

To get the data into a data frame, use

df <- twListToDF(tweets) %>%
  tbl_df()
df

## # A tibble: 5 × 16
##                                                                          text
## *                                                                       <chr>
## 1 RT @Rbloggers: R Jobs for R users – 10 jobs from around the world (2016-11-
## 2 RT @tarakc02: Currently in development: rmapzen, an #rstats package to acce
## 3 New #rstats coding style makes it as readable as #perl : https://t.co/pRxUm
## 4 Definition of existential pain = looking at code I wrote 2 years ago #rstat
## 5 @stefanbache @hspter @carroll_jono Maybe, though that does not play nice wi
## # ... with 15 more variables: favorited <lgl>, favoriteCount <dbl>,
## #   replyToSN <chr>, created <dttm>, truncated <lgl>, replyToSID <chr>,
## #   id <chr>, replyToUID <chr>, statusSource <chr>, screenName <chr>,
## #   retweetCount <dbl>, isRetweet <lgl>, retweeted <lgl>, longitude <lgl>,
## #   latitude <lgl>

Practice using twitteR

1. Create a new R project on your computer. You can use Git for VCS or not - it is just for practice in class today
2. Setup your API key with a Twitter app
3. Authenticate using the twitteR package in R
4. Find the 50 most recent tweets by Donald Trump and store them in a data frame
   • userTimeline() can be used to retrieve tweets from individual users
   • searchTwitter() finds tweets from any public account that references the username

setup_twitter_oauth(consumer_key = getOption("twitter_api_key"),
                    consumer_secret = getOption("twitter_api_token"))

## [1] "Using browser based authentication"

trump <- userTimeline("realDonaldTrump", n = 50)

trump_df <- twListToDF(trump) %>%
  tbl_df()
trump_df

## # A tibble: 45 × 16
##                                                                           text
## *                                                                        <chr>
## 1  My transition team, which is working long hours and doing a fantastic job, 
## 2  Australia, New Zealand, and more. I am always available to them. @nytimes i
## 3  I have recieved and taken calls from many foreign leaders despite what the 
## 4  The failing @nytimes story is so totally wrong on transition. It is going s
## 5  I am not trying to get "top level security clearance" for my children. This
## 6  Very organized process taking place as I decide on Cabinet and many other p
## 7  The Electoral College is actually genius in that it brings all states, incl
## 8  If the election were based on total popular vote I would have campaigned in
## 9  The debates, especially the second and third, plus speeches and intensity o
## 10 I will be interviewed on @60Minutes tonight after the NFL game - 7:00 P.M. 
## # ... with 35 more rows, and 15 more variables: favorited <lgl>,
## #   favoriteCount <dbl>, replyToSN <lgl>, created <dttm>, truncated <lgl>,
## #   replyToSID <lgl>, id <chr>, replyToUID <lgl>, statusSource <chr>,
## #   screenName <chr>, retweetCount <dbl>, isRetweet <lgl>,
## #   retweeted <lgl>, longitude <lgl>, latitude <lgl>