Assessing word and document frequency

One measure of how important a word may be is its term frequency (tf), how frequently a word occurs within a document. The problem with this approach is that some words occur many times in a document, yet are probably not important (e.g. “the”, “is”, “of”). Instead, we want a way of downweighting words that are common across all documents, and upweighting words that are frequent within a small set of documents.

Another approach is to look at a term’s inverse document frequency (idf), which decreases the weight for commonly used words and increases the weight for words that are not used very much in a collection of documents. This can be combined with term frequency to calculate a term’s tf-idf, the frequency of a term adjusted for how rarely it is used. It is intended to measure how important a word is to a document in a collection (or corpus) of documents. It is a rule-of-thumb or heuristic quantity, not a theoretically proven method. The inverse document frequency for any given term is defined as

\[idf(\text{term}) = \ln{\left(\frac{n_{\text{documents}}}{n_{\text{documents containing term}}}\right)}\]

To calculate tf-idf for this set of documents, we will pool all the tweets from iPhone and Android together and treat them as if they are two total documents. Then we can calculate the frequency of terms in each group, and standardize that relative to the the term’s frequency across the entire corpus.

tweet_words_count <- tweet_words %>%
  count(source, word, sort = TRUE) %>%
  ungroup()
tweet_words_count

## # A tibble: 3,235 × 3
##     source                   word     n
##      <chr>                  <chr> <int>
## 1   iPhone             #trump2016   171
## 2  Android                hillary   124
## 3   iPhone #makeamericagreatagain    95
## 4  Android                crooked    93
## 5  Android                clinton    66
## 6  Android                 people    64
## 7   iPhone                hillary    52
## 8  Android                   cruz    50
## 9  Android                    bad    43
## 10  iPhone                america    43
## # ... with 3,225 more rows

total_words <- tweet_words_count %>%
  group_by(source) %>%
  summarize(total = sum(n))
total_words

## # A tibble: 2 × 2
##    source total
##     <chr> <int>
## 1 Android  4901
## 2  iPhone  3852

tweet_words_count <- left_join(tweet_words_count, total_words)

## Joining, by = "source"

tweet_words_count

## # A tibble: 3,235 × 4
##     source                   word     n total
##      <chr>                  <chr> <int> <int>
## 1   iPhone             #trump2016   171  3852
## 2  Android                hillary   124  4901
## 3   iPhone #makeamericagreatagain    95  3852
## 4  Android                crooked    93  4901
## 5  Android                clinton    66  4901
## 6  Android                 people    64  4901
## 7   iPhone                hillary    52  3852
## 8  Android                   cruz    50  4901
## 9  Android                    bad    43  4901
## 10  iPhone                america    43  3852
## # ... with 3,225 more rows

tweet_words_count <- tweet_words_count %>%
  bind_tf_idf(word, source, n)
tweet_words_count

## # A tibble: 3,235 × 7
##     source                   word     n total         tf       idf
##      <chr>                  <chr> <int> <int>      <dbl>     <dbl>
## 1   iPhone             #trump2016   171  3852 0.04439252 0.0000000
## 2  Android                hillary   124  4901 0.02530096 0.0000000
## 3   iPhone #makeamericagreatagain    95  3852 0.02466251 0.6931472
## 4  Android                crooked    93  4901 0.01897572 0.0000000
## 5  Android                clinton    66  4901 0.01346664 0.0000000
## 6  Android                 people    64  4901 0.01305856 0.0000000
## 7   iPhone                hillary    52  3852 0.01349948 0.0000000
## 8  Android                   cruz    50  4901 0.01020200 0.0000000
## 9  Android                    bad    43  4901 0.00877372 0.0000000
## 10  iPhone                america    43  3852 0.01116303 0.0000000
## # ... with 3,225 more rows, and 1 more variables: tf_idf <dbl>

Which terms have a high tf-idf?

tweet_words_count %>%
  select(-total) %>%
  arrange(desc(tf_idf))

## # A tibble: 3,235 × 6
##     source                   word     n          tf       idf      tf_idf
##      <chr>                  <chr> <int>       <dbl>     <dbl>       <dbl>
## 1   iPhone #makeamericagreatagain    95 0.024662513 0.6931472 0.017094751
## 2   iPhone                   join    42 0.010903427 0.6931472 0.007557680
## 3   iPhone          #americafirst    27 0.007009346 0.6931472 0.004858508
## 4   iPhone             #votetrump    23 0.005970924 0.6931472 0.004138729
## 5   iPhone             #imwithyou    20 0.005192108 0.6931472 0.003598895
## 6   iPhone        #crookedhillary    17 0.004413292 0.6931472 0.003059061
## 7   iPhone          #trumppence16    14 0.003634476 0.6931472 0.002519227
## 8   iPhone                    7pm    11 0.002855659 0.6931472 0.001979392
## 9   iPhone                  video    11 0.002855659 0.6931472 0.001979392
## 10 Android                  badly    13 0.002652520 0.6931472 0.001838587
## # ... with 3,225 more rows

tweet_important <- tweet_words_count %>%
  arrange(desc(tf_idf)) %>%
  mutate(word = factor(word, levels = rev(unique(word))))

tweet_important %>%
  group_by(source) %>%
  slice(1:15) %>%
  ggplot(aes(word, tf_idf, fill = source)) +
  geom_bar(alpha = 0.8, stat = "identity") +
  labs(title = "Highest tf-idf words in @realDonaldTrump",
       subtitle = "Top 15 for Android and iPhone",
       x = NULL, y = "tf-idf") +
  coord_flip()

• Most hashtags come from the iPhone. Indeed, almost no tweets from Trump’s Android contained hashtags, with some rare exceptions like this one. (This is true only because we filtered out the quoted “retweets”, as Trump does sometimes quote tweets like this that contain hashtags).

• Words like “join”, and times like “7pm”, also came only from the iPhone. The iPhone is clearly responsible for event announcements like this one (“Join me in Houston, Texas tomorrow night at 7pm!”)

• A lot of “emotionally charged” words, like “badly” and “dumb”, were overwhelmingly more common on Android. This supports the original hypothesis that this is the “angrier” or more hyperbolic account.

Sentiment analysis

Since we’ve observed a difference in sentiment between the Android and iPhone tweets, let’s try quantifying it. We’ll work with the NRC Word-Emotion Association lexicon, available from the tidytext package, which associates words with 10 sentiments: positive, negative, anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.

nrc <- sentiments %>%
  filter(lexicon == "nrc") %>%
  select(word, sentiment)
nrc

## # A tibble: 13,901 × 2
##           word sentiment
##          <chr>     <chr>
## 1       abacus     trust
## 2      abandon      fear
## 3      abandon  negative
## 4      abandon   sadness
## 5    abandoned     anger
## 6    abandoned      fear
## 7    abandoned  negative
## 8    abandoned   sadness
## 9  abandonment     anger
## 10 abandonment      fear
## # ... with 13,891 more rows

To measure the sentiment of the Android and iPhone tweets, we can count the number of words in each category:

sources <- tweet_words %>%
  group_by(source) %>%
  mutate(total_words = n()) %>%
  ungroup() %>%
  distinct(id, source, total_words)
sources

## # A tibble: 1,172 × 3
##                    id  source total_words
##                 <chr>   <chr>       <int>
## 1  676494179216805888  iPhone        3852
## 2  676509769562251264  iPhone        3852
## 3  680496083072593920 Android        4901
## 4  680503951440121856 Android        4901
## 5  680505672476262400 Android        4901
## 6  680734915718176768 Android        4901
## 7  682764544402440192  iPhone        3852
## 8  682792967736848385  iPhone        3852
## 9  682805320217980929  iPhone        3852
## 10 685490467329425408 Android        4901
## # ... with 1,162 more rows

by_source_sentiment <- tweet_words %>%
  inner_join(nrc, by = "word") %>%
  count(sentiment, id) %>%
  ungroup() %>%
  complete(sentiment, id, fill = list(n = 0)) %>%
  inner_join(sources) %>%
  group_by(source, sentiment, total_words) %>%
  summarize(words = sum(n)) %>%
  ungroup()

## Joining, by = "id"

head(by_source_sentiment)

## # A tibble: 6 × 4
##    source    sentiment total_words words
##     <chr>        <chr>       <int> <dbl>
## 1 Android        anger        4901   321
## 2 Android anticipation        4901   256
## 3 Android      disgust        4901   207
## 4 Android         fear        4901   268
## 5 Android          joy        4901   199
## 6 Android     negative        4901   560

(For example, we see that 321 of the 4901 words in the Android tweets were associated with “anger”). We then want to measure how much more likely the Android account is to use an emotionally-charged term relative to the iPhone account. Since this is count data, we can use a Poisson test to measure the difference:

# function to calculate the poisson.test for a given sentiment
poisson_test <- function(df){
  poisson.test(df$words, df$total_words)
}

# use the nest() and map() functions to apply poisson_test to each sentiment and 
# extract results using broom::tidy()
sentiment_differences <- by_source_sentiment %>%
  group_by(sentiment) %>%
  nest() %>%
  mutate(poisson = map(data, poisson_test),
         poisson_tidy = map(poisson, tidy)) %>%
  unnest(poisson_tidy, .drop = TRUE)
sentiment_differences

## # A tibble: 10 × 9
##       sentiment estimate statistic      p.value parameter  conf.low
##           <chr>    <dbl>     <dbl>        <dbl>     <dbl>     <dbl>
## 1         anger 1.492863       321 2.193242e-05  274.3619 1.2353162
## 2  anticipation 1.169804       256 1.191668e-01  239.6467 0.9604950
## 3       disgust 1.677259       207 1.777434e-05  170.2164 1.3116238
## 4          fear 1.560280       268 1.886129e-05  225.6487 1.2640494
## 5           joy 1.002605       199 1.000000e+00  198.7724 0.8089357
## 6      negative 1.692841       560 7.094486e-13  459.1363 1.4586926
## 7      positive 1.058760       555 3.820571e-01  541.4449 0.9303732
## 8       sadness 1.620044       303 1.150493e-06  251.9650 1.3260252
## 9      surprise 1.167925       159 2.174483e-01  148.9393 0.9083517
## 10        trust 1.128482       369 1.471929e-01  350.5114 0.9597478
## # ... with 3 more variables: conf.high <dbl>, method <fctr>,
## #   alternative <fctr>

And we can visualize it with a 95% confidence interval:

sentiment_differences %>%
  ungroup() %>%
  mutate(sentiment = reorder(sentiment, estimate)) %>%
  mutate_each(funs(. - 1), estimate, conf.low, conf.high) %>%
  ggplot(aes(estimate, sentiment)) +
  geom_point() +
  geom_errorbarh(aes(xmin = conf.low, xmax = conf.high)) +
  scale_x_continuous(labels = percent_format()) +
  labs(x = "% increase in Android relative to iPhone",
       y = "Sentiment")

Thus, Trump’s Android account uses about 40-80% more words related to disgust, sadness, fear, anger, and other “negative” sentiments than the iPhone account does. (The positive emotions weren’t different to a statistically significant extent).

We’re especially interested in which words drove this different in sentiment. Let’s consider the words with the largest changes within each category:

tweet_important %>%
  inner_join(nrc, by = "word") %>%
  filter(!sentiment %in% c("positive", "negative")) %>%
  mutate(sentiment = reorder(sentiment, -tf_idf),
         word = reorder(word, -tf_idf)) %>%
  group_by(sentiment) %>%
  top_n(10, tf_idf) %>%
  ungroup() %>%
  ggplot(aes(word, tf_idf, fill = source)) +
  facet_wrap(~ sentiment, scales = "free", nrow = 4) +
  geom_bar(stat = "identity") +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  labs(x = "",
       y = "tf-idf") +
  scale_fill_manual(name = "", labels = c("Android", "iPhone"),
                    values = c("red", "lightblue"))

## Warning in inner_join_impl(x, y, by$x, by$y, suffix$x, suffix$y): joining
## factor and character vector, coercing into character vector

This confirms that lots of words annotated as negative sentiments are more common in Trump’s Android tweets than the campaign’s iPhone tweets. It’s no wonder Trump’s staff took away his tweeting privileges for the remainder of the campaign.