Reference no: EM133381133

Question: Tf-idf is a powerful method for surfacing the words that are most distinctive for each document. The example in the video above shows that it can be useful for finding the major plot points or themes that are unique to each book in a set of books. Silge and Robinson show that it can be useful for distinguishing among works by one author, and for differentiating physics textbooks. And Stephen Ramsay shows, it can be useful for differentiating, for instance, the words used by different characters in a novel.

Once you're done watching the video and reading the assigned works check out the this week's project on Posit Cloud. There, I've shared with you the code for downloading 5 books from Project Gutenberg and analyzing them using tf-idf. Play around with it by choosing some other books or making different decisions, and see how it changes the end results.

Finally, come back here for this week's discussion on tf-idf, consider how you might find the technique useful. Would you, for instance, differentiate most common words used by Democrats and Republicans in a political debate? Would you try to understand which words are used more by male authors and female authors? Would you want to understand some other set of texts entirely? Consider how you can imagine finding the technique useful or beneficial, and share with classmates the kinds of things you'd like to do. (You're also welcome to ask any questions you might have, since your classmates might be wondering the same thing.)