Google never seems to stop surprising us with a constant stream of neat little projects that mine large datasets in interesting and unexpected ways. My topic today is yet another of these: Google’s Ngram Viewer.
The dataset the Ngram Viewer explores is an index of the use of one or more words along with its date of use in every book that Google has scanned (to the continuing horror of the publishing industry) through the company’s Google Books programs.
If you haven’t read the history of the Google Books project it’s worth checking out as it outlines the kind of thinking that Google applies to so many programs where acquiring, processing and managing truly enormous amounts of data are involved. In the case of Google Books the results are amazing with world class libraries such as Oxford University’s Bodleian Library and The Library of Congress along with all of the major publishers getting involved.
Anyway, with all of these books indexed as time series Google saw an interesting opportunity to explore how the use of words has changed in publishing over time.
The Ngram Viewer allows you to specify multiple unigrams (single word terms) and or bigrams (terms with two or more words) and then the viewer will plot the frequency of their use over whatever time period you please. You can choose whether to only include books in English or go for Latin, French, or any of the other language choices available.
A great example of how the use of terms has evolved is the one that Google shows in the project’s “about” page — this example compares the use of three ngrams “nursery school”, “kindergarten”, and “child care” over the period 1950 to 2000.
As the “about” page explains: “What the y-axis shows is [for] all the bigrams contained in our sample of books written in English and published in the United States, what percentage of them are “nursery school” or “child care”? Of all the unigrams, what percentage of them are “kindergarten”? Here, you can see that use of the phrase “child care” started to rise in the late 1960s, overtaking “nursery school” around 1970 and then “kindergarten” around 1973. It peaked shortly after 1990 and has been falling steadily since. (Interestingly, the results are noticeably different when the corpus is switched to British English.)
If you feel so moved you can download the Ngram Viewer’s datasets and mangle them with your own tools.
The dataset contents are simple tab-separated variable files that consists of an ngram followed by the year of its use, overall number of uses, number of pages it was used on (which is always less than or equal to the overall number of uses), and the number of books the ngram appeared in.
Be warned that the Ngram Viewer results can be misleading because of errors introduced in the scanning of the books or because the use of some ngrams has changed over time.
The Ngram Viewer is a fascinating experiment and well worth playing around with. That said, I’d set a time limit because I got sucked into a 90 minute session despite other work I needed to get on with … yes, it’s that addictive.
