Originally, Johan Bollen and his students thought that they would discover a positive relationship between the direction of the Dow and Twitter sentiment. However, the connection was not sad tweets on down days and happy ones after the Dow went up. The connection, displayed 3 or 4 days later, related to a tone that was calm or anxious.
The big surprise, though, was that their original prediction was backwards. Looking at millions of tweets for emotional indicators, they discovered that the tweets came first. A slew of calm tweets meant the Dow would probably rise. Anxious tweets and it fell. Their accuracy? An 86.7% success rate.
Now, 3 years after he published his original Twitter paper, Professor Bollen has patented his “Twitter Predictor.” Sort of like a weather forecast, he believes the tone he monitors in social media can predict financial and socioeconomic events.
Dr. Bollen’s invention, I suspect, is a part of a techtonic shift toward big data…instantaneous big data. With the updated “Twitter predictor,” described in U.S. Patent No. 8,380,607, we now can assess hundreds of millions of tweets. Similarly, in real time, scientists can amass and disseminate information like consumer prices that national statisticians release monthly.
Our bottom line? The “wisdom of crowds” enables markets to determine prices that convey information about value, production cost and desirability. So too can big data convey insight that relates to economics, finance, culture and politics.
Sources and resources: Readable, the Bollen et al paper details the mood words, the duration, the design for capturing the tone of millions of tweets and people. A more popular description that includes the patented “Twitter predictor” was in an Indiana University article. However, most meaningful for me was the connection between Bollen’s work and the New Yorker’s James Surowiecki’s The Wisdom of Crowds. I could not ignore the significance of market information compared to unilateral conclusions from government and politicians.
Please note that sections of today’s comments were posted at econlife during 2010.
Here, Kalev Leetaru (Georgetown University) talks about how he has used “big data” through 100 million news articles to create a new world lens.