Could partnership between IBM and Twitter create new crime prediction potential for smart cities?

Could IBM and Twitter improve crime prediction?












The winners of the 2015-16 IBM Smarter Cities Challenge were announced by Jen Crozier at the IBM Edge event in Las Vegas, Nevada last month. The Smarter Cities Challenge is the largest philanthropic program IBM currently runs, through which the firm claims to have donated services worth US$50 million to over 100 cities in the past five years. In addition to announcing the winners, Crozier also announced that each of the sixteen municipalities will receive expert advice on using Watson analytics as part of the prize package. A special bonus offering for three of the winners was also announced: Detroit, Memphis and Melbourne will be gifted with current and historical data sets from Twitter. These data sets will be analyzed with the help of Watson analytics and a Twitter analyst that will be on loan to one of the cities for three weeks. The ‘landmark partnership’ between IBM and Twitter was announced at the end of 2014 aims to ‘transform how businesses and institutions understand their customers, markets and trends’.

According to IBM marketing materials, Watson offers predictive analytics for a range of contexts including helping help retailers predict the behaviour of customers; predicting insurance claims in relation to sever weather; and predicting maintenance requirements for urban infrastructure. The spatiotemporal metadata associated with tweets has shown predictive value for a range of situations including elections, disease outbreaks, political revolutions and urban security threats (Gerber 2014).

Twitter data was famously used for urban security purposes during the G20 protests in Toronto in 2010, which involved surveillance of users and their social networks (Bennett, Haggerty, Lyon and Steeves 2014). Data posted on Twitter was used by security forces to prevent select protestors from even approaching security zones and used as evidence in the prosecution of suspicious and risky individuals (Werbin 2011). Twitter data was front and centre of the high profile case of Byron Sonne, who claimed to be ‘testing’ the security apparatus with his online Twitter activity; it took two years for him to be cleared of criminal charges linked to the incident (CBC 2012).

Beyond national security threats and mega event attacks, researchers are demonstrating the potential for Twitter data to predict criminal activity on a much smaller scale. Traditionally, predicting urban crime levels has involved analyzing historical crime data for a particular geography in combination with demographic information for that geography and extrapolating the results into the future. The Kernel Density Estimation model is the preferred approach for many researchers constructing crime hot spots (Andresen 2015). According to Hart and Zandbergen (2014) this may be attributed to the availability of this function in GIS applications, perceptions of accuracy and ease of use. Kernel Density Estimation (KDE) uses the historic crime data tied to a particular geography to predict the future probability of criminal activity in that space. Researchers have problematized this approach by arguing hot spots do not accurately portray the geography of criminality, which can be improved by using spatiotemporal data (Wang, Gerber and Brown 2012; Bogomolov et al. 2014; Malleson and Andresen 2015).

With the adoption of smart phones and social media applications, oceans of spatiotemporal data have become available to researchers. One of the earlier and more often cited articles on using data from Twitter for the purpose of predicting crime is the work of Wang, Gerber and Brown (2012). Their experiment involved semantic analysis of tweets collected from news agencies in combination with data from the local law enforcement agency in Charlottesville, Virginia. Although the researchers were only able to demonstrate predictive values for a limited range of crimes (‘hit and runs’ and ‘break and enters’), the research was promising (Wang, Gerber and Brown 2012). Since the tweets they used were posted by journalists, the analysis did not benefit from spatiotemporal meta data.

However, two years later, Matthew Gerber (2014) published his own research (funded by the United States Army) that combined historical crime data in the City of Chicago with semantic analysis of tweets and their spatiotemporal meta data. He was able to improve predictions for 19 out of 25 categories of crime (Gerber 2014). In the same year a team of researchers demonstrated mobile data could be used for crime prediction and could increase the granularity of hot spots using London as a case study (Bogomolov et al. 2014). They produced hot spot maps that looked less like the smooth contours of a weather forecast and more like a Rubik’s cube. According to Malleson and Andresen (2015), using spatiotemporal data leverages the characteristics of ambient populations to improve the predictive values for mobile crimes. They claim that by combining historical crime data with spatiotemporal data from Twitter predictions of criminal activity in the City of Leeds were improved (Malleson and Andresen 2015).

The use of data from social networking sites like Twitter to predict criminal hot spots is clearly not a magic bullet for local law enforcement agencies. As Malleson and Andresen (2015) note this data is not reflective of the entire ambient population in a given spatiotemporal cluster. However, traditional hot spot analysis relies on historical data that is not reflective of an actual spatiotemporal cluster either. There are other limitations, which include the challenge of decoding the meaning of each tweet (Gerber 2014). Despite these limitations, as local law enforcement agencies struggle to allocate limited resources the potential for creating more precise hot spots through data analytics will likely be of interest. The city chosen for the bonus consultation from Twitter may have the opportunity to explore the possibilities that the research examined above points to.


Andresen, Martin A. 2015. “Identifying Changes in Spatial Patterns from Police Interventions: The Importance of Multiple Methods of Analysis.” 16(2):148–60.

Bennett, Colin J., Kevin Haggerty, David Lyon, and Valerie Steeves. 2014. Transparent Lives: Surveillance in Canada. Edmonton, Canada: Athabasca University.

Bogomolov, Andrey et al. 2014. “Once Upon a Crime: Towards Crime Prediction from Demographics and Mobile Data.” arXiv:1409.2983 [physics]. Retrieved June 17, 2015 (

CBC. 2012. “G20 Protestor Byron Sonne Cleared of All Charges.” CBC News. Retrieved June 18, 2015 (

Gerber, Matthew S. 2014. “Predicting Crime Using Twitter and Kernel Density Estimation.” Decision Support Systems 61:115–25.

Hart, Timothy and Paul Zandbergen. 2014. “Kernel Density Estimation and Hotspot Mapping.” Policing: An International Journal of Police Strategies & Management 37(2):305–23.

Malleson, Nick and Martin A. Andresen. 2015. “Spatio-Temporal Crime Hotspots and the Ambient Population.” Crime Science 4(10):1–8.

Wang, Xiaofeng, Matthew S. Gerber, and Donald E. Brown. 2012. “Automatic Crime Prediction Using Events Extracted from Twitter Posts.” Pp. 231–38 in Social Computing, Behavioral – Cultural Modeling and Prediction, Lecture Notes in Computer Science, edited by S. J. Yang, A. M. Greenberg, and M. Endsley. Springer Berlin Heidelberg. Retrieved June 17, 2015 (

Werbin, Kenneth C. 2011. “Spookipedia: Intelligence, Social Media and Biopolitics.” Media, Culture & Society 33(8):1254–65.



Posted in Musings.

Leave a Reply

Your email address will not be published. Required fields are marked *