The Democratization of Big Data

<p> Already a major technology trend, 2012 promises to be a watershed for &quot;big data.&quot; A shorthand term for the proliferation of large datasets, big data also refers to the expansion of analytic techniques for teasing meaning from the vast archives of information produced by the digital world. The New York Times&#39; Steve Lohr declared we have entered the &quot;age of big data&quot; in a <a href="">recent article</a> that compared it with another revolutionary research tool -- the microscope. </p>

February 27, 2012, 7:40 AM PST

By Robert Goodspeed @rgoodspeed

Already a major technology trend, 2012 promises to be a watershed for "big data." A shorthand term for the proliferation of large datasets, big data also refers to the expansion of analytic techniques for teasing meaning from the vast archives of information produced by the digital world. The New York Times' Steve Lohr declared we have entered the "age of big data" in a recent article that compared it with another revolutionary research tool -- the microscope.

As I observed last year, big data is beginning to filter into the urban planning world. Here are a few examples of the intersection of cities and big data (two from this PlaceMatters blog post):

What do all these exciting examples of big data have in common? If you have modest technical skills and work for a local government or community-based organization, you probably do not have access to the data and skills necessary to replicate the projects.

Inequalities of data access are not new in planning. Sixteen years ago David Sawicki and William Craig argued in a Journal of the American Planning Association article titled "The Democratization of Data" that the most important ingredients to expanded access to the first generation of data wasn't advances of computing power or analysis skills, but the rise of data intermediaries that worked with community groups in low-income communities to ensure they had access to quality data and skills. Whether nonprofits, local governments, or university-led projects, these intermediaries helped equalize access to data in the public sphere.

However, as the size of datasets has increased, so have the skills necessary to manage and analyze the data. No longer is mastery of a few desktop applications sufficient for analysis, since wrangling today's large datasets requires database servers and analysts skilled at statistical and algorithmic data mining techniques. Although government datasets may have been the original big data, many of the new datasets are provided by corporations, introducing a morass of ethical and practical challenges. Frequently collected at the individual level, negotiating access requires navigating privacy and security concerns. Even when companies provide public access, extracting and using their data requires programming skills to tap application programming interfaces (APIs) or manipulate unusual data formats.

Finally, lurking beneath the big data hype are problematic unstated assumptions about the nature of truth. In the 1980s, the so-called quantitative-qualitative debate raged across several social science fields among scholars arguing the merits of various research methods. Some researchers stressed the need to collect empirical evidence and rely solely on quantitative analysis for research. Others argued social science required qualitative analysis such as interviews and observation to understand society. Although the debate is different today, important differences of opinion remain.

We should be cautious about claims that big data will necessarily answer important or relevant research or policy questions. Are cell phone traces sufficient to intuit travel behavior, or are surveys or interviews required to understand how people make choices? Can postings to social networking websites provide as much insight as a windshield survey, or an in-depth interview of community residents? The big data hype also runs counter to important developments in social science that stress the role of experiments and counterfactual reasoning, instead of relying on ever-more-complicated statistical models to explain the world.

What are some practical steps that big data could take to expand access by community-based organizations? A start might be to provide data in formats and sizes (perhaps through summary versions) that they can be analyzed in common software packages, such as ArcMap, Excel, and Google Earth. Data providers should provide documentation about the source, variables, and assumptions used to collect and process the data. Existing data intermediaries should explore the new datasets, and strategically expand their expertise where it seems appropriate. Although the proliferation of broadband and Internet-connected smartphones has reduced the prominence of the "digital divide," we must take steps now to reduce the emergence of a new "data divide" between sophisticated analysts and communities seeking to plan for their futures.

Robert Goodspeed

Robert Goodspeed is an Assistant Professor of Urban Planning at the University of Michigan. He holds a PhD from the MIT Department of Urban Studies and Planning and previously worked for the Boston Metropolitan Area Planning Council.

A image of the World's Columbian Exposition overlayed with a picture of Keanu Reeves in the rain from the movie Point Break.

Keanu Reeves Set to Play Daniel Burnham in ‘The Devil in the White City’

Planning is going to get a new level of star power as a limited series adaptation of The Devil in the White City gets ready for television screens in 2024.

August 8, 2022 - Reel Chicago


Opinion: Aging Population, Declining Fertility Requires Long-Term Investments

Faced with the dire consequences of a one-two punch of aging populations and declining birthrates, one writer has suggestions for how policy can help ensure a better future.

August 9, 2022 - Financial Times

View from middle of street in downtown Telluride, Colorado with mountains in background

Marrying Urban Identity and Economic Prosperity

A new book posits that truly successful communities have a strong economic base and a firmly rooted sense of place.

August 5, 2022 - Governing

Aerial view of downtown San Antonio with Tower Life Building in foreground

San Antonio Office Tower To Become Residential

With the building more than half vacant, the new owners of the Tower Life Building plan to convert the historic tower into residences that could include affordable housing.

54 minutes ago - San Antonio Report


Freeway Removal Movement Slowly Gains Steam

Although the concept has recently received more national attention thanks in part to the federal Reconnecting Communities Act, cities have shown reluctance to support highway removal projects.

1 hour ago - Governing

Accessible elevator sign with arrow on a New York City subway platform

MTA Uses Density Bonuses to Improve Accessibility

Under a new zoning law, New York City developers can receive density bonuses for building elevators and other accessibility upgrades for the city’s subway system.

2 hours ago - Queens Eagle

Urban Design for Planners 1: Software Tools

This six-course series explores essential urban design concepts using open source software and equips planners with the tools they need to participate fully in the urban design process.

Hand Drawing Master Plans

This course aims to provide an introduction into Urban Design Sketching focused on how to hand draw master plans using a mix of colored markers.