The Democratization of Big Data

Already a major technology trend, 2012 promises to be a watershed for "big data." A shorthand term for the proliferation of large datasets, big data also refers to the expansion of analytic techniques for teasing meaning from the vast archives of information produced by the digital world. The New York Times' Steve Lohr declared we have entered the "age of big data" in a recent article that compared it with another revolutionary research tool -- the microscope.

4 minute read

February 27, 2012, 7:40 AM PST

By Robert Goodspeed @rgoodspeed


Already a major technology trend, 2012 promises to be a watershed for "big data." A shorthand term for the proliferation of large datasets, big data also refers to the expansion of analytic techniques for teasing meaning from the vast archives of information produced by the digital world. The New York Times' Steve Lohr declared we have entered the "age of big data" in a recent article that compared it with another revolutionary research tool -- the microscope.

As I observed last year, big data is beginning to filter into the urban planning world. Here are a few examples of the intersection of cities and big data (two from this PlaceMatters blog post):

What do all these exciting examples of big data have in common? If you have modest technical skills and work for a local government or community-based organization, you probably do not have access to the data and skills necessary to replicate the projects.

Inequalities of data access are not new in planning. Sixteen years ago David Sawicki and William Craig argued in a Journal of the American Planning Association article titled "The Democratization of Data" that the most important ingredients to expanded access to the first generation of data wasn't advances of computing power or analysis skills, but the rise of data intermediaries that worked with community groups in low-income communities to ensure they had access to quality data and skills. Whether nonprofits, local governments, or university-led projects, these intermediaries helped equalize access to data in the public sphere.

However, as the size of datasets has increased, so have the skills necessary to manage and analyze the data. No longer is mastery of a few desktop applications sufficient for analysis, since wrangling today's large datasets requires database servers and analysts skilled at statistical and algorithmic data mining techniques. Although government datasets may have been the original big data, many of the new datasets are provided by corporations, introducing a morass of ethical and practical challenges. Frequently collected at the individual level, negotiating access requires navigating privacy and security concerns. Even when companies provide public access, extracting and using their data requires programming skills to tap application programming interfaces (APIs) or manipulate unusual data formats.

Finally, lurking beneath the big data hype are problematic unstated assumptions about the nature of truth. In the 1980s, the so-called quantitative-qualitative debate raged across several social science fields among scholars arguing the merits of various research methods. Some researchers stressed the need to collect empirical evidence and rely solely on quantitative analysis for research. Others argued social science required qualitative analysis such as interviews and observation to understand society. Although the debate is different today, important differences of opinion remain.

We should be cautious about claims that big data will necessarily answer important or relevant research or policy questions. Are cell phone traces sufficient to intuit travel behavior, or are surveys or interviews required to understand how people make choices? Can postings to social networking websites provide as much insight as a windshield survey, or an in-depth interview of community residents? The big data hype also runs counter to important developments in social science that stress the role of experiments and counterfactual reasoning, instead of relying on ever-more-complicated statistical models to explain the world.

What are some practical steps that big data could take to expand access by community-based organizations? A start might be to provide data in formats and sizes (perhaps through summary versions) that they can be analyzed in common software packages, such as ArcMap, Excel, and Google Earth. Data providers should provide documentation about the source, variables, and assumptions used to collect and process the data. Existing data intermediaries should explore the new datasets, and strategically expand their expertise where it seems appropriate. Although the proliferation of broadband and Internet-connected smartphones has reduced the prominence of the "digital divide," we must take steps now to reduce the emergence of a new "data divide" between sophisticated analysts and communities seeking to plan for their futures.


Robert Goodspeed

Robert Goodspeed is an Assistant Professor of Urban Planning at the University of Michigan. He holds a PhD from the MIT Department of Urban Studies and Planning and previously worked for the Boston Metropolitan Area Planning Council. See his academic website for more on his teaching and research.

portrait of professional woman

I love the variety of courses, many practical, and all richly illustrated. They have inspired many ideas that I've applied in practice, and in my own teaching. Mary G., Urban Planner

I love the variety of courses, many practical, and all richly illustrated. They have inspired many ideas that I've applied in practice, and in my own teaching.

Mary G., Urban Planner

Cover CM Credits, Earn Certificates, Push Your Career Forward

Logo for Planetizen Federal Action Tracker with black and white image of U.S. Capitol with water ripple overlay.

Planetizen Federal Action Tracker

A weekly monitor of how Trump’s orders and actions are impacting planners and planning in America.

July 2, 2025 - Diana Ionescu

Aerial view of town of Wailuku in Maui, Hawaii with mountains in background against cloudy sunset sky.

Maui's Vacation Rental Debate Turns Ugly

Verbal attacks, misinformation campaigns and fistfights plague a high-stakes debate to convert thousands of vacation rentals into long-term housing.

July 1, 2025 - Honolulu Civil Beat

White and purple sign for Slow Street in San Francisco, California with people crossing crosswalk.

San Francisco Suspends Traffic Calming Amidst Record Deaths

Citing “a challenging fiscal landscape,” the city will cease the program on the heels of 42 traffic deaths, including 24 pedestrians.

July 1, 2025 - KQED

Blue and silver Amtrak train at small station.

Amtrak Rolls Out New Orleans to Alabama “Mardi Gras” Train

The new service will operate morning and evening departures between Mobile and New Orleans.

July 3 - New Orleans City Business

Large spinning swing ride at Chicago's Navy Pier.

The Subversive Car-Free Guide to Trump's Great American Road Trip

Car-free ways to access Chicagoland’s best tourist attractions.

July 3 - Streetsblog Chicago

Aerial view of downtown San Antonio, Texas at night with rotating Tower of the Americas in foreground.

San Antonio and Austin are Fusing Into one Massive Megaregion

The region spanning the two central Texas cities is growing fast, posing challenges for local infrastructure and water supplies.

July 3 - Governing