Scraping Social Media to Analyze Public Engagement

Planners are increasingly using social media for public engagement. How can you analyze what people are saying? NodeXL is an easy to use free tool for scraping and analyzing social media. I use this tool to report from the APA National Conference.

3 minute read

April 4, 2016, 9:00 AM PDT

By Jennifer Evans-Cowley @EvansCowley


Every year I report on Twitter activity at the American Planning Association National Conference. With thousands of planners descending on Phoenix, Arizona this weekend, the #APA16 hashtag is already starting to heat up. Throughout the conference myself, @EvansCowley, and @p18holland will be scraping everything that participants tweet about from the conference. In this blog, Patrick Holland and I share how easy it is to scrape and analyze social media.

What is scraping? Scraping is a technique for extracting information from websites. In this case, I am scraping information from Twitter, looking specifically for anyone using the #APA16 hashtag.

How can I scrape? I use an Excel plugin called NodeXL. NodeXL has a free and a paid version. Included here is part of the free version. NodeXL allows you to scrape social media and also allows for visualization of the data.

After you have downloaded NodeXL, you can get started.

Screenshot of NodeXLSelect the NodeXL tab in the excel spreadsheet. Choose import—then the social media network you would like to analyze. We are using Twitter for this conference, but options for Facebook, YouTube, and Flicker are also available. Once "From Twitter Search Network" has been selected you have the option to "Search for tweets that match this query." Attendees at this conference will be tweeting using the hashtag #APA16 so this is the query we want to match. You can also search for tweets associated with a particular user or keywords. The free version will limit you to 2,000 tweets at a time. In our case, we are scraping using NodeXL each day to ensure we don’t miss any tweets.   

NodeXL screenshotNodeXL imports the social media data onto five separate sheets in Excel. One feature I will highlight is if you select an individual tweet (known as a vertex) on the graph, you can see the interactions people had with that tweet. Pre-conference, people showed particular interest in retweeting and favoriting tweets about technology. The most popular tweet as of April 1st was "@medialabOn 4/3, @MITCities presents its CityScope urban planning/simulation tools at #apa16 mitsha.re/ZTl6H".

Once you have imported your scraped data you will find that NodeXL is quite similar to using your regular Excel spreadsheets. Using NodeXL, you will see who sent each tweet, the time of the tweet, whether the tweet was mentioned by other people, any links included in the tweets, and how connected users are to each other. You will be able to create graphs, filter data, and analyze to your hearts content. For example, in the case of @medialab's tweet on simulation tools was retweeted 31 times and favorited 59 times. We can see this was retweeted by @MeagBooth and favorited by @mysidewalkHQ

Tutorials on YouTube help anyone who would like to learn how to use the program. 

Those at the conference: be sure to use the #APA16 hashtag, and stay tuned for the results of this analysis at the end of the conference.


Jennifer Evans-Cowley

Jennifer Evans-Cowley, PhD, FAICP, is the Provost and Vice President for Academic Affairs at th eUniversity of North Texas. Dr. Evans-Cowley regularly teaches courses to prepare candidates to take the AICP exam. In 2011, Planetizen named Cowley as one of the leading thinkers in planning and technology. Her research regularly appears in planning journals, she is the author of four Planning Advisory Service Reports for the American Planning Association, and regularly blogs for Planetizen.

portrait of professional woman

I love the variety of courses, many practical, and all richly illustrated. They have inspired many ideas that I've applied in practice, and in my own teaching. Mary G., Urban Planner

I love the variety of courses, many practical, and all richly illustrated. They have inspired many ideas that I've applied in practice, and in my own teaching.

Mary G., Urban Planner

Use Code 25for25 at checkout for 25% off an annual plan!

Redlining map of Oakland and Berkeley.

Rethinking Redlining

For decades we have blamed 100-year-old maps for the patterns of spatial racial inequity that persist in American cities today. An esteemed researcher says: we’ve got it all wrong.

May 15, 2025 - Alan Mallach

Logo for Planetizen Federal Action Tracker with black and white image of U.S. Capitol with water ripple overlay.

Planetizen Federal Action Tracker

A weekly monitor of how Trump’s orders and actions are impacting planners and planning in America.

May 21, 2025 - Diana Ionescu

Interior of Place Versailles mall in Montreal, Canada.

Montreal Mall to Become 6,000 Housing Units

Place Versailles will be transformed into a mixed-use complex over the next 25 years.

May 22, 2025 - CBC

Flat modern glass office tower with "County of Santa Clara" sign.

Santa Clara County Dedicates Over $28M to Affordable Housing

The county is funding over 600 new affordable housing units via revenue from a 2016 bond measure.

May 23 - San Francisco Chronicle

Aerial view of dense urban center with lines indicating smart city concept.

Why a Failed ‘Smart City’ Is Still Relevant

A Google-backed proposal to turn an underused section of Toronto waterfront into a tech hub holds relevant lessons about privacy and data.

May 23 - Governing

Pale yellow Sears kit house with red tile roof in Sylva, North Carolina.

When Sears Pioneered Modular Housing

Kit homes sold in catalogs like Sears and Montgomery Ward made homeownership affordable for midcentury Americans.

May 23 - The Daily Yonder