A Very Good Example of Very Bad Transportation Performance Evaluation

Some things are so very bad that they are good, for the sake of amusement and as examples to avoid. Of course, everybody makes mistakes, but some massive disasters involve so many errors by so many people that onlookers can also wonder, "What were they thinking?!"

For example, war historians still discuss the 1415 Battle of Agincourt, in which French King John made just about every mistake possible when trying to repel the invasion by King Henry V (the King and his titled knights insisted on leading the attack rather than allowing the more effective but lower class archers a chance to achieve glory). Aviation enthusiasts are fascinated by the 1919 Christmas Bullet airplane, designed by the totally unqualified Dr. William Christmas with a flexible wing, and promoted by him as the "safest, easiest controlled plane in the world" (both prototypes crashed during maiden flights and killed their test pilots). Movie buffs love to watch Battlefield Earth, based on the novel by science fiction writer and Scientology founder Ron Hubbard, which depicts a rebellion by humans against the brutal alien Psychlos in the year 3000, considered one of the worst movies ever made (described by one critic as, "a flat-out mess, by golly, with massive narrative sinkholes, leading to moments of outstanding disbelief in the muddled writing and shockingly chaotic mise en scène that's accompanied by ear-pummeling sound and bombastic music"). 

A good current example is the recent study, Transportation Performance of the Canadian Provinces by Hartgen, Chadwick and Fields, published by the Fraser Institute, a think tank located in Vancouver, Canada, critiqued in my new paper, A Good Example of Bad Transportation Performance Evaluation.

Hartgen, Chadwick and Fields use a unique set of 23 indicators to evaluate the Canadian transport system. A few of these indicators are appropriate, but most are ambiguous and biased, and some are downright silly. For example, its congestion indicator (annual hours of delay per capita) is reasonable, but its roadway indicator (vehicle kilometers of travel per two-lane kilometer of road) is ambiguous (a higher value could indicate cost efficiency or inadequate roadway supply and congestion) and inherently favors more urbanized provinces over more rural provinces. The highway cost efficiency indicator (provincial expenditures per kilometer of major road) favors provinces with inexpensive, low-volume roads, although the results would be reversed if based on expenditures per vehicle-kilometer, which recognizes that the economic value of roads results from their use. Aviation indicators (passengers and tonnes of cargo per flight) favor provinces with major airports.

Two indicators are particularly illogical: Travel time to Ottawa and Travel time to US border. They assume:

·         Canada has only two economically important destinations: Ottawa and the U.S. In fact, Canada has many regions each with multiple economic activity centers. Economic success requires access to diverse destinations within and between regions.

·         Ottawa is Canada's economic center. Although a major administrative center, it has only modest economic importance. If a single city were to be selected, it should be Toronto.

·         All travelers drive to Ottawa, although most (particularly business and wealthy tourists who have the most economic importance) generally fly.

What were they thinking? These indicators violate the basic principles of good performance evaluation. Consider the transport policies and conditions these indicators imply are desirable:

·         Km of travel per two-lane km of road. Vehicle travel should be encouraged and concentrated on minimal road kilometers. Provinces with busier roads have better transport systems than provinces with lower-volume roads.

·         Annual hours of congestion delay per capita and average commute times. Rural areas have better, more productive transport systems than dense, urban areas with heavy commute volumes, although cities actually tend to have higher per capita economic productivity.

·         Provincial expenditures per km of major road. Provinces should minimize highway expenditures even if that reduces service quality or off-loads costs to other levels of governments. Provinces with better highways and more challenging terrain are inferior.

·         Travel time to Ottawa and Travel time to US border. Central and southern provinces have better transport systems than western, eastern and northern provinces.·         Total employment per truck border crossing. Provinces should minimize border crossings.·         Government costs per ferry trip. Provinces should only subsidize short, high volume routes.·         Port operator expenditures per tonne handled. A province is more productive if it transports easily handled products (for example, coal rather than automobiles). 

The authors acknowledge that their index is arbitrary and incomplete and offer the following defence:

Of course, some provincial road managers would immediately point out that the provinces are not comparable. For example, British Columbia has road traffic, congestion, cost-of-business, weather, terrain, and climatic conditions that are vastly different from Saskatchewan. That might be, but other provinces might counter that they do not have the fiscal resources that the more populous provinces have and, therefore, are constrained as to actions they can take. Following this logic, one would be quickly led to the absurd conclusion that no comparisons between provinces are ever possible. Our view is that, even with different circumstances, each province is responsible for the road system's upkeep and expansion using the resources it has. And each province is dependent on others for accessibility, since modal systems connect provinces. Therefore, a measure that shows how much each province has to work with, relative to system and size, is a useful comparative measure of resources that should translate into results.

This is surely among the most inscrutable, confusing text ever written by a team lead by a university professor. It seems to suggest that only a few, biased "provincial road managers" would question their analysis and that, since good management requires performance evaluation, any indicators can be used regardless of whether or not they are logical and fair. This is comparable to a quack physician prescribing inappropriate tests in order to justify ineffective medical treatments.  

The true, insidious intent of this study is to justify highway expansion and deflect efforts to create more multi-modal transport systems. This is indicated by the fact that highway indicators are given 87% of the index total weight, based on trip modes share, so highway improvements are considered much more important than improvements to other modes, although for economic analysis impacts should be weighted based on value, which would give more importance to air travel, freight transport, and transit improvements.

The authors reveal their ideological bias in the following quote:

A particularly thorny area of evaluation is the impact of transportation investment on induced travel and development. This is a subject of considerable research in recent years, but the findings remain elusive. Even the definitions of such terms as "induced travel" are open to a wide variety of interpretation and measurement problems. While many researchers believe that these impacts exist, it is extremely difficult to demonstrate their presence even for aggregate investments and certainly for a specific project. For this first round, we take the conservative position that these impacts are not sufficiently quantifiable for measurement at the provincial level. 

This statement is inaccurate and indicates the authors are either ignorant or intentionally misrepresenting current knowledge regarding induced travel. During the last decade, many studies published in the professional literature have demonstrated the existence of induced travel and described methodologies for quantifying it. Ignoring induced travel exaggerates highway expansion benefits, leading to waste, which is the opposite of conservative.

Good performance evaluation is essential for effective planning and management. Which indicators are used and how they are defined can significantly affect analysis results, and therefore planning decisions. The analysis by Hartgen, Chadwick and Fields is useless for planning and management. Transportation professionals sincerely interested in performance evaluation can find plenty of good information resources for developing useful and appropriate indicators. Hartgen, Chadwick and Fields provide a useful service by demonstrating what not to do.

Information Resources

CalTrans (2008), Transportation System Performance Measures (TSPM), California Department of Transportation (www.dot.ca.gov/hq/tsip/index.php).

Todd Litman (2007), "Developing Indicators For Comprehensive And Sustainable Transport Planning," Transportation Research Record 2017, TRB (www.trb.org), 2007, pp. 10-15. 

Todd Litman (2008), "A Good Example of Bad Transportation Performance Evaluation: Critique of, 'Transportation Performance of the Canadian Provinces'," Victoria Transport Policy Institute (www.vtpi.org/per_ind.pdf)

OTM (2008), Transportation Performance Measures, Office of Transportation Management, Federal Highway Administration (www.fhwa.dot.gov/planning/toolbox).   

Performance Measurement Exchange (http://knowledge.fhwa.dot.gov/cops/pm.nsf/home), is a website supported by FHWA and TRB to promote better transportation decision-making.

STI (2008), Sustainable Transportation Indicators: A Recommended Program To Define A Standard Set of Indicators For Sustainable Transportation Planning, Sustainable Transportation Indicators Subcommittee (ADD40 [1]), TRB; at www.vtpi.org/sustain/sti.pdf.  

TRB (2008), Performance Measurement Practice (www.trb-performancemeasurement.org), Performance Measurement Committee (ABC30), Transportation Research Board.

WSDOT (2008), Performance Measurement Library, Washington State Department of Transportation; at www.wsdot.wa.gov/Accountability/Publications/Library.htm.