Planetizen/ACSP Statement of Consultation

3 December 2007 - 12:28pm

STATEMENT OF CONSULTATION

Between Planetizen/Urban Insight, Inc. and Members of the ACSP Special Committee Regarding the Planetizen Guide To Graduate Urban Planning Programs

Issued on December 3, 2007

Read ACSP President Michael Hibbard's message to planning educators

This statement of consultation summarizes the plans and procedures for the production of second edition of the Planetizen Guide to Graduate Programs resulting from a series of discussions that took place during November 2007 between the editors of Planetizen and a special committee of the Association of Collegiate Schools of Planning (ACSP). These actions are designed to address the concerns relayed by members of the committee on behalf of the faculty and administrators of ACSP member programs. While this statement of consultation does not create legally binding obligations on the part of either Planetizen or ACSP, it represents the efforts of both parties to develop and cultivate a mutual understanding regarding the Planetizen Guide and establish a productive and cooperative working relationship that will result in a publication that will benefit all members of the planning community.

BACKGROUND

The Planetizen Guide to Graduate Urban Planning Programs is intended to be a comprehensive guide to master’s degree programs in the field of urban planning in the U.S. and Canada. The first edition was published by Planetizen in November 2006 – and features both detailed profiles of 47 urban planning master’s degree programs that responded to a survey conducted by Planetizen as well as various program rankings based on both quantitative measures and reputational data gathered from planning educators and practitioners.

Due to positive feedback from students to the first edition, Planetizen began work on an updated second edition – which is scheduled to be published in Spring 2008. In September 2007, surveys were sent to 98 institutions with master’s degree programs in planning (or equivalent) to collect updated data for the second edition.

In October 2007, several program chairs voiced a number of concerns about the survey and the Planetizen Guide with their colleagues. At a meeting of the Association of Collegiate Schools of Planning (ACSP), the gathered representatives of member programs agreed that a number of issues regarding the survey and guide should be addressed before they would be comfortable responding to the survey.

In response to the concerns raised, Planetizen as well as members of the ACSP leadership suggested that representatives from the two organizations communicate to resolve some of the issues of concern regarding the Guide. A special committee of ACSP members was formed to participate in discussions with Planetizen and help identify what steps could be taken to address these concerns.

The members of the convened ACSP committee include:

Dr. Enid Arvidson, The University of Texas at Arlington
Dr. Elise M. Bright, AICP, Texas A&M University
Dr. John D. Landis, University of Pennsylvania
Dr. Mickey Lauria, Clemson University
Dr. Bruce Stiftel, FAICP, Florida State University

In early November 2007, a series of telephone exchanges were conducted between Planetizen and the designated members of the ACSP committee to discuss the issues raised by various programs and faculty members.

ISSUES OF CONCERN

The ACSP special committee articulated the following issues of concerns regarding the Planetizen Guide to Graduate Urban Planning Programs.

1. Lack of transparency

The committee expressed its overall concern regarding the real and/or perceived lack of information regarding the production of the Planetizen Guide – and specifically the rankings section. While ACSP as an organization does not support rankings of planning programs, it is the opinion of the committee that if any rankings are to be published, they should be accompanied by a detailed explanation of the methodology used in their creation so that the results could be fairly understood and interpreted by the Guide’s consumers and the programs themselves.

Specifically, the committee requested that any future Planetizen program rankings:

include the specific criteria and/or data points used in the comparison;
cite the source of the data; and
explain the algorithm and weighing factors used in the calculation of the final scores.

Additionally, explanations about how Planetizen collects data through its own survey instruments should be provided and be accompanied by information about the survey sample – including the sample frame and design.

How Planetizen plans to respond to this concern:

Planetizen will agree to publish additional information about its rankings methodology in future editions of the Planetizen Guide to Graduate Urban Planning Programs (including the forthcoming second edition).
This methodology section will outline the factors taken into consideration when calculating the rankings, the source of the data used in comparisons, and the general weightings given to each category of factors.

All data collected by Planetizen through its independent surveys will be clearly cited and referenced. An explanation of the survey process will be included to outline how survey samples are created and obtained.

Proposed rankings methodology

Planetizen is planning to use the following methodology to rank programs in its forthcoming guide. The categories and criteria used in the rankings methodology are based upon data from a survey of planning students, educators, and professionals in 2006 that asked respondents to rate which factors are important when selecting a graduate planning program. All data used in the calculation of rankings will be drawn from surveys conducted by Planetizen of faculty, practitioners and the planning programs themselves (see Issues 2 and 3 for additional detail).

Program reputation Faculty opinions Planning practitioner opinions	25%
Faculty measures number of faculty members ratio of full-time to part-time faculty percentage of full-time faculty holding Doctorate degree percentage of faculty holding AICP/FAICP certification faculty publications (per capita) faculty demographic diversity	25%
Student measures Mean GPA of incoming class Mean combined GRE of incoming class Gender/racial diversity of student body Student retention	25%
Resources Financial aid per student Student-faculty ratio Median course enrollment Number of research centers/initiatives Number of established partnerships*	25%

*Defined as a formal relationship between the program and an external organization that facilitates additional educational and/or research opportunities to students and/or faculty.

Committee’s concerns regarding the proposed rankings methodology

Due to concerns regarding certain criteria measuring identical factors (i.e. GPA and GRE scores measuring student quality), the committee suggested that Planetizen analyze the correlation coefficients of its final variables to identify and eliminate duplicative measures that might exaggerate one particular factor in the methodology. It was also suggested that rankings criteria be organized by measures of program quality (such as student GPA and faculty productivity) and measures of desirability by students (such as financial aid and diversity).

How Planetizen plans to respond to this concern:

Planetizen will analyze its variables for correlation and make any necessary changes to its methodology should any criteria be deemed duplicative.
Planetizen will review its proposed methodology in subsequent years to see if criteria can be categorized more appropriately into program quality and student desirability measures.

Additional suggestions from the committee regarding the proposed methodology involve including income/class measures for incoming students. While Planetizen will attempt to collect appropriate measures from schools regarding student income and class background, adequate data for all programs is unlikely to be available in the near future. It is therefore unlikely that such measures can be included in the methodology for the forthcoming guide.

2. Validity of mixed data sets within comparisons

The committee expressed its concerns regarding any comparison of program measures obtained from different data sets. Specifically, the committee recommended comparing programs only when data from the same data set was available. At a minimum, the committee suggested that Planetizen should indicate when it utilizes different data sources for a particular program within a comparison or ranking, and cite the data source.

How Planetizen plans to respond to this concern:

Planetizen will rely on data collected through its survey of urban planning master’s degree programs to obtain quantitative measures of each particular program. In cases where a program does not respond to the survey, it will be omitted from all rankings except the reputational rankings. If a program response to the survey with incomplete data, Planetizen will create an average score based on available data from the program for the purposes of the rankings section.

Should the response rate to the 2007 program survey fall below the 2006 response rate (47 programs), Planetizen will reconsider the use of secondary data sources.

3. Validity of reputation data

The committee expressed its concerns about the use of reputational data collected by Planetizen in the program rankings section. Should reputational data continue to be incorporated in the program rankings, statistical indicators that assure the absence of any measurable bias need to be provided. This would include a sound survey sampling frame and design, and a survey response rate that could correlate to an acceptable degree of statistical reliability.

How Planetizen plans to respond to this concern:

Planetizen will publish information about the methods it uses to collect reputational data for planning programs.

How Planetizen collects reputational data

Planetizen, with the guidance of the opinion research firm Davis, Hibbits & Midghall, conducts three surveys to collect opinion data regarding programs.

1. Faculty/educator survey
Planetizen will attempt to survey the entire population of planning faculty, collecting the name and email of each faculty member listed in the 13th edition of the ACSP Guide to Undergraduate and Graduate Education in Urban and Regional Planning and/or on the website of each planning program. The survey asks respondents to indicate their opinion of a planning program by rating any program they have knowledge of on a scale of 1-7. This year’s sample size is approximately 1467.

a) (i). Committee’s concerns regarding the faculty survey

The committee indicated that it felt only full-time faculty should be surveyed, and suggested that a “Don’t Know” option be listed among the choices. Members also pointed out that some faculty had indicated that they had not received a survey. Additionally, it requested that Planetizen report the number of times (n) each school was rated by respondents.

How Planetizen plans to respond to this concern:

Planetizen will ask respondents to the faculty/educator survey if their employment status at the university/institution is full-time or part-time.
Planetizen will report the number of respondents who rate a program.
Planetizen will contact each program and post a notice on the PLANET listserv to alert faculty about the survey (to ensure the entire population is surveyed).
Planetizen will modify its existing reputation surveys to include an explicit “Don’t Know” option. It will request that any individuals who have responded to the survey already please resubmit their responses via the new survey. Planetizen will also include this option in future reputation surveys.

2. Planning practitioner survey
Planetizen surveys a random sample of 1500 management-level planning practitioners. As Planetizen has not identified a suitable master list to serve as a sample frame for this survey sample (we estimate a population of 10,000 management and senior-level planners and related professionals), the random sample is drawn from a compiled list of planning practitioners (also drawn at random) designed to approximate the target population. This list is stratified according to the following criteria:

Proportional by sector (based on 2004 APA membership survey) Local government Private sector Other government Non-profit	60% 25% 5% 10%
Proportional by location (based on 2004 APA membership survey) Urban¹ Suburban² Small town/rural³	60% 25% 15%
Proportional by population By state/province according to US and Canadian Census

¹Cities with populations over 50,000
²Cities within MSAs with populations between 10,000 and 50,000
³CDPs or counties outside of MSAs with population under 10,000

Planetizen will provide a detailed written explanation of its sampling frame for planning practitioners in future Guides (including the forthcoming second edition). Planetizen will also gather salary data for recent planning graduates from practitioners to supplement and compare with data reported from programs.

3. Graduate student survey
Planetizen is attempting to conduct a census of all current planning students at each program, as well as all recent graduates from the past two years. As this is the first time that Planetizen is attempting to collect opinion data about programs from students, we are closely monitoring the responses we receive to determine what follow-up is necessary to receive a high response rate.

The data from the academic and practitioner surveys will each be weighted at 50% for the overall reputation criteria used in the program rankings. The student data will not be included in the overall program rankings.

4. Validity of program comparisons

The committee expressed its concerns regarding the direct comparison of programs with a significant disparity in areas such as size, resources, degrees offered, and geographic scope. Specifically, the committee felt that smaller programs not be compared with larger programs, and that programs that attracted students from across the country (and globe) not be compared with programs that focus on educating students from the immediate region. The committee suggested that if rankings are to be included in the Planetizen Guide, a broad range of rankings that compares programs that are more similar in nature be published.

How Planetizen plans to respond to this concern:

Planetizen plans to expand the number of rankings in the forthcoming edition of the Guide with sensitivity towards creating rankings categories that provide valuable comparisons between similar programs. We believe this will provide students with useful additional information while helping to recognize programs’ strengths in various areas.
To accommodate these additional categories, Planetizen will supplement its program survey with a small number of additional questions to gather data that will be used to stratify programs into the appropriate categories.

Proposed rankings categories

Overall
Reputational
Doctoral degree granting programs
Master’s degree granting programs
Programs (under certain number of students based on data)
Programs by region
- Northeast
- Midwest
- South
- West
Publicly-supported universities
National universities
Regional universities (drawing 75% of students from within 150-mile radius)
Canadian universities
Programs by tuition*
- Less than $15,000/yr
- $15,000-25,000/yr
- More than $25,000/yr
Largest programs (enrolled students)
Most diverse student body
Largest percentage of international students
By student satisfaction

*Public institutions will be ranked using both in-state and out-of-state tuition rates.

Additional listings

Part-time programs
Historically black colleges and universities

5. Rankings by specialization

The committee expressed its concerns regarding the methodology for the program rankings by specialization included in the first edition of the Planetizen Guide. It was noted that any rankings of specializations should not be based on reputational data or opinion, but on data that provided evidence of a program’s strength in a particular area. Suggestions were made to 1) expand the number of specializations listed, and 2) provide a listing of programs that score well in a particular area, without ranking them. Alternatively, it was suggested that schools might self report the areas in which they believe they should be recognized.

How Planetizen plans to respond to this concern:

As obtaining quantitative measures of the strength of a program in a particular specialty area is resource-prohibitive, Planetizen has relied on reputational data to rank programs by specialization. Students have emphasized to us that rankings in specialty areas are of particular interest, so we are inclined to continue to rank programs in this way. However, Planetizen suggests two potential alternatives for program rankings by specialization.

Use PAB accreditation
Planetizen would publish a list of PAB accredited schools for each specialization area that were deemed to have a program focus in that area during the accreditation process. This list would be provided to Planetizen for inclusion in the Guide by PAB. This list would not contain rankings. A note would accompany this list indicating that the data used in PAB accreditation was collected over the past 5 years.
Include additional measures
Should PAB data not be available for use, Planetizen would publish rankings in specialty areas that would include reputational data but also incorporate various additional measures that could compensate for any “halo” effect.

These could include:
- the number of faculty who indicate a particular specialty area
- the proportion of enrolled students who identify a specialty area as their area of focus
- whether the program is home to a dedicated research institute or center in the specialty area

It should be noted that the availability of such data will depend on the response rate to Planetizen’s faculty and graduate student surveys.

Members of the committee also discussed using a combination of PAB accreditation, reputation, and quantitative measures to arrive at rankings by program specialization.

6. Program survey and profiles

The committee expressed concern about specific questions on the 2007 program survey to which certain programs may find it difficult to respond. These include questions relating to faculty publications, graduating student employment statistics, and awarded financial aid. Furthermore, the committee noted some confusion over how certain schools were included in the first edition of the Planetizen Guide – in cases with incomplete profiles – while others were not included. Finally, some questions were raised regarding the validity of program-reported data, particularly in reference to questions that request data that is difficult to gather.

How Planetizen plans to respond to this concern:

The Planetizen program survey was created to collect quantitative information that students have requested for evaluating graduate planning programs. It was designed in consultation with former program chairs and Planetizen’s opinion research firm DHM. Our objective is to collect a comprehensive set of information about each program. However, we recognize that not all programs are able to respond to every question in the survey. For programs in this situation, a “not available” response is permitted – though we hope such data can be made available in the future.

In consideration of the committee’s concerns:

Planetizen will designate which questions in the program survey are optional to help guide program administrators as they fill out the survey.
Planetizen will notify program administrators about which questions are optional, and encourage their participation whether or not they respond to optional questions.
In cases where a program has difficulty responding to a question that is not optional, Planetizen will do its best to work out a solution for that program. Any data reported by programs that does not meet the standard criteria for a response will be cited in its profile.
All programs that report data via the program survey will be included in the detailed profile section of the Planetizen Guide. If a program does not respond, it will be omitted from this section.
Programs that respond to optional questions will receive an expanded profile that includes the additional data, including a 300-500 word introductory message (if provided).
Planetizen will ask program administrators to identify any questions on the program survey where they found it particularly difficult or impossible to provide a response. The responses will help Planetizen reevaluate its program survey for future years.
Planetizen will use outside data sources (when available) to check the validity of program-reported data (e.g. comparing faculty publications with the number of citations listed in the ISI database).

RESOLUTION

Should the procedures and actions proposed by Planetizen serve to adequately address the concerns of the ACSP special committee, Planetizen requests the following actions be taken by the committee:

ACSP will communicate with each member program to inform them that ACSP’s concerns about the program survey and guide have been satisfactorily addressed. This communication should include a link to this statement of consultation for reference, and clearly state that ACSP does not discourage programs from participating in the Planetizen program survey, nor does it discourage faculty or students from responding to opinion surveys from Planetizen.
ACSP will issue a statement for publication on the PLANET listserv to the same effect.
Members of the ACSP special committee are requested to affirmatively recognize the plans and procedures Planetizen has proposed to respond the concerns of the committee, and refrain from making statements that damage the cooperative nature of these proposed actions.