An Initiative of NORC at the University of Chicago

Career Pathways for Graduate Degree Holders: What Do We Know and Where Do We Go from Here

Career Pathways for Graduate Degree Holders: What Do We Know and Where Do We Go from Here

Introduction

NORC at the University of Chicago, in partnership with the Council of Graduate Schools and with support from the Spencer Foundation, convened a workshop in July 2017 focused on conceptualizing and measuring graduate degree holders’ career pathways. The workshop was particularly concerned with identifying research needs to inform decisions by those most directly connected with graduate degree programs: deans, department heads, faculty, and students. Participants included senior university administrators officially responsible for graduate education, researchers engaged with graduate education, leaders of organizations concerned with graduate education practice, and federal officials who fund graduate education or who serve as stewards of federal data sets that might inform the career pathways discussion. This Working Paper, prepared initially to frame workshop discussions, now is revised to incorporate observations and conclusions of participants. It is written to assist university officials as they attempt to better understand the career pathways of their graduates.

Background

In aggregate, graduate education in the U.S. pays off in significant ways for both individuals and for the nation. For decades recognition of this fact induced prospective students to enroll in graduate programs and led the federal government to make investments in graduate education and the research enterprise. But as the effects of the “great recession” of 2008 continued to be felt through the succeeding years, the specific value of both individual investments and governmental investments in graduate education came under closer scrutiny. This intensive examination led to a recognition of the need to understand the actual career outcomes of graduate alumni.

The first section of this paper takes stock of current sources of information on career pathways, matches it against what stakeholders need to know about those pathways, and explores the barriers to acquiring the information required. The second section highlights specific technical considerations to be taken into account when attempting to track career pathways of graduates. The final section summarizes main points from the July 2017 workshop.

Knowledge Needs and Barriers

What We Know About Career Pathways of Graduate Degree Holders

In understanding career pathways of graduate degree holders it makes sense to start with the data currently available, and such examinations sometimes produce surprising findings. For example, Norman Bradburn’s study of career paths of currently employed PhD holders in the U.S. reports that only about one quarter of those receiving science, engineering, and health PhDs and about 60 percent of humanities PhDs report their occupation as post-secondary teacher or professor. 1 These figures represent only slight declines over the last two decades. While there is variation across fields of doctoral study, it is important to note that only a few fields of doctoral study –in addition to humanities, mathematics, political science, and sociology — had more than half of their PhDs working as post-secondary teachers and professors. 2 Bradburn’s work highlights the importance of knowing the actual career outcomes of our graduate degree holders. Some data collection efforts underway currently can inform our general understanding of career pathways.

1 Norman Bradburn, “Thoughts on the Future of Graduate Education.” NORC Working Paper prepared for Workshop on Graduate Degree Holder Career Pathways: Conceptual and Methodological Approaches to Measurement and Analysis, held on July 25, 2017 in Washington, DC.
2 National Science Foundation, National Center for Science and Engineering Statistics, Survey of Doctorate Recipients, 2013. http://ncsesdata.nsf.gov/doctoratework/2013/

Here we examine five highly informative sources of information currently available on degree recipient outcomes: one major national survey; three selected university data collection efforts; one university- centered administrative data linking project, and the Council of Graduate Schools Career Pathways research projects.

Survey of Doctorate Recipients (SDR)

The National Science Foundation (NSF) sponsors the largest ongoing data collection project on career pathways of science and engineering doctorate holders – the Survey of Doctorate Recipients (SDR). SDR began in 1973 and follows a nationally-representative sample of science, engineering, and health (SEH) doctorate recipients from U.S. institutions from completion of the doctorate through age 75. The survey has, with few exceptions, been conducted every two years, and the sample is augmented each cycle with samples of new doctorate recipients. As evidence of the growing interest in the career pathways of doctoral degree holders, the SDR sample size more than doubled for the 2015 survey cycle to 120,000 individuals, from approximately 47,000 individuals in the 2013 cycle. This sample size increase was designed to significantly improve estimation capabilities at the fine field of degree level, and the new sample is representative of doctorate recipients from U.S. institutions currently living in the U.S. and abroad.

Doctoral career pathways are conceptualized and measured in the SDR primarily in terms of labor force participation, broad sectors of employment, occupations, principal work activities, and occupational ladders and salaries. Key general findings from the SDR include:

  • Labor force participation and employment rates are consistently very high for the doctoral population at all age levels. Age-specific retirement rates are low compared to the general population and have decreased in recent years. 3

3 Selfa, L. and S. Proudfoot (2014). “InfoBrief: Unemployment among Doctoral Scientists and Engineers Remained Below the National Average in 2013 (NSF 14-317). Arlington, VA: National Science Foundation, National Center for Science and Engineering Statistics.

  • Most doctorate recipients work outside of academia. Among employed doctorate recipients residing in the U.S. in 2015, 45 percent were employed in educational institutions, 47 percent were employed in business or industry, and 9 percent in government. 4

4 Chang, W-Y and D. Foley (2017). “InfoBrief: The 2015 Survey of Doctorate Recipients Expands Its Population Coverage and Reporting on Field of Study” (NSF 17-319). Arlington, VA: National Science Foundation, National Center for Science and Engineering Statistics.

  • Research and development was the primary or secondary work activity of 61 percent of the U.S.-residing SEH doctorate holders in 2013 (the most recent year available), well ahead of management/administration (41 percent), teaching (29 percent), and professional services (15 percent). 5

5 National Science Foundation, National Center for Science and Engineering Statistics, Survey of Doctorate Recipients, 2013. http://ncsesdata.nsf.gov/doctoratework/2013/.

  • Occupational ladders are most clearly defined in academia with the tenure-track system. Outside of tenure-track positions, ladders are common but less standardized and are harder to identify and measure.

The SDR is generally agreed to be the most robust data set for understanding career pathways across a wide range of fields. It provides information that is enormously helpful to public policy makers as well as to individual universities considering the nationwide opportunities associated with particular doctoral degree programs. The SDR clearly fulfills the purposes for which it was created but it does not provide the fine-grained information about career pathways of individuals who complete degrees from specific PhD programs nor, of course, does it provide information about master’s degree recipients who do not also earn a doctorate.

Data Collected by Universities on Graduate Degree Recipients

While many universities aspire to track the career pathways of their graduate degree alumni, for most this remains only an aspiration. The 2014 Council of Graduate Schools (CGS) Survey of Graduate Deans found that only 34 percent of responding graduate deans reported formal data collection of any kind, that 53 percent reported some informal data collection, and that 13 percent reported no effort current effort to collect this information at all. 6 Here our focus is only on formal data collection efforts. Some formal data collection efforts are university-wide, while others are conducted at the program level.

Stanford University; the University of California, Berkeley; and Cornell University provide three examples of universities that have collected, in different ways, university-wide, point-in-time data on their own graduate alumni outcomes.

Stanford University.

Stanford initiated in 2013 a project to mine publicly-available, Internet-based information about the careers of 2,420 doctoral recipients who earned degrees either between 2002 and 2004 or between 2007 and 2009. In this work they were able to identify the initial employment and the employment by 2013 of both cohorts of students. The researchers carrying out the study were able to locate public information on current (in 2013) employment for 81 percent of the alumni in the study and initial employment for 74 percent. 7 Stanford has made its employment data available on its website for use by faculty and current and perspective students. The major finding of this study is the significant diversity among the career paths of doctoral graduates. 8

University of California, Berkeley.

UC-Berkeley conducted a survey spanning four decades of graduates. But rather than sampling a population at two points in time, Berkeley sampled doctorate recipients from the graduating cohorts over 40 years (1968 – 2008.) This cross-sectional study of a broad population was designed to establish baseline information on career paths, satisfaction with graduate education, and perceived value of the educational experience. The Berkeley study drills down into several interesting specifics such as a retrospective assessment of the relative value of various elements of the respondent’s doctoral education. The Berkeley study is also publicly available on the website and provides rich information on the experience of the sample of Berkeley graduates over a 40-year period. 9

Cornell University.

Cornell took the approach of generating a cross-sectional view of the employment status and career-related perceptions of doctoral graduates completing their degrees from 1994 through 2014. Keeping the survey open through January 2016, they achieved a 28 percent response rate from the survey and supplemented this survey data with employment and location information on an additional 40 percent of the graduates. This latter information was generated through online searching. Of special interest in the Cornell study is a series of questions on how the respondents’ degree programs might be improved for the benefit of future students. The Cornell data are also available for review on the Cornell website. 10

Each of these three examples, while using different time frames, different survey instruments, and different dissemination strategies, makes significant progress in highlighting the career pathway experiences of students completing degrees at these respective universities. Workshop participants agreed that, while the federal survey programs such as the SDR were enormously helpful in understanding broad trends, the level of disaggregation reflected in these three University initiatives was essential if graduate Deans and other administrators wanted to use the information to help improve practices in particular programs. Program improvement can best be achieved by understanding the experiences of graduates from that particular program. Several workshop participants also asserted the need to ultimately transfer from a point in time approach to collecting career outcomes data by program to a longitudinal survey.

In addition to the graduate school-wide efforts of the types conducted by Stanford, UC-Berkeley, and Cornell there are also a variety of program- or department-initiated surveys designed to understand the career paths of program graduates throughout U.S. universities. This program-level type of alumni survey effort, though not explicitly discussed in our workshop, is found most frequently in professional degree programs, especially in MBA programs, where students considering the program are particularly focused on the career outcomes of program graduates.

UMETRICS-IRIS Administrative Data Linkage Project

Federal and state government agencies routinely collect and compile data on individuals’ employment status and earnings, and social scientists and government officials have made great strides in recent years to develop ways of linking those data to records on a wide variety of subpopulations and samples of particular policy interest. Obtaining access to government records requires special permissions tied to strong assurances of data confidentiality and can involve considerable time and significant costs. Nonetheless, once access is granted, researchers obtain high-quality data with close to 100 percent coverage of individuals within the U.S. labor force and, importantly, little or no response burden on the individuals on whom the data are compiled.

A team of researchers from the Big Ten Academic Alliance group within the fifteen universities which form the Committee on Institutional Cooperation (CIC) have developed a service through which universities can obtain statistical reports on the scientific productivity of their graduates and the economic impacts of their graduates’ research. Initially referred to as the UMETRICS demonstration project, the program has since 2015 become part of a full-featured service called the Institute for Research on Innovation & Science (IRIS) located at the University of Michigan’s Institute for Survey Research. 11 IRIS is a consortium of research universities collaborating to produce high-quality data and analytic tools on “the process, products, and impacts of scientific research activities.”

IRIS obtains student rosters from member universities and links those to a variety of non-university databases with records on various outcomes. The linked data are then analyzed to produce reports on student outcomes which IRIS provides to the member universities. Outcome data sources include publication and patent databases, the Federal Research Data Centers (FRDCs), annual and cumulative earnings data from the Social Security Administration, and unemployment records from the U.S. Department of Labor and state unemployment agencies.

UMETRICS can provide very useful information to universities about the economic impact of their research programs and the contributions of their graduates, especially if aggregated up to the broad field level. But because it does not survey individual graduates of particular programs, UMETRICS, like the national surveys such as the SDR, is unable to provide the fine-grained information on graduates’ assessments of their programs and the contribution of those programs to individual careers that administrators likely need to promote program improvements.

Council of Graduate Schools Career Pathways Project

In April 2012, the Council of Graduate Schools (CGS) and Educational Testing Service (ETS) jointly issued a policy report entitled Pathways Through Graduate School and Into Careers. For the first time in a national study the report connected the dots between the discussion about jobs (as the country continued to recover from the great recession) and the role of graduate school in preparing students for successful and meaningful careers. While the report gave voice to those graduate education leaders who were strongly invested in helping students find employment upon graduation, it also reported findings indicating that graduate deans did not believe students were receiving adequate information to make fully informed career pathway choices. It called on graduate schools and graduate programs to better understand the pathways that lead to students’ ultimate goals in order to prepare them well for the journey. This work formed the foundation for a series of studies to be launched by CGS designed to understand the career pathways of graduate degree holders. These studies had two primary goals: 1) to make career outcomes for graduates of any particular program transparent to program applicants and 2) to provide program faculty concrete information about career outcomes that might be used for program improvement.

Guided by these two goals, CGS launched a series of studies covering the entire spectrum of disciplines from humanities through engineering. These were designed to explore how meaningful information about career pathways of program graduates could be collected and effectively used to both inform students and transform programs. The most recent publication from this line of work, Understanding PhD Career Pathways for Program Improvement, 12 summarizes what we currently know about how information is being collected about career pathways at the doctoral level and how that information is or should be used. A recently-launched CGS study 13 will work with over 15 institutions nationwide to further develop our understanding of graduate degree holder career pathways.

12 Allum, JR, JD Kent, and MR McCarthy. 2014. Understanding PhD Career Pathways for Program Improvement. Council of Graduate Schools. Washington, DC.

What Stakeholders Want to Know

Any discussion of career pathways of graduate degree holders must recognize both that multiple stakeholders need to understand these pathways and that, taken together, the data sets described above address many of their information needs. For policy makers and foundations interested in assessing how their investments impact the availability of human capital to the U.S. economy, the SDR is a rich source of information on the work-related contributions PhD degree holders make to the country. For researchers, a vast array of federal surveys, from the Current Population Survey to the American Community Survey, enable deep workforce analyses that both drive our understanding of employment patterns and trends forward and provide input into policy making. For universities considering investments in new program areas, these data as well as creative uses of UMETRICS data, if properly analyzed, provide useful information pointing to program areas to pursue and to avoid.

As workshop participants noted, however, these existing data sets fall short in addressing the information needs of two of the key stakeholders in the graduate education space: students considering application to a degree program and faculty working to improve a degree program. We know, anecdotally, that graduates are often surprised by the nature of job prospects at the end of their degree programs. In an effort to explore this narrative empirically, the CGS/ETS Career Pathways project conducted an online survey of students who took the GRE General Test between 2002 and 2011. Sixty percent of the enrolled and degree-completed respondents indicated that they received less information than they needed or no information at all about potential careers associated with the degree. Among those who did receive information, only 16 percent indicated that it was very useful. 14

14 Council of Graduate Schools and Educational Testing Service. (2012). Pathways Through Graduate School and Into Careers. Report from the Commission on Pathways Through Graduate School and Into Careers. Princeton, NJ: Educational Testing Service.

There are also serious information needs among the faculty who work diligently to shape the curriculum with little or no systematic knowledge about what their graduates believe about the preparation they received for the jobs they got and the careers they pursued. While the CGS/ETS Deans survey did not provide comprehensive insight into the specific skill preparation missing in particular fields, it did indicate a notable mismatch between the skills graduates reported as “very important” to them and the skills for which they felt “very well prepared” by their degree programs.

In sum, fine-grained career outcome data on graduate experiences and employment are needed for applicants to make informed choices and for faculty to continually improve their programs. This information is currently not available across graduate institutions. The SDR, our strongest national data set addressing career pathways for doctorate holders, describes career outcomes at levels of aggregation well above that of particular degree-granting institutions, let alone degree programs within an institution. This does not provide the program-level and institution-level data that students need to make decisions about applying to and attending specific programs or that faculty need to develop, refine, and reform curriculum within their programs. A few universities are launching data collection efforts and that work is providing valuable point in time information about their own graduates’ experiences. But a recurring theme in the workshop was that more is needed. Moving forward, the gold standard in terms of what information is needed by applicants and faculty would include degree-program-based, comprehensive, longitudinal data that describe in some detail the career trajectories of a program’s graduates and illuminate, at different points in time, the graduates’ assessments of the value of their graduate school education.

Barriers to Acquiring and Disseminating

Barriers to Acquiring and Disseminating Institution-level Graduate Alumni Career Outcome Data

If the information generated by tracking career outcomes of graduate degree holders is as beneficial as argued, why is it that so few universities are currently doing this work? This was a key question motivating the workshop. Initially we posited three categories of reasons: Competition, Limited Budgets, and Technical Challenges.

Competition. Discussion about following the career pathways of graduates became more prevalent subsequent to the “great recession” in 2008. In the initial discussions about following graduates, it was not uncommon for graduate deans to report that they were not sure such a project would be widely embraced by their graduate departments. Faculty might be reluctant to display this kind of information publicly when their competitor programs were not doing so, or worse, were doing so on a very selective basis, highlighting only the stars for potential applicants to see. Much like the issue deans faced in the early days of promoting the need to publicly reveal completion statistics on PhD programs, deans appreciated that their departments operated in a very competitive market place for top students and that, as deans, their first obligation might be to “do no harm.”

As it becomes a more standard practice for universities to publish a wide variety of performance indicators across the campus, including completion and attrition rates in PhD programs, the understanding of what provides competitive advantage may be shifting. Certainly workshop discussion confirmed that reluctance to share data publicly was still alive in some departments. Furthermore, practices around using the information generated varied across institutions. Some found that using data internally, to encourage departments to acknowledge unfavorable graduate outcomes, was a useful strategy. Others agreed but suggested the mode of use was important and that public “blaming and shaming” of faculty or programs was simply not effective in the long run. All agreed that the method for displaying information must be curated to reflect community values and culture, but no one expressed the view that fear of competition was a legitimate reason to avoid pursuing knowledge about career outcomes. Some observed that as top graduate programs and universities nationwide begin to fashion some form of hard data-based reporting on career outcomes of graduates (master’s and doctorate), it is possible that publishing career outcome data of graduates by program may enhance rather than diminish a program’s competitive edge.

Limited Budgets. By 2015, state and local funding for higher education remained below 2008-2011 in inflation-adjusted dollars. 15 For public higher education, even the improving economy has not significantly changed that picture. For PhD-intensive research universities heavily reliant on federal research funding, the financial challenges are substantial. After 2014, when federal funding for R&D in universities fell by nearly 4 percent 16 due to the phase out of the federal stimulus funding, research budgets began to once again rebound. In 2017, however, universities faced the most unstable political context they have faced in decades, especially regarding research and student support. Across the country many academic administrators are planning for defensive cuts. In this context is it reasonable even to think about expending resources to create a new database (tracking career pathways of graduate degree holders) whose full utility is yet to be tested?

One surprising finding from the workshop is that no participant expressed budgetary constraints as the major barrier preventing the university from tracking career outcomes. The more important question was will the development of this kind of database improve the prospects for the institution and its programs? While institutional resources varied considerably among our university participants, all institutions want to spend the resources they do have to advance their teaching and research missions, and a key step toward achieving those ends may be to understand better the career outcomes of their graduates. Even in a world of competing claims for very limited resources, a case can be made for the investment in some form of career pathway project.

Technical Challenges. Collecting useful information and building a growing and adapting body of knowledge about career pathways poses a number of technical and organizational challenges.

Managing alumni lists is challenging. Whether an institution is collecting data via surveys, administrative data sources, or through abstraction, simply identifying the most current name and contacting information for graduate alumni is a challenge for even the best managed programs. Data are often incomplete or collected inconsistently over time. Keeping the data current requires both dedication and systematic computer programs and procedures to keep the information secure and accurate. Across institutions, alumni list sources vary widely in content and completeness.

Workshop participants’ comments confirmed that different universities acquire their information in different ways. Some have robust institutional research (IR) offices, and some graduate schools have their own IR staff to supplement the university IR department. Others contract out their information-needs work. A few graduate schools work with Alumni Offices. One set of public universities reported assistance from their system office. Most institutions acquire information necessary for decision-making through some combination of internally generated work, data from public data sources, and externally contracted services.

Finally, there are questions about which data elements are critical to understanding career pathways across graduate fields. With many independent formal and informal career tracking efforts occurring at universities across the U.S., it is unlikely that the information gathered is consistent, preventing data integration for evaluation or benchmarking. Without the ability to cogently synthesize the data for comparison across universities or programs, the utility of the data becomes diminished for the institutions collecting it and for the broader group of student and policy-making stakeholders. Participating administrators expressed the most interest in career outcome data that facilitated peer group benchmarking by program.

Whatever the content and however the information is secured, all participants agreed that there is a set of technical challenges the graduate schools must face as they try to enrich their understanding of the career pathways of their graduates. Those challenges are addressed in Part II of this working paper.

Technical Considerations

Technical Considerations in Developing Studies of Graduate Career Pathways

The major challenges to collecting and utilizing career pathways data include: defining what you want to know; identifying the graduate population about whom you want to know it; locating this population; establishing the most appropriate data collection mode or modes; developing an instrument or other methodology to collect the data; securing the data; and developing an analysis and reporting scheme that ensures the policy objectives are achieved. At each of these stages in the process there are both technical and organizational challenges. Below we will address the technical considerations and challenges, which universities must address as they attempt to understand graduate career pathways.

Research questions. The success of any research effort depends critically on asking the right questions. Research questions for quantitative studies typically ask about relationships among variables of particular interest, and these are often usefully sharpened into specific hypotheses that the research will test. Developing strong research questions and hypotheses can be facilitated with “theories of action” about how graduate programs attempt to affect graduates’ decisions on career pathways 17 and more specific “logic models” relating system inputs to outcomes via mediating conditions and activities. 18 Advance work articulating a conceptual framework of how one currently understands the program, pathways, and outcomes; the points where evidence is needed; how the new evidence will be assessed is challenging but essential to shaping the overall study design and questionnaire content.

17 For a guide to articulating theories of action see http://info.k-12leadership.org/creating-a-theory-of-action
18 For a discussion of logic models and their role in evaluation research, see W.K. Kellogg Foundation, *Logic Model Development Guide* (2004), downloaded September 13, 2017 from https://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide.

Population definition. Research must be clear about the population of primary interest and the relation of any sampling strategy to that target population. Candidate populations include graduates by program for one or a defined cluster of years, and entering graduate students by program for one or a cluster of starting years.

Collecting data from the population versus a sample. While the full population of graduates (within a particular range of graduation years) is often the target for an institutional data collection effort, drawing a sample may be feasible and could provide cost savings. The size of the population and the likely response rate to the data collection effort are crucial determinants of whether a sample is feasible. The expected number of completions from a sample must be large enough to provide sufficient statistical power for estimating levels and relationships of interest at established standards of statistical significance. As sample size requirements approach the population size, sampling is less feasible.

Data collection mode. There are two principal models of collecting data: 1) obtaining public records, usually from online sources, and abstracting the data from these sources, and 2) survey data collection directly from the population or a sample of it.

  • Abstraction. While the population of doctorate recipients is more visible online than most other populations, some doctorate recipients do not maintain any web presence, and some who do are not diligent about updating their online biographies and CVs. While attractive in terms of cost compared to surveys, relying on online documentation obtained even through the most rigorous web searches is likely to miss most doctorate recipients outside of academia and the most active segment of the non-academic research communities. Furthermore, this approach does not allow for collecting any attitudinal information about the alumni’s graduate experience or career choices.
  • Survey data collection. Survey methods hold the promise of collecting data from a more representative cross-section of the target population. Universities often maintain contact information for graduates and, while many graduates do not maintain up-to-date online CVs or bios, they tend to be possible to locate via online resources. Commercial locating services such as Accurint® maintain and update comprehensive lists of individuals and, for a fee, can provide up-to-date mailing addresses, telephone numbers, and email addresses.
  • Administrative data collection. University records on their degree recipients can be linked to state and federal administrative records relating to the graduates’ unemployment, earnings, and health care to provide comprehensive information on select outcomes. Access to the government records involves preparation of applications and strong measures to safeguard the confidentiality of graduates’ personally identifiable information, as well as expenses to cover government processing costs. A leader in efforts to provide these linkages is the University of Michigan’s IRIS program, discussed in Part I.

Locating the target population. Institutions vary with respect to the quality of their contacting information on graduates, but locating all members of the target population or sample is almost always a challenge. Professional survey organizations require significant funds to conduct comprehensive locating and must utilize an array of resources to locate all individuals for data collection.

  • Domestic vs. international. The most recent data from the SDR (2015) indicates that nearly 40 percent of all SEH doctorate recipients from U.S. institutions were born outside the U.S., and most of those (73 percent) stayed in the U.S. after receiving their doctorate. Locating those who leave the U.S. poses many challenges. The NSF’s SDR sample now includes non-U.S. resident doctorate recipients, and the data collection contractor (NORC) has developed effective procedures for locating and contacting them.
  • Recent vs. long-term graduates. Locating of graduates usually increases in difficulty with each year away from the institution. The locating effort can be greatly helped by use of commercial locating services (e.g., Accurint® or Experian), which for a fee will provide mailing address, email address, and telephone number to data collection projects. The commercial services build their databases from credit and financial records but provide the locating records without any release of current or historical financial information and the locating information is classified as public data. Costs per case are minimal for so-called batch searches, whereby the service is provided with a set of names and other identifying information on a set of individuals. Higher rates apply to access to full locating data resources for individual lookups. A common strategy is to start with the batch search and then use the more expensive individual locating resources only for cases not successfully located in the batch search.
  • Sector of employment. NORC’s experience shows that individuals working in academia and particularly those engaged in research and publication are generally much easier to locate with online searches than individuals employed in business or industry, government, or other sectors. To ensure adequate coverage and representative survey results, care must be taken to carefully search for individuals who did not follow an academic career path.
  • Information available from degree-granting institutions. Contact information from degree-granting institutions may be sufficient to locate new graduates, but for older alumni it will likely be out of date and will normally need to be updated and extended. NORC’s experience has been that universities are generally willing to share some records but not the annual giving databases, which tend to be the most up-to-date.

Instrument development. Developing and deploying a high quality survey instrument is a painstaking and time-consuming process. Aspects of instrument development to consider are described below.

  • Content. The process of defining instrument content typically proceeds from a logic model identifying key constructs and relationships, developing draft items for the constructs, testing and refining the draft items through think-aloud “cognitive interviews” with ten or more members of the target population, and piloting the instrument with a small sample to assess whether the items and the instrument as a whole work as planned. Ideally, much or even all of this development process can be avoided by drawing on past research efforts, borrowing questions and presentation formats from other studies with proven success. In addition to lowering development costs, borrowing instrumentation can have the important benefit of facilitating benchmarking comparisons with national or peer group populations previously surveyed with the same instrument.
  • Technical development. Instrument development has become a sophisticated specialization in the world of survey methodology, and research firms like NORC and its competitors are leading sources of that expertise. Instrument length is generally kept to a minimum in order to reduce respondent burden and improve response rates. Shorter instruments reduce costs associated with multiple prompting efforts, incentive offers, and, if telephone or in-person interviewing is used, labor costs. While feasible lengths vary depending on many factors, we generally find that surveys taking 15 minutes or less to complete require much less recruiting and follow-up effort than those taking 30 minutes or more.Survey methodologists are also expert with respect to question wording and response formats. Led by the pioneering work of Seymour Sudman and Norman Bradburn, 19 this body of knowledge is informed by insights from cognitive psychology as well as the psychology of perception and emotions. The survey methodology literature is replete with examples of how seemingly innocuous differences in wording and format can have large effects on responses. Particularly for developing new survey questions, input and review by experts in this area is beneficial.

19 Sudman, S. and N.M. Bradburn (1982), Asking Questions: A Practical Guide to Questionnaire Design. San Francisco, CA: Jossey-Bass Inc.

Data Collection. The success of data collection efforts with respect to response rates depends heavily on the contacting protocol and follow-up efforts. Attaining high response rates – 70 percent or higher – requires well-crafted contacting materials, incentive offers, and multiple follow-up efforts. A key piece in most successful surveys is an advance letter sent via regular mail to each targeted subject. The letter briefly introduces the study and its purposes, explains why the individual is being asked to participate and how they would participate, describes any incentive being offered, and provides contact information to answer any questions and address any concerns they may have. Receiving a letter leads most recipients to be much more receptive to the next step of the survey, which is usually either an email with a link to an online survey instrument or a phone call asking to interview the individual.

Incentive offers can take a variety of forms. While the early research on incentives found that token cash amounts of a few dollars “pre-paid” up front of the actual data collection (usually mailed questionnaires) was quite effective in eliciting cooperation, more recent evidence points toward greater effectiveness of contingent “post-paid” payments made only to those who complete the survey. Survey research has an interesting history of evolving incentives which seems now to be converging on the use of Amazon.com gift certificates, which are easy to purchase online in virtually any quantity and, via email, to distribute immediately to respondents.

Highly educated graduates tend to be extremely busy people and even the best advance letter and incentive offers are not going to elicit more than about a 30 percent response rate from the initial contacts. Further prompts via email will steadily improve participation, and improvements are normally seen with email prompts sent once a week across a 12 week period to keep the study alive while not aggravating the slow-to-respond with more frequent prompting.

Adaptive non-respondent follow-ups are increasingly recommended by survey researchers seeking to minimize potential non-response bias, whereby findings are distorted by respondents and non-respondents systematically differing with respect to how they would answer questions. For example, graduates who are having greater career difficulties may be less likely to respond to a survey than their more successful counterparts. To try to reduce such biases, researchers identify nonresponding individuals likely to differ the most from respondents on key post-graduation experiences and outcomes and concentrate follow-up efforts and even incentive offers on those individuals. As these targeted individuals tend to be the most difficult from whom to gain cooperation, this is often purposefully done at the expense of the overall response rate. However, a lower overall response rate in favor of reduced non-response bias is increasingly viewed as an acceptable trade off by leading federal statistical agencies.

Reporting and Data Delivery and Dissemination. Survey firms such as NORC strongly endorse the AAPOR Transparency Initiative, which establishes high standards for reporting on all aspects of sampling and data collection. This well-codified system of standards is available online at no cost and is an invaluable tool for anyone undertaking survey research or contracting for it.

Once data are complete and documented, decisions need to be made about how to make the data available while keeping the identity of the participants and their individual responses secure. Will a public use file be made available, or will data only be analyzed and distributed internally? What level of suppression or rounding will be necessary to keep individuals’ identities secure? Who will have access to the microdata and how will that be maintained in a secure manner? There are data management solutions for maintaining data securely while providing accessibility. Whatever approach is used, it should be considered well in advance of data collection.

Workshop Observations and Conclusions

Graduate Degree Holder Career Pathways Workshop: Observations and Conclusions

The Graduate Degree Holder Career Pathways Workshop was designed to explore, with the expert community, the conceptual and methodological approaches to measuring and analyzing career outcomes. It is challenging to summarize this extraordinarily rich discussion and there was no attempt to try to achieve a consensus on a final set of points, but this list of observations and conclusions does represent a synthesis of the main points raised throughout the workshop, especially by the university administrators in attendance. We focused disproportionately on this group since it is their practice the gathering was ultimately designed to inform.

  • There is strong interest in the graduate education community for collecting information about doctoral career pathways as evidenced by the extraordinary response of 73 universities to the CGS Career Pathways project launched in July of 2017.
  • While existing national data sets, especially the National Science Foundation’s Survey of Doctoral Recipients (SDR), are enormously useful for providing broad information about the varied doctoral degree pathways, these data sets fall short in addressing the information needs of two key stakeholders in the graduate education space: students considering application to particular degree programs and faculty working to improve an existing degree program.

Discussion focused on three barriers to acquiring and utilizing doctoral outcome data: competition, limited budgets, and technical challenges. The conclusions among the participants were roughly as follows:

  • Competition: Discussion focused on how doctoral career pathways and outcomes data should be shared. Deans observed that in cases when career outcome data were made available to faculty, faculty were willing to take action, but with respect to how these data were shared, one dean noted that anything this controversial needs to begin with a conversation. Another dean pointed out that “blaming and shaming” rarely works. Several participants agreed that since the publication of such data might be misinterpreted in a way that would be harmful to programs and institutions, that it is very important to “have a plan for how to tell the story before telling the story.” However, there was an emerging consensus that, while the dissemination of information on career outcomes must be curated in a way to reflect University community culture, fear of competition will not in the long run frustrate the broadening of knowledge about graduate career outcomes. Going forward, top graduate programs and universities nationwide are likely to provide information on career outcomes of graduates and see that as a way to gain rather than diminish competitive advantage.
  • Limited Budgets: Budgetary constraints and competing claims for investment are part of every graduate school’s reality, but in this collection of universities no participant expressed budgetary constraints as a major barrier preventing their University from tracking career outcomes. Discussion focused more on how much of an investment would be needed and if the information could be garnered in a way that was sustainable. In the words of one dean: data collection needs to be feasible, easy, and cost-effective to be sustainable.
  • Technical Challenges: Technical challenges emerged as the topic that generated the most extensive discussion in the workshop.
    • First, some university officials noted the difficulty of identifying technically prepared individuals, either from the current staff or through new hires, ready to design and implement the data collection effort, and to analyze the results. One dean noted that very highly qualified faculty had full research agendas and, understandably, were not inclined to lend their talents to the graduate school data collection effort.
    •  Several specific data collection challenges were noted. A number of deans mentioned the special challenges of tracking career pathways of international alumni.
    • Response rates were a topic of interest, and both deans and researchers expressed a particular concern with understanding how non-respondents would differ from respondents as essential to any valid analysis of the data. Several ideas about how to reduce non-response bias and boost response rates were discussed including a variety of incentive programs, including random selection for cash payments as well as other strategies.
    • Many indicated that their own institutional data collection effort would be significantly enhanced if similar information were collected on institutions against whom they would like to benchmark their own performance. They noted that national benchmarking, available in part from the SDR, is much less useful than benchmarking within peer group.
    • One of the deans from an institution engaged in studying the career paths of their graduates indicated that a point-in-time survey does not necessarily inform the university about career pathways of graduates. She raised the question: how do we transition from a point-in-time study to a longitudinal survey? Another dean raised the possibility of forming some kind of consortium where someone would be responsible for encouraging universities to update the data, or through some central mechanism ensure data currency.

Another topic highlighted periodically throughout the discussion was the limited attention to career pathways of Master’s degree recipients in this workshop. Because the data sets that do exist and the major institutional efforts that are underway focus primarily on doctoral degree recipients, the workshop gravitated toward this degree class in an effort to understand conceptual and methodological challenges of tracking career pathways of alumni. However, since the vast majority of graduate degree recipients are Master’s degree holders, university participants expressed a strong view that this exploration needs to be extended to Master’s degree holders as well.

In summary, workshop participants expressed the view that the gold standard in terms of information needed by applicants to make choices and by program faculty to improve programs would include degree program-based, comprehensive, longitudinal data that describe in some detail the career trajectories of program graduates and that illuminate, at different points in time, the graduates’ assessments of their graduate school education. All agreed that while progress is being made, we still have a long way to go in achieving the gold standard.

Suggested Readings

The following readings were included in the introductory packet of information sent in advance of the workshop:

GS Report “Understanding PhD Career Pathways for Program Improvement”

Council of Graduate Schools and Educational Testing Service. (2012). Pathways Through Graduate School and Into Careers. Report from the Commission on Pathways Through Graduate School and Into Careers. Princeton, NJ: Educational Testing Service.

Stewart, Debra W. 2013. “Tracking the Careers of Graduates: A New Agenda for Graduate Schools,” Change Magazine, pp. 41-49 | Published online: 15 Mar 2013.

Proudfoot, S.L. and T.B. Hoffer. 2016. Science and Engineering Labor Force in the US. In The Science and Technology Labor Force: The Value of Doctorate Holders and Development of Professional Careers, edited by L. Gokhberg, N. Shmatko, and L. Auriol. Cham, Switzerland: Springer.

Milesi, C., and Bartolone, J. (2016). “Locating and Abstracting Postsecondary Degree Information for the Survey of Doctorate Recipients”. Research Highlights. NORC at the University of Chicago.

Cornell University. Doctoral Alumni Career Outcomes for Degree Years 1994 – 2014.

Paper prepared for the Workshop on Graduate Degree Holder Career Pathways: Conceptual and Methodological Approaches to Measurement and Analysis held on July 25, 2017 in Washington, DC and sponsored by a grant from the Spencer Foundation. The authors thank Norman Bradburn, Karen Grigorian, Jake Bartolone, and Jill Connelly for numerous helpful comments and corrections on earlier drafts.

About The Higher Education Analytics Center

The Higher Education Analytics Center at NORC leverages our more than 50-year commitment to and extensive experience with higher education to offer effective, affordable research and data collection to institutions of higher education and other related organizations. Specifically, we deliver the data and insights necessary for these institutions’ continued functioning and improvement. We approach all work with deep technical expertise, a spirit of collaboration, and a commitment to scientific integrity. Our services range from large-scale cross-sectional and longitudinal data collection and analyses, to individualized consulting on discrete issues that can be addressed by data analytics. We also provide consultation on evaluation methods and data collection design. Learn more at heac.norc.org

About NORC

NORC at the University of Chicago is an objective and non-partisan research institution that delivers reliable data and rigorous analysis to guide critical programmatic, business, and policy decisions. Since 1941, NORC has conducted groundbreaking studies, created and applied innovative methods and tools, and advanced principles of scientific integrity and collaboration. Today, government, corporate, and nonprofit clients around the world partner with NORC to transform increasingly complex information into useful knowledge. www.norc.org

© NORC 2017