Metrics of Diversity: An International Framework

With adoption of Transforming our World: the 2030 Agenda for Sustainable Development, the U.N. issued a global call for action to increase and equalize access to higher education.  One hundred and ninety-three member nations committed to the following principle: “We commit to providing inclusive and equitable quality education at all levels – early childhood, primary, secondary, tertiary, technical and vocational training.   All people, irrespective of sex, age, race, ethnicity, and persons with disabilities, migrants, indigenous peoples, children and youth, especially those in vulnerable situations, should have access to life-long learning opportunities that help them acquire the knowledge and skills needed to exploit opportunities and to participate fully in society.”   This affirmation provides a broad, cross-national understanding of the major dimensions along which equity and inclusiveness should be measured and pursued; because the statement pays particular attention to the full and equal participation of women in all aspects of society, virtually all signatories have set social equity goals related to the full educational participation of women.  Within and across universities and nation-states, graduate educators are deeply engaged in efforts to increase the diversity of the graduate student population and the inclusivity of the academic programs that prepare them.  Indeed, even in countries such as the U.S. that have not signed the Sustainable Development Agenda, these efforts are well underway.  Initiatives are three-pronged – focused on promoting equitable access to masters and doctoral education, ensuring that program curricula reflect a diversity of perspectives, texts, and methodologies, and providing the support that students from different backgrounds and walks of life need to succeed in graduate study and they reflect two fundamental commitments.   The first is a moral commitment to eliminate systems of inequality that have historically privileged some groups of citizens at the expense of others.  In this regard, it is worth noting that higher education, and specifically postgraduate education, continues to reflect these historic inequalities at the same time it is being called upon as an important tool for dismantling them.  Second, relying on the growing body of research that demonstrates the inextricable link between the diversity of teams and the quality of scholarly outcomes (Page, 2007 and 2017; McKinsey & Company, 2014),  universities have sought to improve the quality of postgraduate education and the scholarship it produces by increasing faculty and student diversity.

Despite the attention and effort it has received, progress in increasing diversity of representation, inclusiveness of the curriculum, and graduate degree completion has been uneven.  For example, in the United States, there has been a growing representation of LatinX students but decreasing participation of Native Americans (Okahana and Zhou 2018).  In some areas of Africa, Ethiopia for example, women are significantly under-represented on faculties and in tertiary education and this inequality is further exacerbated for students living in rural areas (Asfaw 2012; Tamrat 2018).  While students from historically underrepresented groups continue to be admitted to postgraduate study in disproportionately low numbers in virtually all fields of study, there has been even less attention paid to the types of inclusive practices, necessary to close the degree completion gap.  As a result, it is only recently that many universities and graduate faculty/mentors have begun to implement organization practices designed to accept, welcome, value, and equally treat individuals from the full array of groups and backgrounds.  This includes efforts to decolonize the curriculum and to provide culturally aware mentoring and student support services (see for example, Chaka, Lephalala, and Ngesi 2108). 

However, given the current lack of 1) a standard diversity and inclusion taxonomy, 2)agreed upon metrics for measuring diversity and inclusion in a postgraduate education context, and 3) a set of widely shared methodological approaches to documenting progress towards meeting diversity and inclusiveness goals, it is difficult for universities, regions/states, or nations to hold themselves accountable and develop data-informed policies and practices that might help them reach their goals more quickly.

This paper provides a general framework for defining and measuring progress towards reaching diversity and inclusion goals, doing so in a way that recognizes the importance of local or regional contexts but also allows for some level of cross-national benchmarking.

Why Measure?  What Statistical Approach?

There are three main reasons to develop and use an agreed upon set of metrics to measure diversity and inclusion:  the ability to 1) set and articulate social equity goals, such as those outlined in the UN Sustainable Development Goals (SDGs)adopted in 2015 by 193  countries;  2) measure progress towards meeting them at university, national, and global levels, and; 3) improve, and amplify the effects of,  programs designed to promote diversity and inclusion.  Each intended use requires a slightly different analytic approach, even though they all may rely on the same or very similar metrics.

Setting and Articulating Goals:  National and Local Contexts

As discussed above, the SDGs provide one common cross-national diversity referent.  However, the concept of locally defined minorities (LDM) provides an important ancillary construct for setting diversity goals.  As introduced by ECHO, LDM highlights the extent to which underrepresentation is historically, politically, and geographically situated (Tupan-Wenno, Camilleri, Frohlich, and King 2016; see also www.echo-net.nl).  While gender diversity occupies a near universal position in diversity goals, the concept of LDMs points to the national and regional contexts that give priority emphases to some of the other sociodemographic dimensions over others.  So for example, in societies that may historically have been rather homogeneous but that are now embracing a growing number of refugees, the inclusion of migrants may be of primary concern and focus. In nations such as Australia, New Zealand, Canada, and the U.S. with histories of often violent dislocation/displacement of large aboriginal populations by Western Europeans colonists, reconciliation with, and social equity for, indigenous populations may be paramount.  In nations such as South Africa, where one of the legacies of apartheid has been the segregation of many black South Africans to rurally isolated parts of the country, social equity targets may be particularly attentive to the intersectionalities of race, ethnicity, gender, and rural place of residence.

In articulating and setting social equity goals, the core underlying assumption for virtually all analyses is  that in the absence of significant historical, political, or geographic barriers, individuals from across the full spectrum of social categories and groups will be represented in education and occupations, proportional to their representation in the general population.  Identification of which populations to include in social equity goals and the metrics and methods used to identify those who belong to them, vary across different geopolitical contexts.  So for example, in nations such as France and Germany, where religion and race, respectively are not administratively recognized/recorded, citizenship status often stands as a proxy for the ethnic and other cultural differences that in other nations might be more directly used to set diversity goals.  However, regardless of the actual indicator used, the basic analytic approach of measuring equitable participation with respect to representation in the general population at large[1] does not vary; as discussed below, this commonality makes some cross-national comparisons possible (and meaningful) even when specific metrics vary.

Measuring Progress:  Representative Diversity and Inclusiveness

There are two basic approaches to establishing benchmarks against which to measure progress on diversity goals:  cross-sectional comparisons with a designated set of peers and/or over time comparisons, within a single entity, be it university, locale, or nation.  The former approach situates progress towards achieving diversity and inclusion goals in the context of the most and least successful organizations of similar scope and mission.  It ignores the intra-institutional changes that can alter the trajectory of diversity efforts and at least hypothetically raises the questions:  If some organizations are doing better, could we?  Despite our challenges and setbacks, have our efforts still born more fruit than those of other comparators?  Comparisons over time within a university (or nation), on the other hand, allows an organizational entity to judge how successfully it is meeting its targets within the time frame established by its strategic plan; importantly, it can do so with reference to the local population, the timing of policy/program initiatives intended to foster diversity and can ask and answer questions about  how successful these intervention have been.  Both strategies are important and certainly can be used in conjunction with one another, but it is the latter strategy that is most important for improving diversity- and inclusion-related programs and initiatives.

 Program Improvement and Culture Change

Finally, the ability to use data to evaluate program effectiveness and to catalyze change in admissions, mentorship or curricular practices requires “local” evidence on diversity, pre- and post- program or policy implementation.  To be relevant to faculty and administrators, data must have the power to persuade them that diversity and inclusion challenges are theirs and not just someone else’s.  The faculty whose responsibility it is to design the graduate curriculum, provide postgraduate supervision and mentorship, and guide the writing of theses and dissertations must be able to see the experiences of their students and their program in the data that is provided.  This is true with respect to efforts/programs that are designed to improve access to, and thus the representational diversity of, postgraduate studies but is especially true for efforts to improve the inclusivity of the curriculum, program climate, and the quality of mentorship for all.  The power of the status quo resides, in part, in the ability of individual mentors, program faculty, or the university writ large to depersonalize the experience of individuals from underrepresented groups and to ascribe responsibility for their limited access and differential success to factors beyond their control.  Local data, collected at the lowest possible denominators make results relevant to the local context.  Given privacy and confidentiality concerns, some level of aggregation, across lab and research groups, programs, departments, or disciplines is likely to be required, particularly when documenting student outcomes, such as degree completion.  Nevertheless, the more granular the analysis can be, the more likely it is to catalyze program, policy, and process improvement.

A Note on Inclusiveness

To date, most research and policy has focus on representational diversity and access to graduate education, usually as a first important step towards promoting equity.  While scholars do distinguish between diversity and inclusion and note the importance of the latter (for example, see Claeys-Kulik and Jorgensen 2018), little attention has thus far been paid to either setting inclusiveness goals or to developing the metrics that would be appropriate for measuring them. However, the two intended and distal outcomes of inclusiveness suggest a possible measurement approach.  Those objectives are 1) better educational outcomes, specifically degree completion, for those who have been historically under-represented in graduate education, and 2) more robust and creative science and scholarship.  With respect to the first of these, some universities have made progress in monitoring degree completion rates (Council of Graduate Schools 2013; Sowell et al. 2008).  It is possible, in turn, to use these data to measure degree completion gaps and progress, over time, in closing them.  So, for example, some research indicates the existence of racial and gender differences in degree completion rates that could be indicative of less than welcoming academic practices and culture (Okahana et al. 2018; Sowell et al. 2008; Sowell, Allum, and Okahana 2015).  Even though the particular statuses that are most relevant will vary across national contexts, to the extent that one major purpose of a more inclusive environment and curriculum is the reduction/elimination of differential educational outcomes, then the focus on the gap between “majority” and “under-represented” students is meaningful, no matter how the outcome and those groups are identified and defined.

Strategies such as decolonizing the curriculum, mentor training, workshops on implicit bias, microaggressions, or overcoming imposter syndrome are all intended to improve inclusiveness.  However, very few universities have developed measurement strategies to evaluate their efficacy.  One potentially promising approach is the use of campus climate surveys.  (See, for example, Texas A&M University Graduate Campus Climate Study, the University of California Graduate Student Well-being Survey, and Association for American Universities (AAU) Climate Survey on Sexual Assault and Sexual Misconduct.)  Data from these surveys may help to raise awareness of program climate issues that are adversely affecting students from underrepresented groups.  In addition, these surveys may provide evidence of the more proximal impact of inclusiveness efforts; such evidence would be useful in improving those policies and practices.   However, given the paucity of literature, their effectiveness in this regard is still largely unknown (Coalition of Urban Serving Universities, Association of Public & Land-Grant Universities, and American Association of Medical Colleges, 2016) and it is unclear if they could ever be designed in such a way as to provide data useful locally and for cross-national comparison. 

A second desired outcome of diversity and inclusiveness is the positive impact it has on creativity, knowledge production, and problem solving.  Indeed, many universities in otherwise relatively homogeneous regions or countries view increasing international students as a central piece of their diversity efforts (see for example, Abdelfattah 2018; Chye 2018; Matei 2018; Motala 2018;  Ramadan 2018; Schröder and Geldsetzer 2018)  Because of the complexity of measuring these positive outcomes, it is quite likely that universities and other policy makers will simply need to rely on the strong empirical evidence, generated by Scott Page and others (for example, see Bourke, Garr, Van Berkel, and Wong;2017; Dillon and Bourke, 2013; Fitzpatrick and Sharma 2017) to make this case for the importance of diversity and inclusion.

[1] In some instance, particularly where the category of interest is continuous rather than binary, socioeconomic status or parental education, for example, comparisons may be between participation rates of those occupying the least privileged position on the continuum vis-à-vis those occupying the most privileged position