CCL Home > Reports & Data > Systematic Reviews
The main purpose of this report is to systematically review the research evidence devoted to measuring quality in post-secondary education. Of the 1,859 studies initially collected, 39 studies eventually met the criteria for inclusion and analysis. A coding matrix was developed based on Finnie and Usher (2005) to categorize the results of the included studies and to allow cross-study comparisons. This matrix permitted us to organize the indicators used to capture the post-secondary education experience as a process of inputs and outputs.
During the review and analysis process, the studies were divided into three categories. If they provided simple numeric counts of indicators (e.g., graduation rates, high-school GPAs, and number of transfer students) without analyzing or critically reviewing the indicators, they were identified as descriptive studies. If they provided critical analysis of the indicators and/or measurements of quality but did not test their findings, they were considered analytical/critical studies. If they described an issue or program and then analyzed it using rigorous methodologies and statistical methods of inference, they were deemed empirical studies. In addition, a scoring rubric was created to present our criteria for determining the rigour of the research in terms of methodology. We examined the content, data collection, sampling strategies, statistical analysis and validity evidence provided.
The studies in each category were further classified according to the four approaches identified in Bogue’s (1998) framework of quality measurement: traditional peer review, the assessment and outcome movement, Total Quality Management, and accountability and indicator reporting. This framework, in combination with the rubric, provided an overview of all the reviewed articles. Based on the findings, we made recommendations on methods of conducting quality measurement in post-secondary education and highlighted relevant findings from exemplary articles.
After summarizing this information, we performed an analysis of trends by identifying the indicators most frequently used in different types of studies and theoretical approaches. In addition, we determined which measurement tools were most often used.
With respect to the group of empirical studies, the usage of key indicators was more evenly distributed than in the purely descriptive and critical studies. This result is clearly attributable to the discrepancies in sample sizes of our three groups of studies, with the empirical studies making up the bulk of studies (N=22). In addition to the sample size issue, since the empirical studies were also those studies scoring in the highest quality category (see Section 4.0), the only trends that could be extracted for further consideration were those identified by the empirical studies of the highest quality. In aiming to establish such trends using the data at hand, the following key conclusions may be formulated with regards to quality measurement in PSE based on the robust empirical studies.
In terms of the limitations of this review, it is acknowledged that the number of articles using empirical research to directly determine the actual “quality of quality measures” was rather small. There may be several reasons for this scarcity. First, there is very little sound and solid validity evidence in the field of educational research and measurement; it often takes researchers years to accumulate evidence supporting the validity of their own measures, or the established measures they use, in their particular educational context. Second, this evidence is often gathered in a piecemeal fashion, and, therefore, it is difficult to provide definitive statements about the “quality of a measure of quality” at any one time.
The primary strength of this review is the comprehensive, yet focused, literature search. This systematic search enabled us to obtain a relatively large sample size of actual articles for full review; a total of 39 articles were evaluated and synthesized. The comprehensive capture result is related to the review’s second strength: the included articles cover a wide range of topics pertaining to quality in higher education. The included literature also demonstrates a variety of approaches to the assessment of quality in PSE. We have synthesized information from purely descriptive literature, giving broad insight into the type and extent of quality indicators currently in use. Evaluating these articles before the critical and empirical articles provided us with a necessary understanding of the current use of quality indicators and a sound conceptual basis for synthesizing the information provided in the more rigorous critical and empirical articles.
View the full report (676 KB)