Lessons in Learning

Student achievement: What should we really be measuring?

October 13, 2005

Introduction

How much and how well are our children learning in school? Do they have the skills to succeed in tomorrow’s world? Are they prepared to meet the prerequisites of pursuing further education, to face the demands of the labour market and to become active citizens of our society? Parents, students, employers, and the general public all want the answer to these questions, and governments and educators have designed a range of tools for monitoring and reporting learning outcomes and performance to measure the success of our learning systems.

This issue of Lessons in Learning looks at three critical measures of success:

  1. Student achievement in the core areas of language, mathematics and science;
  2. The disparity of student achievement observed among different groups divided according to socio-economic status; and
  3. High-school dropout rates.

What do these three measures tell us? Is it possible to achieve success on all three fronts at once? What should we do based on this information?

An understanding of a variety of measures of success is necessary for all education partners – from parents and students to government decision makers – to determine what is working and what needs improvement. Assessment of results provides an opportunity for jurisdictions to compare strategies with each other and with other countries. They are the basis for improvement in education at all levels of the system.

1) Student Achievement

Standardized tests have become an increasingly important tool to measure student achievement. They fell out of favour during the 1970s, largely due to a variety of concerns ranging from their misuse to their lack of cultural sensitivity. However, they were brought back in the 1990s, as part of an ambitious effort to address concerns that Canadian children were falling behind those of other industrial countries.

First, let’s examine the results of three large national and international testing programs that collectively measure student achievements in language skills (writing and reading), mathematics and science at key grades and age groups at the junior and senior high-school levels. These tests are:

  1. School Achievement Indicators Program (SAIP), administered by the Council of Ministers of Education, Canada (CMEC), which measures the academic performance of 13-year-old and 16-year-old students in reading, writing, science and mathematics;
  2. Trends in International Mathematics and Science Study (TIMSS) which compares the achievement of Canadian students with students in other countries; and
  3. Programme for International Student Assessment (PISA-a) (PISA-b) developed by the member countries of the OECD to provide international indicators of the skills and knowledge of 15-year-old students. PISA is administered in Canada through a partnership of the Council of Ministers of Education, Canada (CMEC), Statistics Canada and Human Resources and Skills Development Canada

It is worth noting that provincial and territorial governments also undertake a wide range of tests within their jurisdictions to help guide them in their administrative decision-making. These standardized tests enable Canada and the provinces and territories to understand how well our students are doing over time, how they measure up compared to students across Canada and how well Canadian students perform in comparison with other countries. Testing helps Canadian jurisdictions appear more accountable and provides the means for making changes directed at constant improvement in education.

What we know: Testing Programs

Language skills

When we analyze students’ tests results from all three tests, the following highlights emerge:

Among 13-year-olds in Canada: (Source: SAIP 1994, SAIP 1998, SAIP 2002)

Among 15 and 16-year-olds in Canada: (Source: SAIP 1994, SAIP 1998; SAIP 2002; PISA 2000; PISA 2003)

  • Writing also improved between 1994 and 1998. In 2002, students generally met expected levels; (SAIP 1998)
  • In reading, between 2000 and 2003, Canadian students’ performance was unchanged;
  • Girls performed consistently better than boys in writing and reading achievements. (PISA 2003) (SAIP 2002)

When compared to students from other countries, Canada is performing relatively well. (Figure 1)

  • In 2000 and 2003 only Finland performed statistically better in reading than Canada;
  • In 2003, Korea, Australia, Lichtenstein and New Zealand were at the same level as Canada;
  • In 2000 and 2003 all provinces performed on or above the OECD average.

Figure 1: Estimated average scores in Reading for provinces and countries

 (Source: PISA 2003: http://www.pisa.gc.ca/81-590-xie2004001.pdf

Science

Among 13-year-olds in Canada:

  • Improvements in science understanding and practical tasks occurred between 1996 and 1999; (SAIP 1999, page 21)
  • In 2004, the science results revealed no significant difference in achievement between boys and girls. (SAIP 2003)

Among 15 and 16-year-olds in Canada:

  • Although students’ performance in science improved between 1996 and 1999, there was a decline in 2004, with fewer students achieving the expected level; (Figure 2)
  • Students’ performance declined slightly between 2000 and 2003. (PISA 2003, page 41)

Figure 2

(Source SAIP 2004: http://www.cmec.ca/saip/science3/public/03ResultsCanada.en.pdf)

Compared to students from other countries, Canadian students do not fare as well in science relatively to their reading and mathematics performance:

  • In 1995 and 1999, students from Asian countries performed better than Canadian students; (TIMSS 1995, page 33) (TIMSS 1999, page 32)
  • In 2003, four countries performed better than Canada; (PISA)
  • Between 2000 and 2003, Canada was among five countries that observed a decline in science performance, raising questions for educators and policy makers about what would be necessary for improvement in this area. (PISA 2003, page 41)
Mathematics

Among 13-year-olds in Canada:

Among 15 and 16-year-olds in Canada:

Compared to students from other countries, Canadian students are performing relatively better in mathematics than in science:

  • In 1995 and 1999, students from Asian countries performed better than Canadian students; (TIMMS 1995, page 23) (TIMSS 1999, page 33)
  • In 2003, the only students that out-performed Canadian students in mathematics were from Finland and Hong Kong-China. Japan and Korea students were at the same level along with a cluster of other countries (Figure 3); (PISA 2003, page 15)
  • In 2003, Alberta was the only province that performed better than the Canadian average. Quebec, Ontario, Manitoba and British Columbia performed as well as the Canadian average. (PISA 2003, page 22)

Figure 3:
Estimated average scores in Mathematics for provinces and countries
 
(Source: PISA 2003 http://www.pisa.gc.ca/81-590-xie2004001.pdf; page 15)

2) Observations

The tests suggest that over the past decade Canadian students generally have performed relatively well in math, science, reading and writing. Nevertheless, the results also indicate that there are significant regional differences within Canada. For example:

  • In 2003, Alberta generally performed better than the Canadian average; (PISA 2003) (PISA 2003)
  • Students in the Atlantic provinces and Saskatchewan tend to perform below the Canadian average in most domains; (PISA 2003) (PISA 2003)
  • Outside Quebec, in those provinces where there are both English- and French-language school systems, the performance of francophone students in reading and science tends to be lower than that of students in English-language school systems. (PISA 2003)

Although Canadian students are performing relatively well on standardized tests, it is clear that there is room for improvement in the following areas:

  • Reading performance for boys
  • Math performance for girls
  • Reading and science in French minority-language settings
  • Science performance as identified by PISA and also by the failure to meet the expected level in SAIP 2004
  • Selected provinces and the Territories in most domains.

There are many other dimensions to the success of an educational system than student achievement in language, mathematics and science. Two critical ones are the system’s capacity to achieve academic excellence for all students regardless of their socio-economic circumstances, and to encourage students to complete high school.

Achievement for all

Typically, students from high socio-economic status families score higher on tests of academic achievement than students from lower socio-economic status families. OECD says that an understanding of this relationship is a useful starting point for analyzing the distribution of educational opportunities. Raising student performance and softening the impact of socio-economic background on success in education are critical objectives for education systems in all OECD countries. (OECD 2001; page 186)

Evidence from PISA tells us that, while all countries show a relationship between performance and socio-economic status, Canada displays above-average levels of student performance in reading and, at the same time, a below-average impact of socio-economic status on student performance. (OECD 2001; page 210) In other words, success is associated with education systems that seek academic excellence for all, regardless of family circumstances.

Canada is generally regarded as one of the most culturally and ethnically diverse countries in the world. This puts Canada’s strong international showing in a special light – one that demonstrates that valuing equity and achievement are complementary goals. A smaller gap in educational achievement holds the promise of greater social cohesion, as well as the promise of a population prepared to meet the demands of society and the economy in the 21st century.

3) Dropout rates

High-school completion is the basic prerequisite for moving into post-secondary education or for entering the work world, so another measure of the success of an education system is the percentage of young people who ultimately do not complete high school (the dropout rate).

Substantial progress has been made in reducing the high-school drop-out rate in Canada. The Youth in Transition Survey (YITS) - a longitudinal study from Statistics Canada and Human Resources and Skills Development Canada, shows that the national dropout rate among 18 – 20 year-olds decreased to 12 percent in 1999 from 18 percent in 1991 (Table 1).

Table 1: A Comparison of High School Dropout Rates Among 20-Year-olds, 1991 and 1999 (Percent)
  School Leavers Survey Youth in Transition Survey
1991 1999
Total Men Women Total Men Women
Canada 18 22 14 12.0 14.7 9.2
Newfoundland 24 29 19 10.5* 15.2* 5.7*
Prince Edward Island 25 33 17 16.4** 22.3** 9.2**
Nova Scotia 22 29 13* 10.1* 14.5* 5.0**
New Brunswick 20 23 16* 7.6* 11.7* 3.7**
Quebec 22 26 18* 16.0 19.9 12.0
Ontario 17 22 10* 9.5 11.2 7.8
Manitoba 19 20 18 14.8 15.7* 13.9
Saskatchewan 16 16* 16* 7.3 9.9* 4.5*
Alberta 14 16* 12* 12.5 13.8* 11.1*
British Columbia 16 17* 14* 12.9 16.7* 8.9
Source: http://www11.hrsdc.gc.ca/en/cs/sp/hrsdc/arb/publications/research/2002-000121/page06.shtml (Note: Estimates with coefficient of variation in the range of 16.6% to 33.3%, indicating a higher level of measurement error, are marked with a single asterisk (*). Two asterisks (**) identify estimates for which the CV exceeds 33.3%.)

Among provinces, New Brunswick, Newfoundland and Labrador, Saskatchewan and Nova Scotia experienced the largest reductions in the dropout rate between 1991 and 1999. The smallest reductions over the same time were in Manitoba, Alberta and British Columbia. In addition, the long-term prospects for young people who do not finish high school are dismal. It is important to gain a better understanding of why dropout rates remain high in some regions, in specific communities and for young men in particular.

The Implications for Decision-making

The Challenge of Pursuing Success on Many Fronts

In recent years, authorities responsible for education in Canada have made considerable progress in preparing benchmarks for desired student achievement. However, much less progress has been made in assessing how well these results are being achieved in areas other than language, science and mathematics. For example, concern is often expressed about low levels of student knowledge of Canadian history, citizenship, poor physical fitness and lack of critical thinking. Moreover, little attention has been paid to measuring performance in these areas.

Ideally, all valued goals of schooling should be measured and monitored. In practice, we have chosen to measure only a narrow range (at least in part because these are the easiest domains to measure and compare with jurisdictions and countries). Critics argue that this practice leads to excessive emphasis being placed on the few school subjects – math, language skills and science – that are measured. A frequent, unintended result is that other valued outcomes such as the creative arts and civic engagement are given lower priority, resulting in cutbacks to their funding or to the time allocated to them in the school curriculum.

Some interesting observations can be made by comparing the results of the student achievement levels with drop-out rates and overall levels of achievement. Pursuing many goals at the same time presents challenges for our educational system. For example, in the 2000 cycle of PISA testing, Alberta, Quebec and British Columbia all performed above the Canadian average in science, math, and reading. However, Alberta and British Columbia made less progress than other provinces in reducing the high-school dropout rate through the 1990s. Dropout rates were also high in Quebec, where one in five boys does not complete high school.

In New Brunswick, while smaller percentages of students achieved high scores in the PISA reading, math or science tests, progress was made in reducing dropout rates to levels that are among the lowest in the country for both boys and girls.

This comparison raises the following questions:

  • At what point do efforts to raise academic standards result in a higher dropout rate among weaker students?
  • To what extent do efforts that focus on increasing the high-school retention rate come at the expense of overall achievement levels?
  • What steps do we need to take to ensure high school graduation with high achievement for all students?

Ideally, schools would meet all of the goals expected of them. In a context of competing demands and fixed resources, this is a challenge. The role of parents, the public, and governments is to understand what these challenges are and to decide what the priorities should be, both at a system-wide level and at the level of individual schools.

4) Now what?

Where We Go From Here

International and domestic comparisons are valuable, but it is essential to place them in context. Different jurisdictions can have different priorities, different education systems, cultures and educational values, and different strategies for achieving success. Nonetheless, it is still possible to learn from the experience and success of others.

Tests can contribute to positive change when the results are used to identify areas for improvement – whether in terms of curriculum, the special needs of individual schools or other factors – and when these areas are addressed with policies and practices designed to improve performance. Moreover, testing in any subject is productive only as a means to develop improvement strategies that are focused on achievable goals to increase student success.

In Canada, one common goal is high levels of secondary school completion with high achievement for all. The challenge is to identify the most effective tools for achieving that goal. We should be proud of our achievements – and we should learn from them too.

Success stories can be found at all levels of the education system. By sharing information across provinces, across school boards and districts, and across schools, we can learn how innovative approaches developed in all parts of the country can lead to improved performance.

Systematic efforts are under way nationally and within individual provinces and territories to establish priorities for education and to determine the scope of the outcomes to be measured. Having the appropriate data on student performance in a wide range of subjects is therefore valuable.

Nevertheless, it is important to recognize that schools, educators, and governments are not the only essential ingredients of success in our education systems. Parents, communities and employers also play vitally important roles.

All elements and sectors of Canadian society have an interest in working collectively to improve the educational success of our young people. The higher the level of literacy and numeracy that students achieve while in school, the greater the opportunities that await them as adults.

We need to understand better all of the factors that contribute to strong learning outcomes. What policies and practices make for effective schools? Does class-size matter? What is the appropriate role for technology in the classroom? What role can parents play in helping their children to achieve their best? What is the role of early childhood education in preparing children for school? How can we provide better educational opportunities for aboriginal children? What needs to be done to reduce the gender gap? How can communities and business contribute to achievement in individual schools?

These are all important questions which CCL will address in future Lessons in Learning.

5) Useful Links

Programme for International Student Assessment, www.pisa.gc.ca
Statistics Canada, www.statcan.ca
Council of Ministers of Education, Canada, www.cmec.ca
Human Resources and Skills Development Canada, www.hrsdc.gc.ca


Who pursues postsecondary education, who leaves and why: Results from the Youth in Transition Survey http://www.statcan.ca/english/research/81-595-MIE/81-595-MIE2004026.pdf

Education and labour market pathways of young Canadians between age 20 and 22: an overview http://www.statcan.ca:8096/bsolc/english/bsolc?catno=81-595-M2004018

In and out of high school: First results from the second cycle of the Youth in Transition Survey, 2002 http://www.statcan.ca:8096/bsolc/english/bsolc?catno=81-595-M2004014

At a Crossroads: First results for the 18 to 20-Year-Old Cohort of the Youth in Transition Survey (2002). Ottawa: Human Resources Development Canada and Statistics Canada. http://www11.hrsdc.gc.ca/en/cs/sp/hrsdc/arb/publications/research/2002-000121/page09.shtml

Measuring up: The performance of Canada’s youth in reading, mathematics and science. OECD PISA Study – First Results for Canadians aged 15 (2001). Ottawa: Human Resources Development Canada, Statistics Canada, and Council of Ministers of Education, Canada. http://www.pisa.gc.ca/pisa/81-590-xpe.pdf

Measuring up: Canadian Results of the OECD PISA Study. The Performance of Canada’s Youth in Mathematics, Reading, Science and Problem Solving, 2003 First Findings for Canadians Aged 15. Ottawa: Human Resources and Skills Development Canada, Statistics Canada, and Council of Ministers of Education, Canada. http://www.cmec.ca/pisa/2003/indexe.stm

Science in Canadian Schools 2004. SAIP Science III 2004 Highlights. (2005) Council of Ministers of Education, Canada. http://www.cmec.ca/saip/science3/public/highlights.en.pdf

A reference to all three tests:

(i) SAIP: Since 1993, the Council of Ministers of Education, Canada (CMEC) has measured the academic performance of 13-year-old and 16-year-old students in reading, writing, science and mathematics through the School Achievement Indicators Program (SAIP). Three cycles of the tests were administered: the first between 1993 and 1996; the second between 1997 and 1999; the third cycle began in 2001 with mathematics and was completed in June 2004 with science.
(ii) TIMSS: Trends in International Mathematics and Science Study: In an era of global competitiveness, governments and educators have conducted several studies, including the Trends in International Mathematics and Science Study (TIMSS), comparing the achievement of Canadian students with students in other countries. TIMMS was developed by the International Association for the Evaluation of Educational Achievement and administered by the provincial ministries and departments of education. Three TIMSS reports have been written: 1995, 1999 and 2003 (although only Quebec and Ontario participated in 2003).
(iii) PISA: Programme for International Student Assessment: The member countries of the OECD initiated the Programme for International Student Assessment in 2000 to provide international indicators of the skills and knowledge of 15-year-old students. Three PISA cycles, every three years, have been planned, each focusing on a different literacy domain. In 2000, the major focus was reading literacy, with mathematical and scientific literacy as minor domains. In 2003, the major focus was mathematical literacy and in 2006, scientific literacy will be the focus. The cycle will be repeated, starting again in 2009. In Canada, about 28,000 students from more than 1,000 schools participated. Additional tables and charts are available to support this documentation at the following websites:www.pisa.gc.ca, www.statcan.ca, www.cmec.ca et www.hrsdc.gc.ca.