CCL Home > Reports & Data > Lessons in Learning
Full article
Lessons in Learning: Homework helps, but not always
Find more...
CCL news in your inbox: The Learning Link
Despite Canada’s strong performance on international assessments of mathematical skill among 15-year-olds, many Canadian students have weak math skills and struggle in their math classes. It has been suggested that classroom strategies fostering exploration and discovery, and guiding students to construct their own learning and knowledge can be effective in ensuring that all students acquire strong mathematical skills. However, a recent review of evidence conducted by CCL suggests that such an approach has modest effects on student achievement.
The development of mathematical skills is crucial for contemporary social and labour market success, according to the Organisation for Economic Co-operation and Development (OECD). “The objectives of personal fulfilment, employment and full participation in society increasingly require that all adults should be mathematically, scientifically and technologically literate.”[1]
Canadian results from the 2006 Programme for International Student Assessment (PISA) show Canadian students successfully meeting this important outcome.[2] Canada ranks second of 17 countries and receives an ‘A’ from the Conference Board of Canada.[3] However, the proportion of Canadians with low-level math skills, scoring at or below Level 1 on the PISA assessment, actually increased slightly from 10.1% in 2003 to 10.8% in 2006 (see Pisa numeracy levels text box below). In the same cycle, Finland, the top-ranking nation, reduced its proportion of students with low-level math skills from 6.8% to 6.0%,[4] a result the Conference Board attributes to a Finnish educational initiative designed to improve mathematics and science outcomes.
PISA numeracy levels
Students performing below Level 1 (mathematics score below 359) are not able to show routinely the most basic type of knowledge and skills that PISA seeks to measure.
At Level 1 students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined.
At Level 2 students can employ basic formulae and procedures. They are capable of direct reasoning and of making literal interpretations of the results.
At Level 3 students can select and apply simple problem-solving strategies. Students at this level can interpret and use diagrams, graphical displays, symbolic expressions and other representations based on different information sources and reason directly from them.
At Level 4 students can select and integrate different diagrams, graphical displays, symbolic expressions and other representations, linking them directly to aspects of real-world situations. Students at this level can utilize well-developed skills and reason flexibly, with some insight, in these contexts.
At Level 5 students can develop and work with models for complex situations, identifying constraints and specifying assumptions. Students at this level can work strategically using broad, well-developed thinking and reasoning skills, appropriate linked representations, symbolic and formal characterisations, and insight pertaining to these situations.
At Level 6 students can apply insight and understanding along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for solving problems in novel situations.
Source: Organisation for Economic Co-operation and Development, Programme for International Student Assessment, PISA, 2003.
Further details available at http://www.statcan.gc.ca/pub/81-004-x/def/4153348-eng.htm
The primary years provide the foundation for lifelong learning and are “critical in establishing relationships among a child’s intuition, mathematics and the real world.”[5] The future mathematic success of Canada’s students depends on successful early educational experiences.
Some researchers and educators have argued that developmentally appropriate practice (DAP), a teaching approach in which children construct their own learning and knowledge through guided interactions with their physical and social environments, holds promise as a foundation for early learning.[6] According to DAP, the teacher’s role is to facilitate learning, extend activities, challenge students and emphasize critical thinking, problem-solving and co-operation.[7] In short, the teacher is “a dispenser of occasions” rather than a dispenser of knowledge.[8] DAP is designed to capitalize on children’s intrinsic drive to understand their own experiences through exploration and experimentation.[9]
DAP classrooms are meant to attend to children’s overall development, including cognitive, social and physical aspects. DAP classrooms also integrate learning across multiple subjects, and focus on the process of learning so that students “learn to learn.”[10]
Some critics consider DAP non-academic and lacking in structure. “A program can be judged to be high quality despite the fact that it only marginally provides what young children need in order to become literate.”[11] Others suggest DAP may not be appropriate for all children or all age groups.[12] DAP is sometimes unfavourably contrasted with the explicit teaching of skills typified by direct instruction.[13]
To make sense of the debate concerning the effectiveness of developmentally appropriate practices, the Canadian Council on Learning recently conducted a systematic review and meta-analysis of empirical literature examining DAP effects on the mathematics achievement of students in kindergarten through third grade. Results synthesize the best evidence currently available.
After querying 13 databases and 16 websites for literature published between 1990 and 2007, 37 articles and dissertations were eventually deemed eligible for review and meta-analysis.
While the review originally intended to study DAP, seven categories were eventually imposed on classroom strategies examined in the 37 articles. These included DAP as a whole and six variables that could be considered individual components of DAP: computer use, manipulative use, multiple strategy instruction, individualized instruction, student selection of activities and positive teacher–student interactions. Below are brief explanations of each individual component.
For each outcome reported in each study, an effect size was calculated. Effect size is a measure of the degree to which classroom strategies affect student outcomes (see textbox for further details on interpreting effect sizes. From these individual effect sizes in each study, an average effect size was calculated for each of the seven categories of independent variables.
Effect sizes can be positive or negative. In this case, positive effects indicate that DAP strategies are associated with improved student outcomes, while negative effects indicate deleterious effects on student outcomes.
Figure 1 illustrates the average effect size produced by treatments in each of the seven categories. Overall, the comprehensive set of DAP practices produced the largest average effect size. The individual interventions all produced noticeably smaller effect sizes. Allowing students to select their own activities produced the smallest average effect; using learning strategies produced the largest, other than DAP. The essential implication is that a coherent set of practices produces a larger effect than any single individual practice. A number of caveats temper this conclusion, however, and it is useful to examine each category more carefully.
Figure 1: Average effect sizes produced by meta-analysis of each category of variables
The DAP studies compared classrooms described as ‘high DAP’ and ‘low DAP.’ Because DAP is a continuum rather than a dichotomy, this distinction is at best subjective, and arguably unsound.[15] As well, the average DAP effect size is based on only eight individual effect sizes (see Figure 2). Further analysis revealed substantial variation in effect size magnitude among these eight outcomes. Statistical tests indicate that the calculated average effect size of .5 may not be replicable in future meta-analyses of DAP effects containing different studies. In sum, while DAP effects appear to be real, and non-trivial, we cannot safely assume they will be of similar magnitudes across classrooms.
An effect size (d) provides a standard measure of the magnitude of study outcomes across different studies, different outcomes and different interventions. The use of a standard measure has become extremely helpful to researchers and practitioners in their attempts to compare the results of different studies.
Despite their widespread utility, there is no standard approach to interpreting effect sizes. Cohen has suggested the following rule of thumb: effect size d = .2 should be considered small, d = .5 is medium, and d = .8 is large.[i] In educational research, d = .2 is generally considered small, d = .4 is considered medium, and d = .6 is large.[ii]
The magnitude of an effect is not the only consideration in deciding whether to implement an intervention. Other considerations apply such as the cost of implementing the intervention, the number of students who would benefit, and the extent to which the intervention competes with other interventions or other aspects of the curriculum.
For example, for an intervention that has very low implementation costs, from which most students are expected to benefit and which does not require teachers to sacrifice any or very much of their teaching time, an effect size d = .1 for an intervention might be considered large enough to proceed. In contrast, it might be impractical to implement a costly and time consuming intervention with a much larger effect size.
[i ] J. Cohen, Statistical Power Analysis for the Behavioral Sciences (2nd ed.). (Hillsdale, NJ: Erlbaum, 1988).
[ii] J. Hattie, Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement. (New York, NY: Routledge, 2009).
Computer and technology use, manipulative use and learning strategies all produce effect sizes that further analyses indicate cannot be assumed replicable if new meta-analyses were conducted on different studies. In other words, the effects of these pedagogical techniques vary widely. This is not surprising. It is far more likely that the application a student uses with her computer has a greater effect than the mere use of his or her computer.
Similarly, the use of specific appropriate learning strategies is more important than the use of learning strategies in general. Indeed, we found treatments featuring teacher-led learning strategy instruction to be the most effective. Overall, there is reason to suppose learning strategy instruction will have some benefits. Although the effects of this practice vary, 28 outcomes contributed to the calculated average effect size, which was the largest for any single intervention (see Figure 2). On balance, learning strategy instruction can be assumed to be valuable.
Teachers’ roles figure importantly in the manipulatives meta-analysis. Teachers’comfort level with and appropriate implementation of manipulatives strongly mediate manipulatives’ effects.
Figure 2: Number of effect sizes calculated for each category of variables
The effects associated with individualized information, positive interactions and student-selected activities represent stable average effect sizes. Our analyses indicate these results would likely be replicable across different studies examining the same topic. Likely there is not as much variation within the practical manifestations of a category like ‘interacting positively with students’ as with ‘using computers.’ However, our analyses do show that relatively few studies with contradictory results would be needed to nullify the effects of student-selected activities, which produced the smallest average effect size.
Overall, the average effect sizes of developmentally appropriate practices are modest and sometimes erratic, but also statistically significant and positive.
None of the seven independent variables produced large effect sizes in the meta-analyses: the largest effect size was produced by the comprehensive set of DAP practices. Student achievement appears to be most effectively enhanced when they work in coherent environments of multiple appropriate instructional practices.
Most modern research on learning recognizes that children learn in different ways and at different rates. Pre-service teacher education often stresses using multiple modes of instructional delivery, and mathematics curricula in most jurisdictions recognizes this fact. Our results indicate this eclectic approach is appropriate, as no one instructional technique produced an overwhelmingly positive increase in student achievement, yet all seven factors appeared to have some significant positive impact. And notably, individualized instruction displayed a slightly stronger and more stable average effect than most other single interventions.
While computer literacy may be a worthy goal in itself, our evidence does not suggest computer use is the best way to enhance mathematics performance. Depending on the cost of technology, decision-makers may want to consider whether their resources would be better directed to ensuring children are learning in an environment rich in the other factors conducive to achievement. Creating welcoming physical surroundings staffed by teachers well educated in developmentally appropriate techniques, within class sizes appropriate to facilitate such practices may represent an equal or better way to increase student achievement than heavy investments in technology.
The modest and varying effects of the instructional strategies reported here suggest other factors must be responsible for creating student success. One likely explanation is the variations among students and teachers. As the National Association for the Education of Young Children articulates “what the teacher does is paramount.”[16] Student achievement likely depends more on the degree to which adequate resources are directed toward capacity and knowledge building in teachers, and in attracting appropriate candidates to the profession, than on the degree to which one instructional approach is chosen over another.
Overall, there appears to be little risk in adopting the instructional techniques identified here. The effect sizes of each of the seven factors are generally modest but positive, and on balance, children’s mathematics achievement is not impeded by these techniques. Teachers appear to have little to lose by employing them. This implication is made in light of our findings that:
Indeed, the main implication may be that DAP practices (or lack of them) as operationalized in our studies, are simply not the most important factor in primary-school mathematics achievement.
[1] Organisation for Economic Co-operation and Development (OECD), PISA 2006: Science Competencies for Tomorrow’s World, Volume 1: Analysis (Paris: 2007).
[2] P. Bussière, T. Knighton & D. Pennock, Measuring up: Canadian Results of the OECD PISA Study. The Performance of Canada’s Youth in Science, Reading and Mathematics, (Ottawa: Statistics Canada, 2007). Catalogue no. 81-590-XIE No. 3.
[3] Conference Board of Canada, Education and Skills: Students with Low Level Math Skills (Ottawa: 2009).
[4] OECD, PISA 2006: Science Competencies for Tomorrow’s World, Volume 1: Analysis.
[5] British Columbia Ministry of Education, The primary program: a framework for teaching (Victoria: 2000)
[6] S. Bredekamp & C. Copple, (Eds), Developmentally Appropriate Practice in Early Childhood Programs, National Association for the Education of Young Children, (Washington, DC).
[7] R. Novick, "Actual schools, possible practices: New directions in professional development," Educational Policy Analysis Archives, 4 (14), (1996).
[8] C. B. Phillips, "The hundred languages of children," Young Children, 49(1), (1993).
[9] R. Novick, "Actual schools, possible practices: New directions in professional development."
[10] M. L. Van Horn, E. O. Karlin, S. L. Ramey, J. Aldridge & S. W. Snyder, "Effects of developmentally appropriate practices on children's development: A review of research and discussion of methodological and analytic issues," Elementary School Journal, 105 (4), (Ann Arbor, Michigan: 2005).
[11] D. Dickinson, "Shifting images of developmentally appropriate practice as seen through different lenses," Educational Researcher 31, (Washington, DC: 2002).
[12] S. Lubeck, "Is developmentally appropriate practice for everyone?" Childhood Education, 74 (5), (1998).
[13] W.C. Becker & S.Engelmann, "Systems for basic instruction: Theory and applications," in A.C. Catania & T.A. Brigham (Eds.), Handbook of Applied Behavior Analysis: Social and Instructional Processes (New York: Irvington, 1978).
[14] Ontario Ministry of Education, The Ontario Curriculum Grades 1-8: Mathematics, (Toronto: 2005).
[15] Van Horn, Karlin, Ramey, Aldridge & Snyder, "Effects of developmentally appropriate practices on children's development."
[16] National Association for the Education of Young Children (NAEYC), “Key messages of the position statement on developmentally appropriate practice,” (Washington, DC: NAEYC, 2008).