Richard M. Jones
Associate Director, Assessment & Reporting
Education Quality and Accountability Office (EQAO), Ontario
Canada is a confederation of ten provinces and three territories. Areas of responsibility are divided between the federal and provincial/territorial governments. The federal level is respon-sible for portfolios such as national defense, external affairs, fisheries, shipping and railways, telecommunications, energy and the banking and monetary systems. The provinces and territo-ries are responsible for areas such as social services, health, forestry, highways and education.
Hence, there is no federal department of education. This means that each provincial/territorial jurisdiction establishes its own policies related to aspects such as curriculum, teacher certifica-tion, school accreditation and reporting of student learning progress. School boards set their local policies within the framework of policies established by the provincial/territorial ministries of education. These policies include aspects such as school operations, curriculum implementa-tion, and the hiring of teachers and support staff. Decisions concerning instruction and class-room practices are made at the school and classclass-room level. Since there are thirteen provin-cial/territorial jurisdictions in Canada, this article attempts to summarize education and evalua-tion systems across all of these jurisdicevalua-tions.
Public education is provided free to all Canadian citizens and permanent residents until the end of secondary school usually age eighteen. The ages of compulsory education vary from one jurisdiction to another, but generally, schooling is mandatory from age six or seven to age six-teen.
Canada’s public education systems are divided into two levels: elementary and secondary. Most jurisdictions offer a one-year kindergarten program for children prior to grade one. Some school systems offer a two–year program of junior and senior kindergarten. Children aged three to six are in these programs. Elementary education is subdivided into the primary level, including grades one to three, the junior level, including grades four to six and the intermediate level, including grades seven and eight or grades seven to ten. Secondary education is generally subdivided into junior high school, including grades seven to nine and senior high school, in-cluding grades ten to twelve (grade eleven in the province of Quebec). In some systems, secon-dary school begins at grade seven or eight; in others elementary school continues until grade eight.
Educational evaluation – an overview
Student assessment in Canadian schools utilizes a broad range of methods to obtain informa-tion about educainforma-tional achievement for formative and summative purposes. Most prov-inces/territories have established systems of formal assessments that involve program assess-ment (providing information about student achieveassess-ment in key grades and subject areas to education stakeholders for accountability purposes and/or providing information to teachers, students and parents for the purposes of improving student learning and achievement) and/or credentialing examinations required for high school graduation.
At the national level, the provincial/territorial ministers of education have established an organi-zation called the Council of Ministers of Education, Canada (CMEC) to provide a forum for communication on issues of common concern, such as education funding, education programs and assessment. In the early 1990s, the CMEC established a national testing program called the School Achievement Indicators Program (SAIP) in response to the ministers’ need for
compara-Educational Evaluation around the World 44
tive information about student achievement in the areas of reading, writing, mathematics and science. The assessment, developed and coordinated by a consortium of provinces, usually involves the administration of one annual assessment in one of the subjects previously men-tioned. The assessment is administered to randomly selected samples of schools and 13 and 16-year-old students. Information on student achievement is reported only at the provin-cial/territorial and national levels. Because of the sampling framework, no achievement infor-mation is available at the individual student, school or school board levels. Further inforinfor-mation about the SAIP is available on the CMEC Web site (www.cmec.ca).
From time to time, Canada and/or individual provinces/territories participate in international assessments, such as the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS) conducted under the auspices of the International Association for the Evaluation of Educational Achievement (IEA), as well as the Programme for International Student Assessment (PISA) conducted by the Organisation for Economic Co-operation and Development (OECD). When the country or one or more of its jurisdictions becomes involved in an international assessment, the project is coordinated within Canada by the jurisdictions themselves, through the CMEC, or through another Pan-Canadian organization such as Statistics Canada.
When individual jurisdictions conduct evaluations, or when national testing is administered through the CMEC, the jurisdictions’ Education Acts form the legal or statutory basis for this work. Generally, provincial/territorial Education Acts make school participation mandatory.
Values and purposes
In their summary of the purposes of centralized student testing across Canada, Taylor and Tubianosa (2001) found that the main purposes were as follows:
to monitor the education system;
to identify strengths and weaknesses;
to certify and promote;
to provide information for curriculum design, instructional methodology and resource allocation;
to gauge change/progress over time;
to give direction for further research.
To make some determination about the quality or health of the education system, it is neces-sary to have appropriate indicators/measures of attainment. Some performance indicators such as retention, participation and graduation rates are often available internally to jurisdictions’
ministries of education. However, other comparative measures, such as student achievement information, are seen as important additional elements that help educators and the public make decisions about system quality and effectiveness.
Evaluation has the potential to provide information about areas of strength and weakness in student achievement. This type of information can be used to inform future directions for pro-gram planning at the jurisdictional and local levels, as well as implementation and/or modifica-tion to classroom instrucmodifica-tional practices. Program evaluamodifica-tions, based on the intended curricu-lum, can provide such information in an objective fashion.
Examinations are sometimes used for the purposes of certification/credentialing and promotion.
If developed and administered appropriately, they can ensure a level of fairness in the evalua-tion of students. High stakes tests such as this can be used to place students in programs, classes and schools, based on their achievement, and they can indicate whether a student will be promoted to the next grade or whether a student will graduate from high school.
Information from program evaluations is an important reference for persons such as curriculum developers as they plan for curriculum review and revision. This type of information is also in-valuable to those who develop instructional resource materials for classroom use.
Educational Evaluation around the World 45
In order to determine whether student achievement is improving over time, and whether changes to curriculum and other educational reforms are having the desired effect, it is impor-tant to have access to objective information about performance trends. The administration of well-designed, centralized assessments can be the source of such information.
Educational researchers are interested in studying how students learn most effectively and what factors/variables have an impact on, or are associated with, student achievement and attitudes.
Secondary analysis of results from centralized program assessments can provide a useful source of data for this type of work.
Areas of consensus and concern
Generally speaking, there is a consensus that educational evaluation, if designed and adminis-tered appropriately, has the potential to provide important information related to the foregoing purposes. However, there are those (primarily in the teaching profession, but also some in the broader community) who are concerned about issues related to the impact on the curriculum, test construction, and the misuse of results and other unintended consequences.
Some critics of large-scale student assessment claim that, because of the restrictions imposed by paper-and-pen testing formats, only certain knowledge and skills can be assessed. Another concern is that if teachers teach to the test, then some important curricular elements may be neglected. Others are concerned that since the focus of many testing programs is on so-called basic skill areas, such as reading, writing, mathematics and science, other important subjects that are not usually tested may be considered of lesser importance in the school program.
Many large-scale testing programs, including most provincial/territorial learning assessments, credentialing programs and national and international assessments, are comprised of a variety of test item formats to gather information on a wide range of student knowledge and skills.
Some critics, however, contend that assessments that rely heavily on multiple-choice items result in an emphasis on lower-order thinking skills. In addition, there is a concern that if tests are inappropriately constructed, test bias, where items are culture, language, or gender spe-cific, may result in unfairness or advantage to certain groups of students. Other criticisms of large-scale student evaluation relate to the misuse of data, particularly the inappropriate rank-ing of school results, the unintended consequences of testrank-ing, such as stress and anxiety on the part of students and teachers, and the impact of results on grade retention and dropout rates.
Objects
Assessment programs
All of the provinces (except Prince Edward Island) use formal, large-scale assessments to obtain systematic information about student achievement. Most provincial assessment programs in-volve some form of program/learning assessment, as well as credentialing examinations. How-ever, there are differences in various aspects, such as the subject areas and grade levels tested, the schedule of administration and design elements, including whether all students (a census) or a random sample of students are assessed. The following table, adapted from Taylor and Tubianosa (2001), summarizes the general features of assessment programs among Canadian jurisdictions.
Educational Evaluation around the World 46
Provincial Assessment Programs
Province Subject(s) Grade(s) Design Schedule
Alberta Provincial Assessments
Educational Evaluation around the World 47
In addition to these provincial assessment and credentialing examination programs, some juris-dictions have implemented other initiatives to support public accountability and educational system improvement. For example, for many years the province of British Columbia conducted a system of school accreditation in which members of the school community and an external evaluation team reported on the effectiveness (strengths and weaknesses) of various elements associated with the school. This program operated on a mandatory six-year cycle. However, recently the province abandoned school accreditation in favour of a school district review proc-ess.
Recognizing that effective school and school board planning are important components of the improvement process, Ontario mandates that all school boards in the province submit annual improvement plans to the province’s Education Quality and Accountability Office (EQAO), the
Educational Evaluation around the World 48
agency that administers all provincial, national and international assessments within the prov-ince. School board improvement plans are reviewed according to specific criteria, and reports are provided back to boards to inform them of the extent to which their improvement plans conform to the required content.
In addition, most Canadian provinces/territories have implemented education quality indicators programs. The programs all vary to some extent, but in general, data and information that have some relevance to education and student achievement are collected from a variety of sources (e.g., surveys, Statistics Canada, Ministry of Education and other government departments) and are reported in a variety of ways. Generally speaking, the data provide contextual information that assists education stakeholders in their interpretation of student achievement results. Some examples of education indicators are socio-economic information; perceptions of school cli-mate/school safety; perceptions of school leadership; and home learning supports, such as access to a computer, quiet study space, and access to books in the home. In some instances, education indicators are reported in their own separate reports. In other cases, education indi-cators are released concurrently with student achievement data.
Stakeholders
The preceding table indicates that the vast majority of assessment programs and all of the cre-dentialing examinations involve all-student or census testing. This means that in most cases, individual student reports are received/available to students, parents/guardians, and teachers.
(In some instances, school administrators and guidance counselors may also see individual stu-dent results.)
In the great majority of cases, schools and school boards receive summaries of their own re-sults, and summaries of provincial achievement results are reported by means of print and/or Web-based documents to the public at large, the education community, the Ministry of Educa-tion (including curriculum developers) and the media.
Generally, evaluation results are viewed as useful for education planning and improvement purposes, as well as for public accountability. There are always concerns expressed by some about the potential for inappropriate uses of data, such as rank-ordering schools and school boards, and using results to evaluate teachers.
Methods
Policy Design and Implementation
Within the provinces and territories, ministries or departments of education have taken respon-sibility for developing student assessment policy for both assessment in the classroom and for more formal large-scale provincial student assessments. In some cases, ministries have estab-lished detailed assessment policy frameworks, including quite detailed guiding principles. In other cases, ministries and school boards have less formal assessment policies but promote the use of fair testing practices, e.g. through quoting documents such as the Principles of Fair Stu-dent Assessment Practices for Education in Canada (1993 and 2000).
According to Perry Fagan and Spurrell (1995), in their study of policies and practices of minis-tries and school boards across the country:
“The extent of the jurisdictions’ responsibility for student evaluation, policy design and implementation varied depending on the type of evaluation or assessment covered by the policy. For example, there appeared to be a much more specific set of policies for provincial examinations than for other types of evaluation. In five of the provinces with provincial examinations, the department of education is responsible for both design and implementation of policy. In two provinces, the department and the boards are responsible, and in the two territories, the exami-nations are the responsibility of the department of two other provinces. In other areas of assessment policy, the involvement of others in the system was much more evident. Some provinces indicated that the department and boards were
Educational Evaluation around the World 49
equally involved in policy development and design, and others indicated that the department, boards, and schools were all involved. In some cases, where there were no provincial examinations, ministry officials indicated that only boards, or schools, or both, were responsible for assessment policy….All but one of the 12 jurisdictions indicated that they had methods in place for ensuring that policies, practices, and guidelines were implemented.” (p.16)
The authors identified nine general approaches that the jurisdictions used to implement policy, and these included:
creating minister’s orders supplemented by policy guidelines;
publishing documents, handbooks and providing guidelines;
conducting on-site visits and reviews;
providing in-service sessions, workshops and seminars;
providing information and reporting results of provincial examinations and assessments;
forming specialist implementation teams for new policies;
assigning coordinators for administering examinations;
holding centralized marking and training sessions;
collecting data and reviewing results from school evaluations;
reviewing school board plans for improvement.
Approach to assessment
Generally, with regard to provincial/territorial assessments, the respective
minis-tries/departments of education are responsible for the evaluations and their methods. Most ministries/departments have student assessment branches whose mandate is to design, admin-ister and report on the assessments. Some ministries rely on external experts to conduct various aspects of the assessments (e.g. item and test development, statistical analysis); however, ulti-mate responsibility for the assessment remains with the ministry. One jurisdiction (Ontario) has created an office for education quality and accountability to assume responsibility for the de-sign and conduct of provincial assessment initiatives.
Again, generally speaking, provincial assessments are criterion-referenced and are based on the learning objectives and standards outlined in the provincial curriculum. While evaluation ap-proaches are usually more quantitative than qualitative, they are, nevertheless, increasingly becoming both results- and process-oriented.
There is usually a close correspondence between the values (the reasons and purposes of the evaluation) and the method (the way the evaluations are planned and implemented). For ex-ample, in Ontario, two guiding principles of the work of the Education Quality and Account-ability Office are to provide information for public accountAccount-ability purposes, as well as for the improvement of student learning. Consequently, it is not enough to simply provide overall scores in various subject areas for individual schools, school boards and the province. Informa-tion on strengths and weaknesses is also provided to individual students so that they, their teachers and parents can work together to help students improve.
In all cases, the results of provincial learning assessments are made public through various means such as press releases and public reports. The extent of and approach to follow-up ac-tivities related to the evaluations, however, is variable. In some cases, specific recommendations are not directed to stakeholders, and once assessment results are publicly released, there is little formal follow-up to determine what, if any, action has been taken. In other cases, specific rec-ommendations are made to specific stakeholders, and steps are taken to follow up to deter-mine what has been done toward improvement of the system. For example, in Ontario, the annual, provincial reports of the assessments of Grades 3, 6, 9 and 10 provide specific recom-mendations to stakeholder groups; letters are sent to these organizations, highlighting the recommendations that are specific to them; and, through various means (e.g. discussions at
Educational Evaluation around the World 50
advisory committee meetings, review of school board improvement plans), indications are ob-tained about the extent to which recommendations are being attended to.
Past and future
In recent years, there have been some general changes in Canadian large-scale assessment practices. Over the past decade, assessment and evaluation programs have become the focal point of many provincial educational reform agendas. Many Canadian jurisdictions (e.g. On-tario) have either established new assessment programs or expanded or modified existing ones.
In describing recent changes in evaluation across Canada, Perry Fagan and Spurrell (1995) ob-serve that there has been a general trend, both in classroom and large-scale assessment, to-ward integrating curriculum, instruction and evaluation to ensure that assessment closely matches learning objectives. This is evidenced by most jurisdictions’ reliance on criterion- and curriculum-referenced assessment as opposed to norm-referenced testing.
Although most large-scale assessment programs (and also most classroom assessments) con-tinue to rely heavily on paper-and-pen testing approaches, there has been a trend in recent years toward looking for ways to move away from a focus on paper-based assessment. Evi-dence of this is in practical/performance-based assessment that goes beyond extended written responses to demonstrations of skills acquisition. Current education and assessment theory
Although most large-scale assessment programs (and also most classroom assessments) con-tinue to rely heavily on paper-and-pen testing approaches, there has been a trend in recent years toward looking for ways to move away from a focus on paper-based assessment. Evi-dence of this is in practical/performance-based assessment that goes beyond extended written responses to demonstrations of skills acquisition. Current education and assessment theory