Peter Vari Director
National Institute of Public Educational Services, Center for Evaluation Studies, Hungary
Values and purposes
Current Hungarian evaluation practice is based on two elements. The first one is a professional tradition that comes from participating in several international surveys. Hungary was the first among the former socialist countries to join the international student achievement surveys, initiated by the IEA in the 1970s. Participating in these surveys gave important experience and knowledge towards establishing the evaluation culture in our country. The Monitor Surveys (the first Hungarian evaluation system) have been developed on this basis. The main features of the surveys are the following: their goal is to measure student achievement in different levels of the educational system (usually in every second year), both in the primary and secondary schools in the fields of reading literacy, mathematics literacy, science, and ICT. This survey system has existed since 1986.
The organizer of these surveys was, and still is, the Center of Evaluation Studies, founded by the Ministry of Education as one of its support institutions at the end of 1960s. The name and structure of this support institute has changed several times over the last three decades, and the number of staff has also changed (from 2 to 12 persons), but the basic tasks and aim of the work of the Centre has remained the same. The Centre now exists as a part of “Árpád Kiss”
National Institute of Public Educational Services, with a staff of 12 researchers.
Apart from the Monitor Surveys, there were and are some research projects in the field of evaluation being carried out at the pedagogical departments of certain universities. This re-search has been developed mainly for diagnostic purposes, but it could not become regular and systematic due to a lack of financial and infrastructural capacities.
International surveys were and are a very important point of reference for judging how success-ful the Hungarian educational system is in terms of international comparison. Databases and methods stemming from international surveys also provide great opportunities for analyses and evaluations on a national level.
Although Hungary has been continuously participating in surveys organized by the IEA since the early 1970s, and regularly a party to the evaluations organized by the OECD, and has had its national monitoring system for more than 15 years, there was not enough attention paid to questions regarding evaluation in the educational system, until the 1990s. There was only a small group of professionals and researchers who were interested in this field. Publications, results and problems were known by only a very small group of educational specialists.
Regular and well-developed evaluation practices had not been developed, either on different levels of the educational system (e.g. institutional level evaluation) or in different spheres of education (e.g. evaluation of educational instruments).
Before the political changes, at the beginning of the 1990s, a system of inspectors existed in the rather centralised educational system that was meant to ensure the quality of education.
Although the system was not able to satisfy this function fully, there were no efforts to develop other elements of quality assurance.
The importance and need for a new system became apparent after the changes, parallel to the abolishment of the inspection system, and with the decentralisation of the educational system.
The new documents of educational regulation and legislation formulated this need at the be-ginning of 1990s.
Educational Evaluation around the World 66
Given the significantly increased local and institutional independence in the nineties, issues concerning the quality and success of education were raised with more and more emphasis.
The external monitoring of schools declined and standardised testing was not introduced until the end of the decade.
Different investigations indicated the fall of test-based school performance, and the growth of the already present discrepancy between location and performance. Several international com-parative analyses suggested that one of the weak points of Hungarian education was quality assurance. Concerns about the quality and success of education were only increased following the modification of regulations on content and the resulting transitional conditions.
Though the Public Education Act 1993 prescribed the mandatory evaluation of public educa-tional institutions as part of the education authority’s responsibility, there was at the time insuf-ficient professional background for this task. There were no experts, instruments, or evaluation bodies, which could help education authorities, or the schools themselves, with the problem.
The Amendment of the Public Education Act in 1996 detailed the evaluation tasks of the dif-ferent levels of stakeholders, but the fact was that in everyday practice there were not many real evaluation events or initiatives, except for the national and international surveys mentioned above.
One of the reasons for this phenomena was, probably, that after the political changes, there were numerous reforms, initiatives, changes, and natural problems in other parts of the educa-tional system (for example developing the new Naeduca-tional Core Curriculum), which held back development in the field of evaluation of the whole system and everyday practice. The decen-tralisation of the educational system, the establishment of a new structure for financing and management, the new elements of administrative and control systems, and the development of the new national curriculum presented too many problems to solve in the educational sector.
But, as a result of all these processes, an intense professional and social debate evolved con-cerning the possible reasons for the inadequacy of quality and the availability of government support in the interests of quality assurance.
The quality assurance issue received special attention in the Amendment to the Public Educa-tion Act 1999. At the same time, there began a remarkable process of adaptaEduca-tion of quality assurance and evaluation tools, already in use in other countries. The private sector played an increasing role in the development of a quality assurance system for public education, a role that was consciously backed by government policies. In the quality assurance business, most of the current players are members of the professional sphere of education, with a smaller seg-ment attached to private business rather than the public service sector.
The aims of quality assurance initiatives within education are twofold: the control of quality;
the assurance of development. Apart from being familiar with the results of evaluation proc-esses, it has become essential to bring into focus the culture of evaluation, its tools and the possibilities arising from it. Besides the institutions working under the Ministry of Education and professionals who have received different levels of training, a competitive private market in evaluation work has also emerged.
Although the legal framework states the aims of evaluation for all levels of public education, as well as the experience and competence of those executing evaluation processes, by the end of the 1990s one of the primary aims of state educational policy had become the establishing of a quality assurance system at the institutional level. Nationwide assessment at this level coincided with, and already covered a certain section of the desired evaluation work. In order to further the process, the “Comenius 2000” program was established, which, through a competitive tender process and professional administration, secured the creation of a quality assurance system at the institutional level.
At the same time, professionals also responded to the growing culture of quality assurance:
pedagogical institutions began transforming their work towards a profile of assessment and evaluation, providing assessment services to schools through the organization of subject sur-veys, often on the basis of commissions arising from both the institutions and local education authorities. Experts and assessors were also trained more specifically for evaluation work.
Educational Evaluation around the World 67
According to present interpretations, control means the comparison of the actual functioning of public educational institutions with current laws, regulations and the institutions’ pedagogi-cal programs. Evaluation, on the other hand, is the comparison of the institutions’ achieve-ments with the professional aims set up by the very same institutions, their education authori-ties and the sector’s leadership. Quality assurance “is the activity whereby the public educa-tional institution continually assures that professional aims and the functioning of the institu-tion come closer to each other as a means of satisfying the needs of teachers, parents, educa-tors, education authorities and the labour market (Paragraph (5), (9) and (23) of § 121, Public Education Act 1993/LXXIX)”.
However, having accepted quality assurance as a key term in public education, Hungary has taken a significant step in the formation of views adopted by the public on the success of edu-cation. Quality assurance, as a term, is used in two senses both in and outside Hungary. Firstly, it refers to organizational and management techniques at the institutional level, and secondly, it refers to all devices and mechanisms which serve to ensure the successful operation of public education as a whole. In judging quality and success, it is especially important how professional objectives are set, as these are referred to in the evaluation phase.
Having said this, the use and interpretation of terms is still not uniform and well organised.
Similarly, there is no real consensus as to the expectations and interpretations of the various actors in the field. Development and organisation will be a challenge for public education over the coming years.
There are two basic standpoints in Hungarian education in terms of aims and values. One of them goes further than the sole evaluation of achievements in traditional curricular subjects and gives an ever-stronger emphasis to cross-curricular competence, literacy and practical knowledge. But there is also an ever-stronger emphasis on the evaluation of achievements compared to earlier achievements, the judging of development or added value, as opposed to assessing performance solely in terms of curricular requirements.
By the turn of the millennia, the basic criteria of a national assessment system had been estab-lished: a legal framework, definition of terms (even though their interpretation is not yet fully effective) and the basic professional basis for carrying out evaluation. Of course, assessment work in Hungary is still in its beginnings. The examining of developments in curricular program packages, textbooks, educational tools or other means of public education is still not resolved, not even in terms of initiatives.
But the most urgent tasks are to synchronize assessment work at the different levels and ter-rains of education; to ensure cooperation between the various agents; and to ensure the co-herence of applied tools and their maximum exploitation. Assessment at the institutional level is still limited, but it is being increasingly enhanced by the so-called National Assessment of Basic Competences, described in detail in the “case” chapter.
Reflecting on the hypothesis set up by the editors of the anthology, we can make the following statements about assessment practices in the Hungarian public education system:
The relationship between education authorities and control agencies is not yet crucial in terms of assessment as education authorities (the local council authorities) request controls in only a few cases.
The Ministry of Education, as a representative of the aims of, and values connected with, educational policies, and as the body that defines the content regulations of educational institutions, is decisive regarding the aims of the assessments it commissions and the val-ues formulated in these reports. That is to say, in Hungary, management and control are not necessarily in the same hands, and thus neither are values nor the relationships be-tween them.
Neither does the culture of any given educational sector play a decisive role in evaluation.
The “culture” of the entire public educational system is still in the making (exceptions are the national and international assessment studies and the assessments made by university workshops), and so there are no differences between the different sectors.
Institutional evaluation is also in its early years, but one can expect that the local estab-lishment of values and tools will have a decisive role here. The institution’s own
pedagogi-Educational Evaluation around the World 68
cal program, and the aims formulated in it, can influence the assessment goals and meth-ods.
In the case of Hungary, one can assume that certain values and methods are of such im-portance in the educational system that they are simply not questioned. Among them is, at the moment, the issue of key competences, and the basic need that the effectiveness and success of the overall educational system, certain forms of training and institutions, has to be measured and followed-up.
Objects
The object of the evaluation can be defined through various criteria. In accordance with the preceding comments regarding assessment practices in the Hungarian public education system, we can present the following summary:
Evaluation at a national and international level is aimed at the whole of public education and thus makes possible an evaluation of the system as a whole. Results may give us a picture of the achievements of schools in different social catchments, or the achievements of different school types. These periodically reoccurring measurements also make the analysis of trends possible.
Schools participating in a survey receive feedback in the form of “school reports” on the per-formance of the studied pupil group. These reports then serve as references for assessment at the institutional level.
In national surveys, private schools are not treated separately but as part of the nationwide sample, in so far as education within the private school/institution is pursued according to the National Core Curriculum. In Hungary, around 90 percent of public educational institutions are managed by the local authorities; only about 10 percent of schools are run by foundations or churches, as “private” schools. On the other hand, if a private school sets its curriculum accord-ing the National Core Curriculum, the school becomes eligible for state fundaccord-ing. In this way, education in most, so-called, private schools basically fits into the curriculum of the non-private system. It is only due to their better financial, personnel and infrastructural situation that they can achieve better performance. In brief, private schools do not warrant classification as a sepa-rate group in terms of assessment.
In case of the above-mentioned surveys (executed mainly by pedagogical institutions; e.g. Bu-dapest Pedagogical Institute) of curricular subjects, the object of the assessment can be clearly determined. It is according to the demands of the schools or school authorities that the knowl-edge, the performance within the given curricular subjects is measured, mostly with reference to the expectations formulated in the curriculum. Results are, of course, reported back at insti-tutional level. Should the need arise, the conductor of the survey can also provide professional advice to the school.
It is mostly in the field of competence, that measurement and evaluation of certain topics can be found. Surveys conducted in the national sample basically fall into this category as they tend not to survey curricular subject areas, but fields of efficiency and competences in a broader, more general sense: reading literacy, mathematics, science literacy.
The subject of the survey is thus basically defined: the framework of international surveys al-ways meticulously outlines it. In the case of national surveys, the first step in all cases is the definition of the research area, the aim and the method. The demands and goals of the current educational policy, the particularities of the area to be surveyed and the possibilities stemming from the size of the sample always play a role in the forming of these cornerstones. Naturally, within this framework, different approaches coexist which will further diversify the definition of the basic elements of evaluation.
Stakeholders
Nationwide surveys are usually commissioned by the political apparatus and are usually pre-pared considering the aims of national educational policy. However, the results are published to the whole of the educational community (in the form of national reports), to the whole of the nation via the press, and to the participating schools who receive tailor-made school reports. It is a matter for the school to decide how, and in what circle it makes its own results public.
Educational Evaluation around the World 69
The surveys (methods, tools, processing) and the results, on the other hand, present an impor-tant professional challenge to the assessing bodies, research institutes and researchers. To fulfil the professional aims to the highest quality is a basic task for all participants.
Surveys at an institutional level aim at the exposure of a national state of affairs. For such out-comes, they give reliable results at an institutional level, in the form of the school reports men-tioned above. There are not surveys of individual student levels, as this is not an aim of the research. It is possible, however, to evaluate individual persons at the local-level and through institutional assessments, and such evaluation is also part of everyday school practice and ex-ams.
Society as a whole and its taxpayers can also be considered as implicated in terms of the evaluations. Society’s interest in the educational system (one of the most important areas of society) is traditionally deep. Taxpayers are informed through the daily and specialized media about the results of surveys. In addition, a ranking of certain secondary-school level educational institutes is available each year as a performance indicator. These lists, however, do not rank schools according to a range of criteria but only according to performances in the higher edu-cation entry exams.
Results of national and international surveys may be published to parents and/or pupils, accord-ing to the individual decision of each school. A number of schools publish the results of surveys via their Web page, school yearbook, or school magazine. Other schools, however, on occa-sion, do not even publish the outcome amongst their own staff.
As has been mentioned before, the political and administrative sector is a standard participant in the surveys, as the agent who commissions the survey, finances it and will eventually use the outcomes. One of the principal addressees is thus the political elite.
Where the commissioner of an evaluation is an agent from another level of the government system, then it depends on their particular aims and demands as to what circle the outcomes may be published within.
Methods
Surveys are usually initiated by the Ministry of Education, as it determines the test-evaluation tasks of public education agents as set out by the Public Education Act, or through
commissioning particular national and/or international assessment works. Commissions for national and international surveys of study achievement are generally handed to the Center for Evaluation Studies of Árpád Kiss National Institute of Public Education Services (KÁOKSZI), the research workshop that has vast experience in this field. The Center is responsible for the forming of methods and tools of evaluation, for the accomplishment of the survey, the publication of the results and their follow-up.
In the case of international surveys, the Center is responsible for maintaining international
In the case of international surveys, the Center is responsible for maintaining international