Summaries of the winning projects from the 2018 Awards
Trinity College London & Lancaster University
In 2017, Trinity College London (TCL) funded a team at Lancaster University to conduct a collaborative research project to explore the feasibility of delivering existing paper-and-pencil reading and writing sections of the Integrated Skills in English (ISE) examination suite online. The project was commissioned in response to changing stakeholder needs for online testing across global educational markets. The resulting project investigated 283 test-takers who each completed reading and writing tasks both in paper-based and online mode. Testtakers also completed a questionnaire to gauge perceptions of the impact, usability and fairness of the delivery modes. Many-facet Rasch measurement (MFRM) analysis of scores revealed that delivery mode had no discernible effect, apart from the reading-to-writing task at ISE Level I, where the paper-based mode was slightly easier. Test-takers generally held more positive perceptions of the online delivery mode, although technical problems were reported. The findings of the study have implications for TCL’s decision to deliver the ISE in an online format, which will have impact on thousands of test-takers worldwide. The study’s findings also have theoretical implications within the field of language assessment, with results providing insight into the relationship between performance, task type and mode of delivery.
Best Transformational Project
Vretta Assessment-for-Learning Program
In Ontario (Canada), as elsewhere, research has shown that too large a proportion of students enter college and university lacking adequate levels of numeracy. The traditional response on the part of colleges has been to require incoming students to take a (written) placement test aimed at streaming students with a weak numeracy background to take an additional mathematics course, both delaying entry to their desired program and costing both them and the college financially. The Vretta Assessment-for-Learning Program (VALP), developed collaboratively with the colleges, turns this approach on its head. By replacing written tests with e-assessments, it enables assessments to be personalized and interactive. More important, it enables students to receive immediate diagnostic feedback and direct access to upgrading e-learning modules. The test is no longer a hurdle in students’ way but a support for their self-improvement, saving them both time and money. For teachers, it raises the general level of numeracy of all students while also flagging topics likely to require special attention in class. For colleges, there are financial savings, through not having to offer remedial courses, as well as institutional benefits from increased student success.
Best Use of Formative Assessment
Vretta Inc. and Luxembourg Ministry of Education
Countries are witnessing an ever-increasing cultural heterogeneity which is proving to be instrumental for the success of their economies by attracting pools of diverse talent from around the world. However, this has also led to Education Ministries facing the growing challenge of responding to a high proportion of students who do not speak the language of instruction at home, in turn affecting attainment in numeracy and literacy. The Luxembourg Ministry Education, in partnership with Vretta, the French Ministry of Education and leading educational testing and socio-economic research institutions, developed a large-scale formative assessment platform called MathemaTIC to alleviate the multilingual challenge and foster a highly interactive learning experience. MathemaTIC diagnoses the skills of 21st Century learners and formatively determines their sequence of learning, providing a truly personalised learning experience. It enables dynamic switching between languages, supporting both numeracy and language growth and enabling effective teacher and parent intervention. It also enables effectively monitoring and supporting through real-time data visualizations. The successful implementation of MathemaTIC in Luxembourg has enabled the expansion of the use of the platform and interactive assessment items in Portugal and France, while other countries are in the process of reviewing and customizing the platform for their use.
Best Use of Summative Assessment
The International Baccalaureate (IB) offers four educational programmes. Assessment for the Middle Years Programme (MYP), aimed at 11 – 16 year olds, has been transformed into an e-Assessment service which includes on-screen examinations in four subject areas and a unique examination testing interdisciplinary skills. The examinations include the use of multimedia to engage students in realistic scenarios and interactive tools to allow students to make predictions, take measurements and to problem solve. Due to these features, a single examination provides an efficient assessment of the broad range of knowledge, skills and concepts to reflect the breadth of each subject’s assessment objectives. The examinations are unique in that questions are open and they assess deep understanding through using of interactive tools and multimedia. They do not require a live internet connection and automatically cater for the needs of the majority of students with inclusive access arrangements. On-screen examinations offer substantial improvements over the previously used moderation system. Whilst moderation allowed the IB to adjust the assessment standards of schools, on-screen examinations now allow the IB to set and maintain the standard of problems set and for marks awarded. They also allow the IB to sample the breadth of the programme fairly.
Excellence in Export
The Nautical Institute
As the professional body for people in control of ships, The Nautical Institute provides a number of qualifications to help operate ships safely and effectively. This includes in the specialist field of Dynamic Positioning, used in high-risk areas such as offshore oil and gas exploration. With 4,000 tests delivered every year across 30 countries, the Institute has 20,000 people currently qualified and employed in this field. By the nature of their work, participants are travelling all over the world, presenting a unique export assessment challenge. The goal was to establish a robust and repeatable e-assessment process run through a worldwide network of 92 training centres, which would be controlled from headquarters in London, where centralized records are held. TestReach provided a stable and secure solution that met all technical requirements. The successful implementation strategy included a 60-day live testing evaluation, deployment of subject matter experts, education of test centres, phased launching of different exams and integration with the Institute’s database. More than 80% of the project is conducted outside of the UK, and is valued at £50,000 over initial 2 project phases. Cultural challenges included adapting to language barriers, and educating training centres on the benefits of e-assessment.
Innovate Awarding - EPA Pro
EPA Pro is Innovate Awarding’s specially designed end-point assessment system which has been developed to make the administration of the new assessment process as simple as possible. EPA Pro will guide the provider from registration to assessment booking, results and certification. It also enables assessment scheduling, whilst providing access to support materials and resources that are designed for the apprentice to effectively prepare for assessment. A bespoke system that can integrate with key assessment technologies, EPA Pro allows the EPAO to manage the assessment workforce ensuring timely delivery of the EPA nationally. EPA Pro is developed in partnership with PearlTech using an agile development to ensure we fit with the changing regulatory landscape and regular customer feedback. EPA Pro is designed to ensure the full end-point assessment journey can be managed and monitored by all stakeholders in a web-based live system. EPA Pro is truly bespoke, designed specifically for the complex end-point assessment process and available for new entrants into the EPA market. Through streamlining the assessment administration process and providing a one-stop shop for all EPA preparation and assessment, we believe the system will support the apprentice and allow them the best opportunity for success in their assessment journey.
Lifetime Contribution 2018
Angela Verschoor is Senior Researcher at Cito, the Netherlands. With a background in discrete optimization. Her interest is the development and application of, amongst other things, computerized adaptive testing (CAT). Since 1995, Angela has been playing a key role in the development and deployment of CAT-systems, such as the MathCAT for entrance into teachers colleges in the Netherlands (see van der Linden, W. and Glas, C., Elements of Adaptive Testing, pp. 137 – 149) and the Mindsteps for formative testing in Grades 3 to 9 in Northwestern Switzerland (see www.mindsteps.ch). She is currently working on the development of the adaptive algorithm powering the Welsh National Reading and Numeracy Tests, which are formative adaptive tests for school learners from year 2 to year 9. Other recent projects included the introduction of ATA and CAT in, amongst others, Georgia, Russia, Kazakhstan and the Philippines.
Angela takes an active role in sharing her expertise with the global assessment community, presenting at a range of international conferences including IACAT and AEA-Europe. She has also published a number of academic papers in conjunction with her Cito colleagues.
In 2010 Angela was accredited as a Fellow of the AEA-Europe.
Angela's contribution to such a wide range of operational on-screen adaptive test systems, demonstrates that the approach can be successfully deployed in a wide range of subjects, languages and cultural setting.