Irish Presidency Conference Better Assessment and Evaluation to Improve Teaching and Learning
19-20 March 2013
The Use of ICT for the Assessment of Key Competences Christine Redecker Yves Punie
IPTS- Who we are
IPTS: Part of Joint Research Centre of the EC: 7 Research Institutes across Europe Mission: “to provide customer-driven support to the EU policy-making process by developing science-based responses to policy challenges that have both a socio-economic as well as a scientific/technological dimension”
IPTS Studies on
ICT for Learning
Future Learning Spaces in 2020 eLearning in the EU10 MS - Digital Competence Review ICT - Active Ageing & Learning Impact Learning 2.0 in formal E&T Learning 2.0 in informal settings Creativity & Innovation in schools in EU The Future of Learning Tell-Net Teacher Networking
2006
2007
2008
2009
2010
2011
2012
2013
ICT for the Assessment of Key Competences Digital Competence
DC Follow up
Mainstreaming Creative Classrooms 1to1 Learning initiatives in Europe Mapping Technologies for Learning Open Education in Europe
Where we stand 1995
2000
2005
Generation 1
Generation 2
Automated administration and/or scoring of conventional tests
Adaptive Testing
2010
2015
2020
Generation 3
2025
Personalised learning
Generation 4
Continuous Data Mining & Analysis Personalised integrated feedback and Behavioural tracking assessment tutoring Learning Analytics
Multiple Choice Short Answer
Automated Feedback Replication of complex real life situations within a test
Intelligent Tutors
Generation Re-invention Transformative strategy: use technology to Games change test formats Virtual laboratories Simulations
Computer-Based Assessment (CBA)
Virtual Worlds Online collaboration Peer assessment
Learning 2.0 Collaborative multimedia learning environments Technology-enhanced learning
Bunderson, Inouye & Olsen (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational measurement (Third ed., pp. 367-407). New York: Macmillan Bennett, R. E. (2010). Technology for Large-Scale Assessment. In P. Peterson, E. Baker & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 8, pp. 48-55). ford:
Efficient testing
1990
Current use of ICT in assessment in schools
There is scope for improvement Central recommendations on the use of ICT in pupil assessment in compulsory education in primary and general secondary education (ISCED 1, 2, 3), 2009/10 Source: Eurydice (2011). Key Data on Learning and Innovation through ICT at School in Europe 2011, Education, Audiovisual and Culture Executive Agency.
Examples
1st Generation CBA: Problem-Solving
http://www.worldclassarena.org/v5/flash/09_y ear_old/backtofront.swf World Class Arena Sample Test: Problem Solving (9 years)
Examples Generation Re-Invention: Problem-Solving
http://www.worldclassarena.org/v5/flash/13 _year_old/pyramids.swf Problem Solving 13 years
Examples
1st Generation CBA: Situational Judgement
EPSO sample situational judgement test questions; http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm
EPSO sample assessment centre e-tray exercise http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm
Examples
Generation Re-Invention: e-tray exercise
Future eAssessment Strategies 1995
2000
2005
Generation 1
Generation 2
Automated administration and/or scoring of conventional tests
Adaptive Testing
2010
2015
2020
2025
Data Mining & Analysis Behavioural tracking Learning Analytics
Multiple Choice Automated Feedback Intelligent Tutors
Generation Re-invention Transformative strategy: use technology to change test formats Computer-Based Assessment (CBA)
Generation 3 Continuous integrated assessment
Generation 4
Short Answer
Replication of complex real life situations within a test
Personalised learning
Games
Virtual laboratories Simulations
Personalised feedback and tutoring
Virtual Worlds Online collaboration Peer assessment
Learning 2.0 Collaborative multimedia learning environments Technology-enhanced learning
Bunderson, Inouye & Olsen (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational measurement (Third ed., pp. 367-407). New York: Macmillan Bennett, R. E. (2010). Technology for Large-Scale Assessment. In P. Peterson, E. Baker & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 8, pp. 48-55). ford:
Efficient testing
1990
Examples Towards Generation 3: Datamining Signals http://www.itap.purdue.edu/tlt/sig nals/signals_final/index.htm
Similar: GPS at Northern Arizona University http://resourceconnect.nau.edu/gr ade-performance-status.asp
Examples Towards Generation 3: Behaviour Tracking Example: SNAPP SNAPP is a software tool that allows users to visualize the network of interactions resulting from discussion forum posts and replies. Teachers can rapidly identify patterns of user behaviour. SNAPP can extract all user interactions from various commercial and open source learning management systems, such as BlackBoard and Moodle.
Examples Towards Gen 4: Intelligent Tutoring Systems AutoTutor https://www.youtube.com/watch?v=aPcoZPj L2G8
GnuTutor (http://gnututor.com/) OpenSource version of AutoTutor https://www.youtube.com/watch?v=k6HDE EnYuIA&feature=player_embedded
Technological trends 1995
2000
Generation 1
2005
2010
2015
2020
2025
Generation 2
Adaptive Testing Automated administration and/or scoring of conventional mainstream usetests for high stakes testing
- further development of innovative test formats - increased ease of use at school level - trend towards automated scoring → cultural change: increased acceptance of CBA
Personalised learning
Generation 3
Bunderson, Inouye & Olsen (1989). The four generations of computerized educational measurement. In R. L. Linn (Ed.), Educational measurement (Third ed., pp. 367-407). New York: Macmillan Bennett, R. E. (2010). Technology for Large-Scale Assessment. In P. Peterson, E. Baker & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., Vol. 8, pp. 48-55). ford:
Efficient testing
1990
Continuous integrated Multimedia collaborative learning assessment
environments support key competence Generation 4 development. - Peer and portfolio
Personalised feedback and assessment tutoring evolve
naturally.
-Data mining and analysis will make integrated assessment and tutoring possible in the future. We are in a transitional period: Moving from the “Testing Paradigm” to the (future)Re-invention “Tutoring Paradigm” in which explicit Generation Learning 2.0 testing could become obsolete Replication of complex real life situations within a test Computer-Based Assessment (CBA)
Collaborative multimedia learning environments Technology-enhanced learning
Assessing
Key Competences
Key Questions 1. Which ICT tools, formats, environments, devices etc. are most suitable for the assessment of which Key Competences?
2. How can ICT be used to support competence-based assessment strategies that adequately capture the skills and attitudes dimensions of all eight Key Competences and their transversal nature?
Intelligent Tutors etc.
Communication in the mother tongue
D
2
Communication in foreign languages
D
3a
Mathematical competence
D
3b
Basic competences in science and technology
4
Digital competence
5
Learning to learn
6
Social and civic competences
FS
7
Sense of initiative and entrepreneurship
FS
8
Cultural awareness and expression
S
FDS
FDS
FS
1
S
Simulations
Virtual worlds games
Self Assessment
FD
Peer Assessment
S
ePortfolios
Quizzes / simple games
F = Formative Assessment D = Diagnostic Assessment S = Summative Assessment
CBA
Focus Areas
FS FD FS
FDS FDS
FS
FS
FDS
FS
•PART 1 •INSPIRATION
Mother Tongue Literacy 1 Communication in the mother tongue
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
Matching spoken and written language
Foreign language learning CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
http://www.nickjr.de/games/107-swipers-buchstabier-buch
2 Communication in foreign languages
Mathematical games 3a
Mathematical competence
CBA (DS)
Quizzes / simple games (FD)
Simulations (FDS)
Inteligent tutoring systems (FDS)
Mathematical games using simulation for secondary school education Source: http://hotmath.com/games.html.
Peer Assessment for Collaborative Work 5 Learning to Learn
Example: SparkPlus http://spark.uts.edu.au/
CBA (F)
Peer Assessment (FS)
Self Assessment (FS)
Peer Assessment using moodle www.moodle.org
Moodle is a free web application that educators can use to create effective online learning sites.
•
Moodle Statistics Registered sites: 77,213 Countries: 229 Courses: 7,116,466 Users: 66,349,750 Teachers: 1,292,731 Enrolments: 56,306,274 Forum posts: 118,413,272 Resources: 63,891,942 Quiz questions: 147,057,434
http://www.youtube.com/watch?v=Tp8v2UBbmN4
ePortfolios 1 Communication in the mother tongue
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
2 Communication in foreign languages
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
8 Cultural Awareness and expression
http://sites.google.com/site/eportfolioapps/overview/model-k-12
ePortfolios
ePortfolios (FS)
Virtual worlds / games (FS)
ePortfolios 1 Communication in the mother tongue
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
2 Communication in foreign languages
CBA (DS)
Quizzes / simple games (FD)
ePortfolios (FS)
8 Cultural Awareness and expression
ePortfolios (FS)
Virtual worlds / games (FS)
http://sites.google.com/site/eportfolioapps/overview/levels
Level 1 - ePortfolio as Storage Collection: Creating the Digital Archive (regularly – weekly/monthly) A Focus on Contents & Digital Conversion Level 2 - ePortfolio as Workspace Collection + Reflection: Immediate Reflection on Learning & Artifacts in Collection (regularly) A Focus on Process & Documentation of Learning
Level 3 - ePortfolio as Showcase Selection/Reflection/Direction (each semester? End of year?) A Focus on Product & Documentation of Achievement → A (formative/summative) assessment approach on the basis of artefacts → Focus on production and reflection
ePortfolios meet Virtual reality 8
Cultural Awareness and expression
ePortfolios (FS)
Virtual worlds / games (FS)
A game approach to arts
http://pheon.org; http://pheon.org/intro.html;
Pheon is a multimedia scavenger hunt that tests players’ ingenuity through exploring the collection of the Smithsonian American Art Museum, making
Science: Molecular Workbench 3b
Basic competences in science and technology
CBA (S)
Virtual Worlds / games (FS)
Simulations (FDS)
http://mw2.concord.org/public/part2/diffat/page3.cml
Simulations for mathematics 3a
Mathematical competence
CBA (DS)
Quizzes / simple games (FD)
Simulations (FDS)
Inteligent tutoring systems (FDS)
Sample task from US future New State Tests (2014/15)
http://www.k12center.org/rsc/pdf/a-seachange-in-assessment-letter-size.pdf
Cognitive Tutor (Carnegie) 3a
Mathematical competence
CBA (DS)
http://www.youtube.com/watch?v=RR4HPrGlUz 8&feature=related
Quizzes / simple games (FD)
Simulations (FDS)
Inteligent tutoring systems (FDS)
Science: Quest Atlantis 3b
Basic competences in science and technology
Example: Quest Alantis Quest Atlantis (QA) is an international learning and teaching project that uses a 3D multi-user environment to immerse children, ages 9-16, in educational tasks. QA combines strategies used in the commercial gaming environment with lessons from educational research on learning and motivation.
CBA (S)
Virtual Worlds / games (FS)
Simulations (FDS)
Citizenship 6 Social and civic competences
CBA (DS)
Peer/Self Assessment (FDS)
Virtual Games / Simulations (FDS)
Global Conflicts http://www.globalconflicts.eu/
Example: Child labour in Bangladesh: http://www.youtube.com/wat ch?v=mf9Ju86IVYw
Educational game, designed to help teach concepts in citizenship, geography, and media. Developed by Serious Games International, it has detailed lesson plans and assignments for students.
Examples
•PART 2
•CRITICAL INSPECTION
CBA for Digital Competence 4
Digital competence
CBA (S)
Simulations (FDS)
Digital Competence → Focus on knowledge
Internet and Communication Core Certification (IC3) exam sample question
4 Digital competence CBA (S)
Digital Competence: IC3 → Focus on operational skills
Internet and Communication Core Certification (IC3) exam sample question http://www.certiport.com/Portal/common/htmllibrary/SalesVideo-Demos/demo_ic3.html
CBA for Digital Competence Simulations (FDS)
Entrepreneurship Questionnaire 7 Sense of initiative and entrepreneurship
CBA (DS)
Peer/Self Assessment (FDS)
Virtual Games / Simulations (FDS)
Entrepreneurship Questionnaire → Focus on personality/disposition as a static fact
? https://www.careeringahead. com.au/survey/Entrepreneurs hip/Generic/questions.asp
Conclusions ICT offers powerful, engaging assessment formats, in particular: CBA
Quizzes / simple games
ePortfolios
Basic literacy (foreign/mother tongue)
Creative writing (foreign/mother tongue)
Basic mathematical competence
Cultural awareness and expression
Caveat:
Self / Peer Assessment Learning to Learn
Simulations
Virtual worlds/ games
Scientific competence Digital competence
Social and civic competences
Educational software Mathematical competence
CBA lends itself to simplified, knowledge-based assessment → Pedagogy comes first! Careful selection of question formats
ICT Assessment approaches are underdeveloped for: Learning to Learn
Sense of initiative and entrepreneurship
Social and civic competences
The Paradigm Shift Era of Computer-Based Assessment (CBA)
Era of Technology-enhanced learning
Focus on: Efficient testing
1990
1995
CBA (Gen.1)
Focus on: Personalised learning
2000
Adaptive Tests (Gen.2)
2005
2010
2015
2020
2025
Technology Automatic Scoring
Gen. ReInvention
Learning 2.0
Immersive Learning
Learning Analytics
Intelligent Tutors
Pedagogy Efficient Testing
Validity of Scores
Quick Feedback
Authentic Contexts
?
→ Primacy of Pedagogy: Pedagogical guidelines for intergrated assessment
?
?
?
• Thank you very much for your attention!
• Christine Redecker • Yves Punie •
[email protected] •
[email protected]
http://ipts.jrc.ec.europa.eu/public ations/pub.cfm?id=5719
further information …… http://is.jrc.ec.europa.eu
Newsletters:
http://is.jrc.es/pages/newsletter.html
Publications:
http://is.jrc.ec.europa.eu/pages/Publications.html
Contact:
[email protected]