Realizeit at the University of Central Florida Results from initial trials of Realizeit at the University of Central Florida, Fall 2014
1
Based on the research of: Dr. Charles D. Dziuban, Director
[email protected] Dr. Patsy D. Moskal, Associate Director
[email protected] Research Initiative for Teaching Effectiveness University of Central Florida
Written by: Dr. Colm Howlin, Principal Researcher
[email protected] Realizeit Learning
Disclaimer: This document presents Realizeit’s interpretation of the results presented by the researchers. The University of Central Florida does not endorse or recommend any commercial products, processes, or services.
Table of Contents 02
Introduction
03
Effect on Student Outcomes
09
Student Perceptions
15
Conclusions
2
Introduction This document summarizes the results of a research study carried out by Dr. Charles D. Dziuban (Director) and Dr. Patsy D. Moskal (Associate Director) of the Research Initiative for Teaching Effectiveness at the University of Central Florida (UCF). The study was conducted during an initial trial of the Realizeit system in two Fall 2014 courses and was designed to provide an initial gauge of the effectiveness and impact of Realizeit on student outcomes. In addition to effectiveness, the UCF research team also measured student’s perceptions of instruction and Realizeit through the use of surveys.
gauge the effectiveness and impact of Realizeit on student learning
The study followed two courses: one in the field of Psychology (General Psychology), and one in the field of Nursing (Physiospathology). To allow the effectiveness of Realizeit to be measured and compared, the courses were run utilizing three different instructional models. For each course, one group of students used Realizeit, one group used the current UCF online platform, while the remaining group engaged in traditional face-to-face or non-adaptive online learning. The Psychology and Nursing comparison courses had several sections, each with a different instructor. The Psychology course consisted of 8 objectives, covered 213 concepts and had 125 students enrolled. The Nursing course consisted of 3 objectives, covered 42 concepts and had 34 students enrolled. These students covered a wide range of demographics. The remainder of this document is divided into two sections. The first examines research into the effect of Realizeit on student outcomes. This is achieved by comparing outcomes on both internal and external exams. Following this, the UCF team built two predictive models which use key learning metrics from Realizeit to predict outcomes on an external exam. The second section examines student’s perceptions and satisfaction with their course experience with Realizeit and the course format. In addition to the survey, the researchers ran a principal component analysis, to determine if there were any underlying themes present in the response data.
3
Effect on Student Outcomes One of the core questions the UCF research team wished to address was the effect Realizeit had on student outcomes. Faculty from Psychology and Nursing had agreed to redesign their online courses to incorporate Realizeit. The instructors of courses delivered through Realizeit had access to the instructor dashboard of Realizeit, which offered them a detailed real-time view of the progress and achievement of their students. The research team compared these courses with corresponding Nursing and Psychology sections which were taught face-to-face or fully online with different instructors.
Realizeit... outperformed both face-toface and the online supported groups
The research team used the grades achieved by the students in their courses as a measure of learning. For the Psychology course they also had available the results obtained by the students on an external General Education Program (GEP) exam. One interesting component to this research was the construction of a predictive model that utilized learning metrics gathered by Realizeit to predict the student outcomes on the external GEP exam. Each of these components is discussed in more detail in the following subsections.
Grade Distributions The first comparison carried out by the researchers was to examine the student success rates on each of the courses across the three different instructional models. In this case, a successful student was defined to be a student that achieved a grade of A, B, or C. The results are summarized in Figure 1. During this initial trial, students who were supported by Realizeit throughout their course outperformed both face-to-face and the online supported groups in both courses. While the differences were not found to be statistically significant due to the small sample sizes, the differences are of interest, particularly the 7 point difference in Psychology. As mentioned earlier, within the face-to-face Psychology group and the online Nursing group, there were several classes; each
4
with a different instructor. Each instructor was responsible for grading their class according to their own grading scheme. This led to a wide variability in the distribution of results obtained across these classes. Using an adaptive system such as Realizeit can provide students with a common basis for assessment. This allows this variability in learning and grading to be minimized, all while ensuring that each student receives instruction that is both personalized and adapted to his or her unique needs. Realizeit is a system that improves with time and data. These initial findings suggests that Realizeit not only performed well, but allowed the students it supported to moderately outperform those in the other groups, some of whom were supported by highly experienced instructors. This is certainly not typical as most new learning environments would yield an initial drop in outcomes due to the learning curve associated with the rollout. Additionally, as the system improves with data and course content iterations we should expect Realizeit to improve and surpass what has already been achieved. Due to the granular nature of data collected by Realizeit, visibility into how these results were achieved by the students and instructors is available to the institution, providing a basis for improvement with each iteration of the course.
Figure 1: Student success rates (A, B, C grades) with Realizeit compared to other course formats
5
Psychology GEP exam As a part of their final assessment, students on the Psychology course were required to complete an external General Education Program (GEP) exam. This provided an opportunity to use an objective measure of achievement across all groups within the course regardless of the instructional model they used.
Realizeit produced a far more homogenous set of results, without the extreme lows experienced in the other groups
The researchers found that there was no statistical or practical difference between the average performance of students from each group, with each group achieving an average of approximately 85% (depicted via the circles in Figure 2). However there is a large difference in the range of results obtained. For each group, the range is depicted using the horizontal bars while the average of the low outliers are depicted using the diamonds in Figure 2. The researchers found that the group of students who used Realizeit produced a far more homogenous set of results, without the extreme lows experienced in the other groups. This suggests that there may be a level of consistency in the outcomes and learning experienced by students using Realizeit that is not found across those in the other groups. As mentioned by the UCF researchers, one possible explanation for the large range of results in both the face-toface and online groups is the huge variability in the experience and standard of instructors and their grading schemes. This variability can be smoothed out by a system such as Realizeit, which can account for individual needs.
Figure 2: The outcomes on the GEP exam. The horizontal bar represents the range of results for each section, the square is the average result, and the circle represents the average of the low outliers.
6
It is also worth noting that the GEP exam accounted for a different percentage of a student’s final course grade depending on which group they were in. This could bias the results somewhat due to it being more important for some groups than others.
Model to Predict Outcome on Psychology GEP Exam Given the results on the external Psychology GEP exam, the researchers were interested in determining if there is a relationship between these results and the metrics reported by Realizeit. Building on this, they wished to use complex relationships to construct a predictive model. They ultimately built two models. The first looked at predicting the GEP exam results from Realizeit module scores. The second used Realizeit-specific learning metrics to achieve the same goal. The Psychology course was broken into 8 objectives or modules, each of which contained several nodes or concepts. Realizeit provides detailed granular metrics on each of these concepts and allows them to be aggregated to the module or course level.
Realizeit Module Scores The first model used the learner’s score, as generated by the course specific grading formula within Realizeit, aggregated at the module level to predict the GEP exam scores. Figure 3 displays the correlation of each of the module scores with the GEP exam score. All modules correlated positively with the exam score with most having a strong to very strong linear relationship. This means that the higher the score the student achieved on these modules, the higher their exam score. The main outlier module is “History and Research.” This could be explained by the fact this is the first module in the course and mainly serves as an introduction. These strong relationships suggest that a predictive model is, indeed, possible. Using a stepwise linear regression procedure, the research team at UCF were able to construct a model using scores on just three of the eight modules that produced an adjusted 𝑅2 of 0.64.
7
Sensation, Perception & Learning Emotions and Health & Personality} 𝑅 2 (𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑) = 0.64 Human Diversity & Development This means that the model explained 64% of the variability seen in the GEP exam scores and also suggests that this model could potentially act as a method to predict student readiness for this external exam. Further work would need to be completed to trial this model on unseen cases to determine its accuracy in a real world setting.
Realizeit Learning Metrics This model examined the relationship between the Realizeit learning metrics aggregated to the course level and GEP exam score. Figure 4 displays the correlation between each metric and the exam score. There is more variation observed here than with the previous model: there are 2 metrics with strong positive correlations, 2 that are very weak and negative, and the remainder cover the full range in between. Again, the UCF researcher team used a stepwise linear regression procedure to construct the model and found that just three of the metrics could produce a model with an adjusted 𝑅2 of 0.67.
Figure 3: The correlation of individual Psychology module scores with the GEP Exam score
8
The metrics that were selected are Num Records (which can be seen as a measure of course engagement), Calculated grade (average score as calculated by the grading formula across all modules) and Knowledge Covered Growth (how much knowledge the student has attained since they started using Realizeit). Num Records Calculated grade } 𝑅 2 (𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑) = 0.67 Knowledge covered growth Both of these models show that there is a strong relationship between the Realizeit measures of performance and learning and the external independent measure. By developing these models, the UCF research team has demonstrated the potential for these metrics when building predictive models for other external exams. This would allow institutions to make accurate predictions for individual learners and build an exam readiness tool that could function as an early warning indicator of at-risk students.
Figure 4: The correlation of individual Realizeit metrics with the GEP Exam score
9
Student Perceptions
83.7% would use Realizeit again
In addition to measuring the effect of Realizeit on student learning outcomes, Dr. Dziuban and Dr. Moskal were also interested in tracking students’ perceptions of their learning. This involved the students completing two surveys. The first was designed specifically for students who used Realizeit in order to measure their perceptions of the system. The second was the Student Perception of Instruction (SPI) survey, which is administered to all UCF students at the end of their courses. In the following subsections we will outline the results from both surveys. This will include a principal component analysis conducted by the UCF researchers of the student responses to the Realizeit survey. This will determine if there are any underlying factors that may exist behind their response patterns.
Student Perceptions of Realizeit
82.8% felt they learned the course material better as a result of using Realizeit
The responses of the students to the survey questions are summarized in Figure 5. Some of these questions address Realizeit directly; others address components that are under the control of the course designers and administrators, such as difficulty and clarity of content. It is worth noting that some of these metrics will improve with further iterations of the course as the curriculum and content improve. In general, the student responses were positive. If given the choice, 83.7% would use Realizeit again and 82.8% felt they learned the course material better as a result of using Realizeit. The remaining questions in the survey help shed light on how these high percentages were achieved. Students responded (89.4%) that Realizeit was easy to use and provided useful and clear feedback and guidance; 91.2% found the instructions clear, and 86.9% found the feedback provided by the system helped them stay on track. For a learning system to be effective the students must trust it. From the questions on accuracy, the UCF researchers found that the students trusted that Realizeit accurately captured
10
their learning progress. Further, 77.7% found that the assessments were effective and 80.9% reported that the ability levels as measured by Realizeit were accurate. The key to any adaptive learning system is that it becomes personalized to each learner over time, and this then drives
Figure 5: The student responses to survey questions
11
engagement. The students in this study found this to be the case with 73.5% reporting that the system became personalized to them and 80.9% indicated it increased their engagement. In addition, a learning system must select and deliver concepts and learning content to each student at the right time, pace, and difficulty. The learning must be difficult enough to be challenging but easy enough to be achievable. Most students felt the learning fell into this category with 55% reporting that the material was neither too easy nor too hard.
73.5% felt that the system became personalized to them and 80.9% felt it increased their engagement
The trade-off for students was time, with 53.9% feeling they spent more time in the Realizeit-supported class than compared to a traditional class. However, as we can see from the previous responses, this additional time was associated with higher engagement, it resulted in them learning the course material better, and overall most would like to use Realizeit again. The results from the questions on interactions with other students were quite interesting. The students felt that they interacted less with others as a result of using Realizeit (70.1%). However, most students prefer little or no interaction (57.7%). This insight demonstrates that there is potential for further study in this area in order to create an environment that fosters useful, effective and engaging interactions between learners in an online setting. From the comments section of the survey, the students noted the Realizeit features they liked most. The comments broadly fell under the headings of:
Ease of use (51 comments) Personalization (31 comments) Organization (31 comments) Improved outcomes (12 comments) Timing (12 comments) Self-paced (8 comments) Interactive (6 comments) Guidance (6 comments)
12
Principal Component Analysis of Student Perception of Realizeit Principal component analysis determines if there are any underlying hidden traits or components that exist behind a set of data. The research team at UCF used this technique to analyze the student responses. The team found two traits. The relationship between each of the survey questions and the traits can be visualized in Figure 6. The researchers labeled the traits as follows:
Did Realizeit create an “Effective Learning Climate?” What was the level of “Engagement Effort” required?
These new traits allowed the researchers to summarize the students’ answers to the survey questions using these two components. Following on from this analysis, the research team then compared the students’ scores on these components to their answers to the survey item “Realizeit helped me learn.” They found that the more positive the student’s response to “Realizeit helped me learn” the higher the “Effective Learning Climate” and the higher the “Engagement Effort” components. Interestingly, this suggests that although it required more effort, the willingness to engage was high because students believed it created an effective learning climate. This supports the conclusions from the previous analysis and implies that if
Figure 6: The reduction of the space from 17 questions to 2 underlying principal components. The shading represents the size of the coefficient that relates a question to the components.
13
you can create the right learning climate, then students will engage and expend the effort. The key ingredient to creating an effective learning climate is to create a system that the learner trusts and believes will help them learn. The best instructors know how to create this environment in their classrooms.
Student Perception of Instruction the use of Realizeit lead to much improved ratings of excellence in instruction
Figure 7: SPI – The ability to communicate ideas effectively – percentage who gave a rating of excellent
The student perception of instruction survey is given to all students at UCF at the end of each course. The research team found that students who used Realizeit were more (and in some cases much more) satisfied with the instruction they received than those in other groups. The figures below show the percentage of students who provided a rating of excellent for three different areas split by course. A higher percentage of students who used Realizeit provided a rating of excellent on the courses’ ability to communicate ideas effectively (Figure 7), the creation of an effective learning environment (Figure 8) and on overall satisfaction (Figure 9). This holds true for both the nursing and psychology courses. This demonstrates that the personalized learning
14
environment led to very positive ratings of excellence in instruction.
Figure 8: SPI – The ability of the course to create an effective learning environment – percentage who gave a rating of excellent
Figure 9: SPI – Overall – percentage who gave a rating of excellent
15
Conclusion The main conclusions drawn from the initial trials of Realizeit at UCF are as follows:
Students who used Realizeit on average moderately outperformed those students who used traditional and online instructional models. Realizeit students’ results on the external Psychology exam were far more homogeneous, without the low outlier values found in the other groups. Realizeit metrics correlated well with an external measure of achievement and could be used to generate a predictive model. Overall the students who used Realizeit where satisfied with the system. 83.7% would use Realizeit again and 82.8% felt they learned the course material better as a result of using Realizeit. Students who used Realizeit were more satisfied with the instruction they received than those in other groups. Realizeit was easy to use and provided useful and clear feedback and guidance. 89.4% agreed that Realizeit was easy to use while 91.2% found the instructions clear. Students trusted Realizeit: 77.7% found that the assessments were effective and 80.9% reported that the ability levels as measured by Realizeit were accurate. 73.5% found that Realizeit became personalized to them over time 80.9% felt Realizeit increased their engagement. The principal component analysis suggests that although it required more effort, the willingness to engage in learning in Realizeit was high because students believed it created an effective learning climate.
Dr. Dziuban and Dr. Moskal are continuing their research on Realizeit. In the Spring 2015 semester they extended this study to include 3 full courses delivered with the help of Realizeit. This includes:
16
One group of 172 students from the General Psychology course, Three groups (one fully online and two blended) totaling 94 students from the Nursing (Physiopathology) course, Two groups totaling 59 students from a College Algebra course.
As the work presented here is published by the researchers, appropriate references will be added to this document.
17
Contact Us
Sales & Marketing
www.realizeitlearning.com Chicago 220 N Smith St, Suite 228, Palatine, IL 60067 Telephone (Toll Free): (888) 959 2814 Telephone (Local): (224) 836 5316
Email:
[email protected]
Research & Development
Dublin Hibernian Estate, Greenhills Road, Tallaght, Dublin 24 Telephone: +353 (1) 4683 800 Email:
[email protected]
www.realizeitlearning.com