Meta-Analysis - Boston College [PDF]

Technology and Assessment tudy ollaborative S C Meta-Analysis: Writing with Computers 1992–2002

Amie Goldberg, Michael Russell, & Abigail Cook Technology and Assessment Study Collaborative Boston College 332 Campion Hall Chestnut Hill, MA 02467

www.intasc.org

Meta Analysis: Writing with Computers 1992–2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and Assessment Study Collaborative Boston College Released December 2002

Michael K. Russell, Project Director/Boston College Copyright © 2002 Technology and Assessment Study Collaborative, Boston College Supported under the Field Initiated Study Grant Program, PR/Award Number R305T010065, as administered by the Office of Educational Research and Improvement, U.S. Department of Education. The finding and opinions expressed in this report do not reflect the positions or policies of the Office of Educational Research and Improvement, or the U.S. Department of Education.

Meta Analysis: Writing with Computers 1992–2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and Assessment Study Collaborative Boston College Released December 2002

Introduction Over the past two decades, the presence of computers in schools has increased rapidly. While schools had 1 computer for every 125 students in 1983, they had 1 for every 9 students in 1995, 1 for every 6 students in 1998, and 1 for every 4.2 students in 2001 (Glennan & Melmed, 1996; Market Data Retrieval, 1999, 2001). Today, some states, such as South Dakota, report a student to computer ratio of 2:1 (Bennett, 2002). As the availability of computers in schools has increased, so too has their use. A national survey of teachers indicates that in 1998, 50 percent of K–12 teachers had students use word processors, 36 percent had them use CD ROMS, and 29 percent had them use the world wide web (Becker, 1999). More recent national data indicates that 75% of elementary school-aged students and 85% of middle and high schoolaged students use a computer in school (U.S Department of Commerce, 2002). Today, the most common educational use of computers by students is for word processing (Becker, 1999; inTASC, 2002) Given the increased presence of computers in schools and use of computers by students, there is an increasing demand for evidence that the use of computers impacts student learning. Most recently, the No Child Left Behind (NCLB) Act repeatedly calls for “scientifically” and “research-based evidence” that programs have a positive impact on student learning. Specific to technology, sections 2402 of the NCLB describes the purposes and goals of the Enhancing Education Through Technology Act as supporting “the rigorous evaluation of programs funded under this part, particularly regarding the impact of such programs on student academic achievement” and encouraging “the effective integration of technology resources and systems with teacher training and curriculum development to establish research-based instructional methods that can be widely implemented as best practices by State educational agencies and local educational agencies” (NCLB, 2001, Section 2402). Given that students

Meta-Analysis: Writing with Computers 1992–2002

4

use word processing in school more than any other computer application, it is logical to ask: do computers have a positive effect on students’ writing process and quality of writing they produce? The study presented here responds directly to the call for research-based evidence that the use of word processors has a positive impact on student writing. As is described more fully below, the study presented here employs meta-analytic techniques, commonly used in fields of medicine and economics, to integrate the findings of studies conducted between 1992–2002. This research synthesis allows educators, administrators, policymakers, and others to more fully capitalize on the most recent findings regarding the impact of word processing on students’ writing.

Word Processing and Student Writing Over the past two decades, more than 200 studies have examined the impact of word processing on student writing. Over half of these studies, however, were conducted prior to presence and wide-scale use of current menu-driven word processors. In addition, these early studies focused on students who were generally less accustomed to working with computer technologies compared to students today. Regardless of these obstacles, syntheses of early research provide some evidence of positive effects. For example, important findings emerged from Cochran-Smith’s (1991) qualitative literature review on word processing and writing in elementary classrooms. Among them, Cochran-Smith found that students of all ages had positive attitudes toward word processing and were able to master keyboarding strategies for use in age-appropriate writing activities. Cochran-Smith also found that students who used word processors spent a greater amount of time writing and produced slightly longer, neater, and more technically error-free texts than with paper and pencil. However, this review of the literature also indicated that word-processing, in and of itself, generally did not impact the overall quality of student writing. Other early research, however, such as Bangert-Drowns’ (1993) quantitative meta-analysis of 28 individual studies spanning elementary through post-secondary school levels, indicates that word processing contributed to a modest but consistent improvement in the quality of students’ writing: approximately two-thirds of the 28 studies’ results were in favor of the word processor over handwritten text. In general, the research on word processors and student writing conducted during the 1980’s and early 1990’s suggests many ways in which writing on computers may help students produce better work. Although much of this research was performed before large numbers of computers were present in schools, formal studies report that when students write on computer they tend to produce more text and make more revisions (Vacc, 1987; Dauite, 1986). Studies that compare student work produced on computer with work produced on paper find that for some groups of students, writing on computer also had a positive effect on the quality of student writing (Owston, 1991; Hannafin & Dalton; 1987). This positive effect is strongest for students with learning disabilities, early elementary-aged students and college-aged students (Phoenix & Hannan, 1984; Sitko & Crealock, 1986; Hass & Hayes, 1986). Additionally, when applied to meet curricular goals, education technology provides alternative approaches to sustaining student interest, developing student knowledge and skill, and provides supplementary materials that teachers can use to extend student learn-

Meta-Analysis: Writing with Computers 1992–2002

5

ing. Although earlier research syntheses reveal just modest trends, individual studies of that era have shown that writing with a computer can increase the amount of writing students perform, the extent to which students edit their writing (Dauite, 1986; Etchinson, 1989; Vacc, 1987), which, in turn, leads to higher quality writing (Hannafin & Dalton, 1987; Kerchner & Kistinger, 1984; Williamson & Pence, 1989). Throughout the 1990’s, however, technology has, and continues to, develop at an astonishing pace. Word processing technologies, are easier to use and no longer the classroom novelty they once were. A new generation of studies that examine the impact of word processing on writing fills today’s journals. In response to improvements in word processing and students comfort with technology, the study presented here builds on Cochran-Smith’s (1991) and Bangert-Drowns’ (1993) work by integrating research conducted since 1991 that has focused on the impact of word processors on the quantity and quality of student writing. The study presented here differs in two ways from the two previous meta-analysis described above. First, while Cochran-Smith’s (1991) study was qualitative in nature and Bangert-Drowns’ (1993) employed a quantitative meta-analytic technique, this study aims to combine quantitative and qualitative methods in order to provide a richer, more encompassing view of all data available for the time period under study. Secondly, the quantitative component provides an expanded scope on student and learning environment level variables in relation to writing performance. These supplemental analyses include factors such as: students’ grade level, keyboarding skills, school setting (urban, suburban, rural), etc. The specific research questions addressed in this study are: • Does word processing impact K–12 student writing? And, if so, in what ways (i.e., is quality and/or quantity of student writing impacted)? • Does the impact of word processing on student writing vary according to other factors, such as student-level characteristics (as described above)?

Meta-Analysis: Writing with Computers 1992–2002

6

Methodology Meta-analytic procedures refer to a set of statistical techniques used to systematically review and synthesize independent studies within a specific area of research. Gene Glass first proposed such methods and coined the term “meta-analysis” in 1976. “Meta-analysis refers to the analysis of analyses … it …refer[s] to the statistical analysis of a large collection of results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature (Glass, p. 3).” The meta-analytic portion of the study was conducted using procedures set forth by Lipsey and Wilson (2001) and Hedges and Olkin (1985). The methodology followed five phases: • identification of relevant studies, • determination for inclusion, • coding, • effect size extraction and calculation, and • data analyses. Each of these phases is described separately below.

Identification of Relevant Studies The search for relevant studies was as exhaustive as possible. Methods used to find studies that focused on word processing included: • Searching online databases such as ERIC, Educational Abstracts, PsychLit, and Dissertation Abstracts, • Searching websites known to reference or contain research related to educational technology such as the US Department of Education, technology and educational research organizations. • Searching scholarly e-journals that may not be indexed • Employing general search engines (e.g., Google) in keyword searches for additional manuscripts that either had not yet been catalogued in ERIC or were currently under refereed journal review (yet posted on the researcher’s own webpage), and • Directly inquiring with researchers known to be active in studying educational technology about relevant work via email. To maximize the pool of studies for consideration, search strategies varied slightly depending on the structure of the source, and included a variety of combinations of terms in each search. Search terms included different forms (i.e. computerized and computer; word process and word processing, etc.) of such words as: computer, writing, word processing, pencil-and-paper, and handwritten. If, based on the article’s abstract/description, relevancy to the present study could not be determined, it was collected for possible inclusion. The resulting collection included 99 articles (see Appendix C).

Meta-Analysis: Writing with Computers 1992–2002

7

Determination for Inclusion The inclusion criteria for the meta-analysis were stringent. Each study had to: • be a quantitative study, conducted between the years of 1992-2002, in which results were reported in a way that would allow an effect size calculation, have employed a research design which allowed for either a measure of word-processing’s impact on writing over time, OR a direct comparison between paper-and-pencil writing and computerized writing, • have ‘quality of student writing’ and/or ‘quantity of student writing’ and/ or ‘revision of student writing’ as its outcome measure(s), • not specifically focus on the effects of grammar and spell-checkers or heavily multimedia-enhanced word processing software, • not examine differences in writing within the context of a test administration (i.e, focused on the mode of test administration rather than the mode of learning), and • focus on students in grades K–12. Independently, two researchers read all collected studies to determine eligibility for inclusion based on above criteria. Any discrepancies between researchers were discussed and resolved. In total, 26 studies met all inclusion criteria. An additional 35 studies/articles were on target regarding the topic, but were either qualitative, insufficient in reporting quantitative data (to enable effect size extraction), or were conceptual or commentary papers that focused on how word processors could be used for instruction. These studies were set aside for separate analysis. The research focus of the remaining 38 articles did not match the purposes of this study. Figure 1 illustrates the results of the literature search and criteria screening, and Figure 2 depicts the studies included in the meta-analysis classified by their measured outcomes.

Figure 1:

Computers and Writing 1992–2002: Articles Collected in Literature Search by Type (N=99)

13%

5% (n=5)

Focus not applicable to meta-analysis

(n=13)

38% (n=38)

17% (n=17)

Included in meta-analysis Ineligible: Commentary/instructional practices articles Ineligible: quantitative studies not including sufficient statistical data Qualitative studies

26% (n=26)

Meta-Analysis: Writing with Computers 1992–2002

Figure 2:

8

Studies Included in Meta-Analysis by Outcomes Measured 9

31% (n=8)

8

23% (n=6)

7 6

19% (n=5)

5

12% (n=3)

4 3

8% (n=2) 4% (n=1)

2 1 0

Quality

Quantity

Quality & Quantity

Revision

4% (n=1)

Quantity & Quality & Quality, Revision Revision Quantity, & Revision

Coding of Study Features and Outcome Measures Study features were coded to aid examination of methodological and substantive characteristics that may contribute to variations in results among studies. Based on a review of the literature, a coding framework was constructed to encompass salient features of each study. According to this coding scheme, two researchers independently coded each study. Afterwards, coding was discussed between researchers on a study-by-study basis. Coding discrepancies, which occurred infrequently, were discussed and resolved by consulting the original research study. The final coding frame encompasses seven categories of study descriptors including: publication type, research methodology, student characteristics, technology related factors, writing environment factors, instructional support facators, and outcome measures. Appendix A contains a full description of the variables and all thirty-three levels included in the coding framework. After all studies were coded, a variable representing “methodological quality” (Shadish & Haddock, 1994; Moher & Olkin, 1995) was derived from a subset of the coded variables. For each study, methodological quality was based on a sixteen point scale. This scale was based on the following formula: • one point was assigned for each dichotomous variable coded as “yes” in the “Research Methodology” category, • one point for studies obtained from refereed journals (“Publication Type”), • a maximum of three points for the “Intervention time/Duration of study” and “Sample Size” variables,

Meta-Analysis: Writing with Computers 1992–2002

9

• heterogeneity of the sample’s gender and race/ethnicity were each awarded one point (“Student Characteristics”), and • mention of at least one demographic descriptor for the study’s sample (i.e., gender, race, geographic setting (rural, urban, suburban). Finally, there was some ambiguity in study reporting which sometimes made coding study features a challenge. Where the presence or absence of a feature could not be reasonably detected (explicitly or by implication), an additional code, “no information available,” was employed. The codes assigned to each study along with all data used to calculate effect sizes are presented in a datafile available from inTASC.

Extracting and Calculating Effect Sizes The meta-analytic portion of the data analysis requires the calculation of effect sizes. Conceptually, an effect size represents the standardized difference between two groups on a given measure. Mathematically, it is the mean difference between groups expressed in standard deviation units. In this study, for example, effect sizes were calculated taking the mean performance difference between computerized and paperand-pencil groups and dividing it by a pooled standard deviation. Generally speaking, effect sizes between .2 and .5 standard deviation units are considered small, those between .5 and .8 standard deviation units are medium, and effect sizes .8 or greater are considered large. In order to decrease the probability of falsely concluding that word processing has an effect on student writing (i.e., committing a Type I error), the unit of analysis is an “independent study finding.” For each of the three outcome measures, an independent effect size was calculated. For studies that reported more than one measure for a particular outcome for the same sample (i.e., “writing quality” was often measured in more than one way per study; mechanics, content, organization, etc. were frequently encountered sub-domains), overall means and standard deviations across these measures were calculated and used to calculate a single effect size. In this way, the assumption of independence was preserved and inflated Type I error rates were controlled for, yet no study findings were ignored. At the outset of the study, we had hoped to base the calculation of effect sizes using gain scores (the difference between scores on post-test and pre-test measures). Unfortunately, a considerable number of studies either lacked a pre-post design, or failed to report pre-test data. This precluded the most compelling perspective from being meta-analyzed: comparing gain scores between paper-and-pencil and computer writing groups. In order to maximize the number of studies included in the analysis, the few pre- and post-test designs were analyzed only in terms of post-test data. This enabled results from the pre/post studies to be analyzed with post-only design data. For all three outcomes (i.e., quantity of writing, quality of writing, and revisions), the standardized mean difference effect size statistic was employed. Since it has been documented that this effect size index tends to be upwardly biased when based on small sample sizes, Hedges’ (1981) correction was applied. Effect sizes from data in the form of t- and F-statistics, frequencies, and p-values were computed via formulas provided by Lipsey & Wilson (2001).

Meta-Analysis: Writing with Computers 1992–2002

10

Adjusting for Bias and Applying Inverse Variance Weights Following standard meta-analytic procedures, an inverse variance weight was applied to each effect size. Essentially, this procedure weights each effect size by the inverse of its sampling variance in order to give more weight to findings based on larger sampling sizes. Thus, all inferential statistical analyses were conducted on weighted effect sizes. Outlier analyses of the sampling weights and effect sizes were also performed. No outliers were identified for effect sizes. However, for the “Quantity of writing” analyses, two sampling weights were more than two standard deviations from the mean sampling weight. Following a procedure originally employed by Lipsey (1992), the inverse variance weights in this study were adjusted so that they did not over-weight the effects found in these two studies.1

Data Analysis Three types of data analyses were performed. First, using the effect size extracted from each study, an overall effect size across studies was calculated and tested for statistical significance. Second, analyses were performed to investigate the potential effects of publication bias. Finally, to investigate the extent to which study features moderated the effect on outcome measures, regression analyses were performed. Below, we describe the methods used to explore publication bias and moderating effects.

Publication Bias Publication bias analyses were performed via Forest plots, funnel plots, and the fail-safe N analysis. Forest plots were used to visually convey the contribution of each study to its meta-analysis, by plotting study effect sizes and corresponding confidence interval bars in a single display. Funnel plots, another widely-used technique for detecting publication bias, were also employed. These plots graphically investigate possible gaps among the studies’ findings by simply plotting effect sizes against sample sizes. Finally, a fail-safe n analysis (Orwin, 1983) was conducted for each meta-analysis. This analysis addresses the “file-drawer” problem in meta-analytic research and provides an estimate of the number of insignificant, unpublished studies that would have to exist in order to render a statistically significant meta-analytic finding insignificant.

Significance and Homogeneity Analysis For each meta analysis, an independent set of effect sizes were extracted, weighted and then aggregated. Prior to exploring the extent to which other factors, such as grade level or publication type, influence the effect sizes, a test for homogeneity was conducted. In essence, the test of homogeneity examines whether the group of effect sizes are part of the same population of effect sizes and thus are not influenced by any other variable. As Table 1 indicates, the effect sizes included in the quantity and quality meta analyses are heterogenous. For this reason, additional analyses were conducted in an attempt to identify other factors that may influence the study findings. Due to the small number of studies that included measures of revisions, a formal test for homogeneity was not possible.

Meta-Analysis: Writing with Computers 1992–2002

Table 1:

11

Results of Tests for Homogeneity Quantity of Writing (n=14) Min ES Max ES Weighted SD -1.617 11.971 4.913

Homogeneity (Q) 4120.6571

df 13

P .0001

Quality of Writing (n=15) Min ES Max ES Weighted SD -2.897 30.117 10.801

Homogeneity (Q) 24396.9961

df 14

P .0001

Moderator Variable Multiple Regression Models To explore factors that may influence the effect of word processing on the quantity and/or quality of student writing, regression analyses were conducted in which the coded study features were independent variables. These analyses were limited by two conditions. First, these analyses could only include study features that were reported by most researchers. Second, for each study feature included in the regression analyses, there had to be variation among studies. For several study features, all studies received the same code and thus did not vary. These two conditions severely limited moderator analyses. For each outcome variable, frequencies of study feature variables were examined. After suitable independent variables were identified, variables with more than two levels were recoded into dummy variables. These variables were then categorized into groups by theme. For example, variables such as “presence of control group,” “length of intervention,” “type of publication,” and “conversion of handwritten student work to word processed format” fell under the theme labeled “Study’s Methodological Quality.” Variables such as: “technical assistance provided to students”, “student participation in peer editing,” “students receive teacher feedback,” were included in the “Student Support” theme. Ideally, for each outcome, each themed group of variables would be entered as a single block and themed groups would be entered stepwise into a single regression model. However, this was not statistically possible due to the small number of effect sizes. Instead, each themed group of variables was entered as a single block of independent variables and each theme was analyzed in separate regression models.

Summary of Findings In this section, we present a summary of the findings. Readers who are familiar with meta analytic techniques or who desire a more technical presentation of the findings are encouraged to read Appendix B. The analyses focuses on three outcome variables commonly reported by studies that examine the impact of word processors on student writing. These variables include: Quantity of Writing, Quality of Writing, and Revision Behavior. Below, findings for each of these variables are presented separately.

Meta-Analysis: Writing with Computers 1992–2002

12

Quantity of Writing Fourteen studies included sufficient information to calculate effect sizes that compare the quantity of writing, as measured by word count, between computer and paper-and-pencil groups. Figure 3 depicts the effect sizes and the 95% confidence interval for all 14 studies sorted by publication year. The fifteenth entry depicts the mean weighted effect size across all fourteen studies, along with the 95% confidence interval.

Figure 3: Author

Forest Plot of Quantity of Writing Meta-Analysis Publication Grand Year N*

Adjusted Effect Lower Upper Size 95%CI 95%CI

Owston, et al.

1992

136

0.00

-0.34

0.34

D'Odorico & Zammuner

1993

51

0.56

0.00

1.12

Snyder

1993

51

0.78

0.21

1.35

Peterson

1993

36

1.31

0.59

2.03

Hagler

1993

76

0.47

0.01

0.93

Olson

1994

14

0.02

-1.03

1.07

Jones

1994

20

0.48

-0.41

1.37

Brigman

1994

12

1.23

0.00

2.46

Wolfe, et al.

1996

60

-0.05

-0.41

0.3

Nichols

1996

60

0.87

0.34

1.4

Dybahl, et al.

1997

41

-0.14

-0.77

0.48

Godsey

2000

44

1.31

0.66

1.96

Padgett

2000

32

0.52

-0.18

1.23

Barrera, et al.

2001

36

0.21

-0.44

0.87

mean

669

*Grand N = npaper + ncomputer

Effect Size -2

-1

0

1

2

-2

-1

0

1

2

0.541 0.380 0.702 Lower 95% Confidence lnterval

Upper 95% Confidence lnterval Effect Size

Figure 3 indicates that four of the fourteen studies had effect sizes that were approximately zero or negative, but which did not differ significantly from zero. Figure 1 also shows that four of the fourteen studies had positive effect sizes that

Meta-Analysis: Writing with Computers 1992–2002

13

differed significantly from zero. In addition, the mean weighted effect size across all fourteen studies is .50, which differs significantly from zero. Thus, across the fourteen studies, the meta analysis indicates that students who write with word processors tend to produce longer passages than students who write with paper-and-pencil. Recognizing that our search for studies may have missed some studies that have not been published, a “fail-safe N” analysis (Orwin, 1983) was conducted to estimate the number of studies needed to report no effect to nullify the mean adjusted effect size. This analysis indicates that in order to reverse the effect size found, there would need to be 24 unpublished studies that found no effect. Given that only 14 studies that fit the selection criteria were found and that only four of these had non-positive effect sizes, it seems highly unlikely that an additional 24 studies that found non-positive effects exist. This suggests that our meta-analytic findings are robust to publication bias. As described above, regression analyses were performed to explore factors that may influence the effect of word processing on the quantity of student writing. These analyses indicated that student supports (i.e., keyboard training, technical assistance, teacher feedback, and peer editing) were not significant factors affecting the quantity of student writing. Similarly, student characteristics (i.e., keyboard experience prior to the study, student achievement level, school setting, and grade level) also were not significant factors affecting the quantity of student writing, although grade level did approach statistical significance. Finally, the study characteristics (i.e., publication type, presence of control group, pre-post design, length of study) were not related to the effect of word processing on the quantity of student writing. Recognizing that studies that lasted for less than six weeks may not provide enough time for the use of word processors to impact student writing, a separate set of regression analyses were performed for the sub-set of studies that lasted more than six weeks. For this sub-set of studies, a significant relationship between school level and effect size was found. On average, effect sizes were larger for studies that focused on middle and high school students as compared to elementary students. All other factors were remained insignificant. In short, the meta analysis of studies that focused on the effect of word processing on the quantity of student writing found a positive overall effect that was about one-half standard deviation. This effect tended to be larger for middle and high school students than for elementary students

Quality of Writing Fifteen studies included sufficient information to calculate effect sizes that compare the quality of writing between computer and paper-and-pencil groups. Figure 4 depicts the effect sizes and the 95% confidence interval for all 15 studies sorted by publication year. The sixteenth entry depicts the mean weighted effect size across all fifteen studies, along with the 95% confidence interval. Figure 4 indicates that four of the fifteen studies had effect sizes that were approximately zero or negative, but which did not differ significantly from zero. Since the power in meta-analysis is the aggregation of findings across many studies, it is not unusual to find a subset of studies that contradict the overall trend of findings. In this case, a qualitative examination did not reveal any systematic differences among these

Meta-Analysis: Writing with Computers 1992–2002

14

studies’ features as compared with those studies reporting positive effect sizes. Figure 4 also shows that the eleven remaining studies had positive effect sizes and that seven of these effect sizes differed significantly from zero. In addition, the mean adjusted effect size across all fifteen studies is .41, which differs significantly from zero. According to Cohen’s criteria for effect sizes, this is considered a small to moderate effect. Thus, across the fifteen studies, the meta analysis indicates that students who write with word processors tend to produce higher quality passages than students who write with paper-and-pencil.

Figure 4: Author

Forest Plot of Quality of Writing Meta-Analysis Publication Grand Year N*

Adjusted Effect Lower Upper Size 95%CI 95%CI

Owston, et al.

1992

136

0.38

0.04 0.72

Hagler

1993

38

0.96

0.49 1.44

Jones

1994

20

1.25

0.29 2.21

Jackiewicz

1995

58

0.62

0.09 1.15

Keetley

1995

23

0.20

-0.62 1.02

Lam & Pennington

1995

34

0.25

-0.42 0.93

Nichols

1996

60

0.01

-0.5 0.52

Lichetenstein

1996

32

0.77

0.05 1.49

Wolfe,et al.

1996

120

-0.06

Breese, et al.

1996

44

0.83

0.21 1.44

Langone et al.

1996

12

0.43

-0.71 1.58

Jones & Pellegrino

1996

20

-0.61

-1.5 0.29

Lerew

1997

150

0.88

0.55 1.22

Dybdhal, et al.

1997

41

-0.20

-0.83 0.42

Head

2000

50

0.43

-0.13 0.99

838

0.410

0.340 0.481

mean *Grand N = npaper + ncomputer

-0.42

Effect Size -2

-1

0

1

2

-2

-1

0

1

2

0.3

Lower 95% Confidence lnterval

Upper 95% Confidence lnterval Effect Size

Meta-Analysis: Writing with Computers 1992–2002

15

Recognizing that our search for studies may have missed some studies that have never been published, the “fail-safe N” analysis was again conducted. This analysis indicates that in order to reverse the effect size found, there would have to be 16 unpublished studies that found no effect. Given that only 15 studies that fit the selection criteria were found and that only four of these had non-positive effect sizes, it seems highly unlikely that an additional 16 studies that found non-positive effects exist. As described above, regression analyses were performed to explore factors that may influence the effect of word processing on the quality of student writing. These analyses indicated that student supports (i.e., keyboard training, technical assistance, teacher feedback, and peer editing) were not significant factors affecting the quality of student writing. Similarly, the study characteristics (i.e., type of publication, employment of random assignment, employment of pre-post design, single vs. multiple classroom sampling, length of study, etc.) were not related to the effect of word processing on the quality of student writing. However, when examining student characteristics (i.e., keyboard experience prior to the study, student achievement level, school setting, and grade level), a statistically significant relationship was detected between grade level and quality of writing: as school level increased, the magnitude of the effect size increased. Recognizing that studies that lasted for less than six weeks may not provide enough time for the use of word processors to impact student writing, a separate set of regression analyses were performed for the sub-set of studies that lasted more than six weeks. For this sub-set of studies, no significant relationships were found. This suggests that the relationship between school level and quality of writing occurred regardless of the length of study. In short, the meta analysis of studies that focused on the effect of word processing on the quality of student writing found a positive overall effect that was about four tenths of a standard deviation. As with the effect for quantity, this effect tended to be larger for middle and high school students than for elementary students.

Revision Behavior Only six of the thirty studies that met the criteria for inclusion in this study included measures related to revisions. Of these six studies, half were published in refereed journals, half took place in elementary schools, and only one employed a sample size greater than thirty. Because of the small sample size (only 6) coupled with the reporting of multiple measures of revisions, which could not be combined into a single measure for each study, it was not possible to calculate an average effect size. Nonetheless, these six studies all report that students made more changes to their writing between drafts when word processors were used as compared to paper-and-pencil. In both studies focused on revision and quality of writing, revisions made by students using word processors resulted in higher quality writing than did students revising their work with paper and pencils. It should also be noted that one study found that students writing with paper-and-pencil produced more substantive revisions than did students who used word processors.

Meta-Analysis: Writing with Computers 1992–2002

16

In short, given the small number of studies that compared revisions made on paper with revisions made with word processors coupled with the multiple methods used to measure revisions, it is difficult to estimate the effect of computer use on student revisions.

Qualitative Analysis of Excluded Studies In total, 65 articles published between 1992 and 2002 that focused on the effects of computers on student writing were found during our search (see Appendix C). Of these, 30 met our criteria for inclusion in the quantitative portion of the meta-analyses. In many cases, the studies that were excluded contained information about the effect of computers on student writing, but did not report sufficient statistics to calculate effect sizes. In several cases, the excluded studies did not focus on the three variables of interest – quantity of writing, quality of writing, and revisions – but instead provide information about the effect of computers on other aspects of student writing. And in still other cases, excluded studies employed qualitative methods to explore a variety of ways in which computers may impact student writing. In this section, we summarize the findings across the excluded studies. We do this both to supplement the findings of our quantitative meta analysis and to check that our criteria for inclusion in the meta analysis did not systematically bias our analysis. It is important to remember that the studies summarized below were selected as possible candidates for inclusion in our meta analyses and were selected because it was believed they included information about the effect of computers on the quantity of writing, quality of writing, and the amount of revisions made while writing. Thus, the sample of studies summarized here is not representative of all studies that focus on computers and writing.

Writing as a Social Process Several of the excluded studies examined how interactions among students were effected when students wrote with computers. These studies describe in rich detail the social interactions between students as they engage in the writing process. In general, these studies found that when students use computers to produce writing, the writing process becomes more collaborative and includes more peer-editing and peer-mediated work (Snyder, 1994; Baker & Kinzer, 1998; Butler & Cox, 1992). As an example, Snyder (1994) describes changes in classroom-talk when students use computers rather than paper-and-pencil. In Snyder’s study, teacher-to-student communications were predominant in the “pens classroom” while student-to-student interactions occurred more frequently in the “computers classroom.” In addition, Snyder describes how the teacher’s role shifted from activity leader in the “pens classroom” to that of facilitator and “proof-reader” in the “computers classroom.” Snyder attributed this change to students’ increased motivation, engagement and independence when writing with computers.

Meta-Analysis: Writing with Computers 1992–2002

17

Writing as an Iterative Process One study focused specifically on how the writing process changed when students wrote on computers versus on paper (Baker & Kinzer, 1998). This study found that when students wrote on paper, the writing process was more linear such that students generally brainstormed, outlined their ideas, wrote a draft, then revised the draft, produced a second draft, and then proof read the draft before producing the final version. When students produced writing on computers, however, the process of producing and revising text was more integrated such that students would begin recording ideas and would modify their ideas before completing an entire draft. Students also appeared more willing to abandon ideas in mid-stream to pursue a new idea. In this way, the process of revision tended to begin earlier in the writing process and often was performed as new ideas were being recorded. Rather than waiting until an entire draft of text was produced before beginning the revision process, students appeared to critically examine and edit their text as ideas flowed from their mind to written form.

Computers Motivate Students A few of the excluded studies noted that computers seemed to motivate students, especially reluctant writers. In her case study of two third grade “reluctant writers,” Yackanicz’s (2000) found that these students were more willing to engage and sustain in writing activities when they used the computer. As a result, these students wrote more often, for longer periods of time, and produced more writing when they used a computer instead of paper-and-pencil.

Keyboarding and Computers One study that focused on a range of middle school students found that it tended to take students longer periods of time to produce writing on computers as compared to on paper (Jackowski-Bartol, 2001). Although no formal measures of keyboarding skills were recorded, Jackowski-Bartol attributed this difference to a lack of keyboarding skills and inferred that as students keyboarding skills improve, the amount of time required to produce writing on computers would decrease.

Effects on Student Writing Several of the excluded studies examined the effect of computers on various aspects of students’ final written products. In an examination of writing produced by high school students who participated in a computer technology infusion product, Allison (1999) reported improvement in students literacy skills, attitudes toward writing, and an increase in the number of students who demonstrated high-order thinking skills in their writing. In a three-week study of 66 sixth graders who were randomly assigned to write on computer or paper, Grejda and Hannafin (1992) found that the quality of student writing was comparable, but students who used word-processors introduced fewer new errors when revising their text as compared to students who re-wrote their work on paper. In a three year study that examined the effect of computers on student writing, Owston and Wideman (1997) compared changes in the quantity and quality of writing of students attending a school in which there was one computer for every fifteen students versus a school in which there was one computer for every three students.

Meta-Analysis: Writing with Computers 1992–2002

18

After three years, Owston and Wideman found that the quality of writing improved at a faster rate in the high access school and that the mean length of composition was three times longer in the high access school. The researchers, however, acknowledged that their findings do not take into account differences between teachers or the demographics of the students. Nonetheless, the researchers state that these variables did not appear to explain the superior writing produced by students in the high access school. Not all studies, however, report positive effects of computers on student writing. In a three year study in which seventy-two students wrote on computer and paper, Shaw, Nauman and Burson (1994) report that the length and quality of writing produced on paper was higher than writing produced on computer. This finding occurred even though students who wrote on computer had received keyboarding instruction. The authors described writing produced on computer as “stilted” and less creative.

Discussion Responding directly to calls for systematic, research-based evidence of the effects of computers and student learning, this study employed meta-analytic techniques more commonly used in medicine and economics to summarize findings across multiple studies. Although a large number of studies initially identified for inclusion in the meta-analysis had to be eliminated either because they were qualitative in nature or because they failed to report statistics required to calculate effect sizes, this study indicates that instructional uses of computers for writing are having a positive impact on student writing, both in terms of quantity and quality. In addition, the findings reported in the excluded studies are consistent with both the findings of our quantitative meta analyses and many of the findings presented in Cochran-Smiths (1991) and Bangert-Downs (1993) summaries of research conducted prior to 1992. In general, research over the past two decades consistently finds that when students write on computers, writing becomes a more social process in which students share their work with each other. When using computers, students also tend to make revisions while producing, rather than after producing, text. Between initial and final drafts, students also tend to make more revision when they write with computers. In most cases, students also tend to produce longer passages when writing on computers. Early research consistently found large effects of computer-based writing on the length of passages and less consistently reported small effects on the quality of student writing. In contrast, although our meta-analyses of research conducted since 1992 found a larger overall effect size for the quantity of writing produced on computer, the relationship between computers and quality of writing appears to have strengthened considerably. When aggregated across all studies, the mean effect size indicated that on average students who develop their writing skills while using a computer produce written work that is .4 standard deviations higher in quality than those students who learn to write on paper. On average, the effect of writing with computers on both the quality and quantity of writing was larger for middle and high school students than for elementary school students.

Meta-Analysis: Writing with Computers 1992–2002

19

For educational leaders questioning whether computers should be used to help students develop writing skills, the results of our meta-analyses suggest that on average students who use computers when learning to write produce written work that is about .4 standard deviations better than students who develop writing skills on paper. While teachers undoubtedly play an important role in helping students develop their writing skills, the analyses presented here suggest that when students write with computers, they engage in the revising of their work throughout the writing process, more frequently share and receive feedback from their peers, and benefit from teacher input earlier in the writing process. Thus, while there is clearly a need for systematic and high quality research on computers and student learning, those studies that met the rigorous criteria for inclusion in our meta-analyses suggest that computers are a valuable tool for helping students develop writing skills.

Endnotes 1 The smallest grand sample size among the fourteen studies measuring “quantity of writing” was 12, while the largest

grand sample size was 136. This variation in sample size resulted in a mean inverse variance weight of 12.30 (SD = 8.75), and a range from 2.52 through 31.03. The two largest weights were slightly greater than two standard deviations above the mean in value, and therefore were winsorized down to the value of two standard deviations above the mean, 29.80.

Meta-Analysis: Writing with Computers 1992–2002

20

References Albertson, L. R. & Billingsley, F. F. (1997). Improving Young Writers’ Planning and Reviewing Skills while Story-Writing. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, March 24–28, 1997. Allen, G. & Thompson, A. (1994). Analysis of the Effect of Networking on ComputerAssisted Collaborative Writing in a Fifth Grade Classroom. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA, April 4–8, 1994. Allison, B. (1999). Facilitating Student Achievement in Writing through the Deft Employment of Computer Technology. Unpublished Master’s Action Research Project, Saint Xavier University/IRI. Baker E. & Kinzer, C.K. (1998). Effects of technology on process writing: Are they all good? National Reading Conference Yearbook, 47, 428–440. Bangert-Drowns, R. L. (1993). The word processor as an instructional tool: A meta-analysis of word processing in writing instruction. Review of Educational Research, 63(1), 69–93. Barrera, M. T., III, Rule, A. C., & Diemart, A. (2001). The effect of writing with computers versus handwriting on the writing achievement of first-graders. Information Technology in Childhood Education Annual, 28. Becker, H. J. (1999). Internet Use by Teachers: Conditions of Professional Use and Teacher-Directed Student Use. Irvine, CA: Center for Research on Information Technology and Organizations. Bennett, R.E. (2002). Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning and Assessment, 1(1). Retrieved November 1, 2002 from http://www.bc.edu/research/intasc/jtla/journal/ v1n1.shtml Biesenbach-Lucas, S. & Weasenforth, D. (2001). E-mail and word processing in the esl classroom: How the medium affects the message. Language Learning & Technology, 5(1), 135–165. Bogard, E. A. (1998). The Effects of Computer-Mediated Writing on the Quality and Quantity of Foreign Language Composing. Unpublished Doctoral Dissertation, University of South Florida. Borthwick, A. G. (1993). Effects of Keyboarding/Typewriting on the Language Arts Skills of Elementary School Students. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA, April 12–16, 1993. Bowman, M. (1999). Children, word processors and genre. Scottish Educational Review, 31(1), 66–83. Bracey, G. (1992). Creative writing: Do word processors help or hurt students’ creativity? Electronic Learning, 12(3), 12. Breese, C., Jackson, A., & Price, P. (1996). Promise in impermanence: Children writing with unlimited access to word processors. Early Child Development and Care, 118, 67–91.

Meta-Analysis: Writing with Computers 1992–2002

21

Brigman, D. J. P. (1995). The Effects of Writing with Pencil and Paper Compared with Writing with a Computer and a Word Processing Program on First-Grade Students Who Have Difficulty Writing. Unpublished Doctoral Dissertation, The Johns Hopkins University. Bruce, B. & Peyton, J. K. (1992). A Situated Evaluation of Computer Networking to Teach Writing. Technical Report No. 565. Urbana, IL: Center for the Study of Reading. Bucci, S. M. (1996). Seven at-Risk Fifth-Graders and Factors That May Influence the Quality of Their Written Text: From Paper and Pencil to Word Processed Drafts (Writing). Unpublished Doctoral Dissertation, University of Pittsburgh. Burley, H. (1994). Postsecondary Novice and Better Than Novice Writers: Effects of Word Processing and a Very Special Computer Assisted Writing Lab. Paper presented at the Southwestern Educational Research Association, San Antonio, TX, January 27–29, 1994. Butler, S. & Cox, B. (1992). Diskovery: writing with a computer in grade one: A study in collaboration. Language Arts, 69, 633–640. Casey, J. M. (1992). Writing to Read in the Classroom: A Literature-Based Writing Literacy Environment. Simi Star Project Report. (ED 367 952). Creskey, M. N. (1992). The Perspective of hindsight. International Journal of Instructional Media Vol. 19(1), 79–84. Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of related literature. Review of Educational Research, 61, 107–155. Cohen, J. (1977). Statistical power analysis for the behavioral sciences. (1st ed.) Hillsadale, NJ: Erlbaum. Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20 (May), p. 141–59. DeFoe, M. C. (2000). Using Directed Writing Strategies to Teach Students Writing Skills in Middle Grades Language Arts. Unpublished Ed.D. Practicum II Report, Nova Southeastern University. D’Odorico, L. & Zammuner, V. L. (1993). The influence of using a word processor on children’s story writing. European Journal of Psychology of Education, 8(1), 51–64. Dodson, L. E. (2000). The Effects of Using Technology to Enhance Student Ability to Read, Organize and Write Informative Text. Unpublished Master’s Thesis, University of Manitoba. Dybdahl, C. S., Shaw, D. G., & Blahous, E. (1997). The impact of the computer on writing: No simple answers. Computers in the Schools, 13(3), 41–53. Ediger, M. (1996). Middle School Pupil Writing and the Word Processor (ED393101). Escobedo, T. H. & Allen, M. (1996). Preschoolers’ Emergent Writing at the Computer. Paper Presented at the Annual Meeting of the American Educational Research Association, New Tork, NY, April 8–12, 1996. Etchison, C. (1989). Word processing: A helpful tool for basic writers. Computers and Composition, 6(2), 33–43.

Meta-Analysis: Writing with Computers 1992–2002

22

Fan, H.-L. & Orey, M. (2001). Multimedia in the classroom: its effect on student writing ability. Journal of Research on Technology in Education, 33(5). Fletcher, D. C. (2001). Second graders decide when to use electronic editing tools. Information Technology in Childhood Education Annual, 155–74. Freitas, C. V. & Ramos, A. (1998). Using Technologies and Cooperative Work to Improve, Oral, Writing, and Thinking Skills: Voices from Experience. Proceedings of the Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT). Gaddis, B., Napierkowski, H., Guzman, N., & Muth, R. (2000). A Comparison of Collaborative Learning and Audience Awareness in Two Computer-Mediated Writing Environments. Paper presented at the Annual Proceedings of Selected Research and Development Papers Presented at the National Convention of the Association for Educational Communications and Technology, 23rd, Denver, CO, October 25–28, 2000. Gallick-Jackson, S. A. (1997). Improving Narrative Writing Skills, Composition Skills, and Related Attitudes among Second Grade Students by Integrating Word Processing, Graphic Organizers, and Art into a Process Approach to Writing. Unpublished M.S. Practicum Project, Nova Southeastern University. Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8. Glass, G. V., McGaw, B., & Smith, M.L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: Sage Publications. Glennan, T. K. & Melmed, A. (1996). Fostering the use of educational technology: Elements of a national strategy. Santa Monica, CA: RAND. Godsey, S. B. (2000). The Effects of Using Microsoft Word[R] on Journal Word Counts in the High School English Classroom. Unpublished Master of Arts Action Research Project, Johnson Bible College. Greenleaf, C. (1994). Technological indeterminacy. The role of classroom writing practices and pedagogy in shaping student use of the computer. Written Communication, Vol. 11(1), 85–130. Grejda, G. F. & Hannafin, M. J. (1992). Effects of word processing on sixth graders’ holistic writing and revisions. Journal of Educational Research, 85(3), 144–149. Hagler, W. J. (1993). The Effects of the Word Processor on the Revision Behaviors of Sixth-Grade Students. Unpublished Doctoral dissertation, Auburn University. Hannafin, M. J. & Dalton, D. W. (1987). The effects of word processing on written composition. The Journal of Educational Research, 80 (July/Aug.) p. 338–42. Haas, C. & Hayes, J. R. (1986). Pen and paper versus the machine: Writers composing in hard-copy and computer conditions (CDC Technical Report No. 16). Pittsburgh, PA: Carnegie-Mellon University, Communication Design Center. Hartley, J. (1993). Writing, thinking, and computers. British Journal of Educational Technology, 24(1), pp. 22–31.

Meta-Analysis: Writing with Computers 1992–2002

23

Head, B. B. (2000). Revision Instruction and Quality of Writing by Eighth-grade Students Using Paper and Pencil or Word Processing. Unpublished Doctoral disertation, Oakland University. Hedges, L. (1981). Distributed theory for Glass’s estimator of effect size and related estimators, Journal of Educational Statistics, 6, 107–128. Hedges, L. & Olkin, I. (1985). Statistical Methods for Meta-Analysis. Orlando, FL: Academic Press. Hood, L. M. (1994). Effects of Computer Correspondence on Student Writing (Technical Report). Curry School of Education: University of Virginia. Hydrick, C. J. (1993). The Interaction of a Nine-Year-Old with a Word Processor. Unpublished Doctoral dissertation, Arizona State University. Jackiewicz, G. (1995). The Effect of Computer Based Instruction on Writing at the Elementary Level. Unpublished Master of Arts Thesis, Kean College of New Jersey. Jackowski-Bartol, T. R. (2001). The Impact of Word Processing on Middle School Students. Unpublished Master’s Thesis, Chestnut Hill College. Jankowski, L. (1998). Educational computing: Why use a computer for writing? Learning and Leading with Technology, 25(6), 30–33. Jones, I. (1994a). The effect of a word processor on the written composition of second-grade pupils. Computers in the Schools, 11(2), 43–54. Jones, I. (1994b). First-Grade Students’ Computer Supported Writing: Metacognitive and Linguistic Effects. Unpublished Doctoral Dissertation, University of Georgia. Jones, I. & Pellegrini A. D. (1996). The effects of social relationships, writing media, and microgenetic development on first-grade students’ written narratives. American Educational Research Journal, 33(3), pp. 691–718. Joram, E., Wooddruff, E., Bryson, M. & Lindsay, P. (1992). The effects of revising with a word processor on written composition. Research in the Teaching of English, 26(2), 167–193. Keetley, E. D. (1995). Comparison of First Grade Computer Assisted and Handwritten Process Story Writing. Unpublished Master’s Thesis, Johnson and Wales University. Kehagia, O. & Cox, M. (1997). Revision changes when using word processors in an english as a foreign language context. Computer Assisted Language Learning, 10(3), 239–253. Kerchner, L. B. & Kistinger, B. J. (1984). Language processing/word processing: Written expression, computers, and learning disabled students. Learning Disability Quarterly, 7(4), 329–335. Kumpulainen, K. (1994). Collaborative writing with computers and children’s talk: A cross-cultural study. Computers and Composition, 11(3), 263–273. Kumpulainen, K. (1996). The nature of peer interaction in the social context created by the use of word processors. Learning and Instruction, 6(3), 243–261. Kurth, R. J. (1987). Using word processing to enhance revision strategies during student writing activities. Educational Technology, 27(1), 13–19.

Meta-Analysis: Writing with Computers 1992–2002

24

Lam, F. S. & Pennington, M. C. (1995). The computer vs. the pen: a comparative study of word processing in a Hong Kong secondary classroom. Computer Assisted Language Learning, 8(1), 75–92. Langone, J., Levine, B., Clees, T.J., Malone, M., & Koorland, M. (1996). The differential effects of a typing tutor and microcomputer-based word processing on the writing samples of elementary students with behavior disorders. Journal of Research on Computing in Education, 29(2), 141–158. Lerew, E. L. (1997). The Use of Computers to Improve Writing Skills among LowAchieving Hispanic Students. Unpublished Doctoral dissertation, University of La Verne. Lewis, R. B. (1998). Enhancing the Writing Skills of Students with Learning Disabilities through Technology: An Investigation of the Effects of Text Entry Tools, Editing Tools, and Speech Synthesis. Final Report. San Diego, CA: Department of Special Education, San Diego State University. Lewis, R. B., Ashton, T. M., Haapa, B., Kieley, C. L., & Fielden, C. (1999). Improving the Writing Skills of Students with Learning Disabilities: Are Word Processors with Spelling and Grammar Checkers Useful? Learning Disabilities: A Multidisciplinary Journal, 9(3), 87–98. Lichtenstein, N. (1996). The Effect of Word Processing on Writing Achievement. Unpublished Master’s Project, Kean College of New Jersey. Lipsey, M. W. W. & Wilson, D.B. (2001). Practical Meta-Analysis (Vol. 49). Thousand Oaks, CA: Sage Publications. Lipsey, M.W. (1992) ‘Juvenile delinquency treatment: a meta-analytic inquiry into the variability of effects.’ In T.D. Cook, H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.J. Light, T.A. Louis and F. Mosteller (Eds) Metaanalysis for explanation. New York: Russell Sage Foundation. Lohr, L., Ross, S. M., & Morrison, G. R. (1996). Using a hypertext environment for teaching process writing: An Evaluation study of three student groups. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology (18th, Indianapolis, IN, 1996). Lomangino, A. G., Nicholson, J., & Sulzby, E. (1999). The Nature of Children’s Interactions while Composing Together on Computers. CIERA Report. Ann Arbor, MI: Center for the Improvement of Early Reading Achievement. Lowther, D. L., Ross, S. M., Morrison, G. R. (2001). Evaluation of a laptop program: Successes and recommendations. Paper presented at the National Educational Computing Conference Proceedings (22nd, Chicago, IL, June 25–27, 2001). Lund, D. M. & Hildreth, D. (1997). Integrating the computer into language arts in a fifth grade classroom: A Developing instructional model. Paper presented at the annual meeting of the National Reading Conference (47th, Scottsdale, AZ, December 3–6, 1997). MacArthur, C.A., Graham, S., & Schwartz, S.S. (1994). Peers + word processing + strategies = a powerful combination for revising student writing. TEACHING Exceptional Children, 27(1), 24–29.

Meta-Analysis: Writing with Computers 1992–2002

25

MacArthur, C. A. (1996). Using technology to enhance the writing processes of students with learning disabilities. Journal of Learning Disabilities, 29(4), 344–354. McBee, D. (1994). The Effect of technology on emergent writing. Research project, Juneau, AK: University of Alaska Southeast. McMillan, K. & Honey, M. (1993). Year One of Project Pulse: Pupils Using Laptops in Science and English. A Final Report. Technical Report No. 26. New York, NY: Center for Technology in Education. Market Data Retrieval. (1999). Technology in Education 1999. (A report issued by Market Data Retrieval). Shelton, CN: Market Data Retrieval. Mead, A. D. & Drasgow, (1993). Equivalence of computerized and paper-andpencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114:3, 449–58. Mehdi, S. N. (1994). Word Processing and the EFL Writing Classroom: A Case Study of Arabic Speakers. Unpublished Doctoral dissertation, Boston University. Moeller, B., Bell, L. & Reich, C. (1993). Using networked technology to create new writing environments for deaf students: Teachers’ strategies and student outcomes. Paper presented at the meeting of the Society for Research in Child Development in New Orleans, LA, March 25–28, 1993. Moher, D. & Olkin, I. (1995). Meta-analysis of randomized controlled trials: A concern for standards. Journal of the American Medical Association, 274, 1962–1963. Mott, M.S. & Halpin, R. (1999). A Comparative Approach to the Examination of the Technical Qualities of a Pen and Paper Writing Assessment for Elementary Students’ Hypermedia-Created Products. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Quebec, Canada, April 19–23, 1999. Mott, M. S., Sumrall, W. J., & Hodges, M. L. (1997). Process and Computer-Based Elementary Writing Curriculum: A Review of Methods and Assessments. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, Memphis, TN, November 12–14, 1997. Mott, M. S. & Klomes, J. M. (2001). The synthesis of writing workshop and hypermedia-authoring: Grades 1–4. Early Childhood Research & Practice, 3(2). Moxley, R. A., Warash, B., Coffman, G., Brinton, K., & Concannon, K. R. (1994). Computer writing development in a prekindergarten class of 4 year olds. Journal of Computing in Childhood Education, 5(2), 211–229. Moxley, R. A., Warash, B., Coffman, G., Brinton, K., & Concannon, K. R. (1997). Writing development using computers in a class of three-year-olds. Journal of Computing in Childhood Education, 8(2), 133–164. Nichols, L. M. (1996). Pencil and paper versus word processing: a comparative study of creative writing in the elementary school. Journal of Research on Computing in Education, 29(2), 159–166. Olson, K. A. (1994). Writing and Revising with Pencils and with Computers: An Analysis of the Processes and Products of Seven First-Grade Children. Unpublished Doctoral dissertation, University of Michigan.

Meta-Analysis: Writing with Computers 1992–2002

26

Orwin, R. G. (1983). A fail safe N for effect size in meta-analysis. Journal for Educational Statistics, 8, 157–159. Osborne, P. (1999). Pilot study to investigate the performance of dyslexic students in written assessments. Innovations in Education and Training International; 36(2), 155–60. Owston, R. D. (1991). Effects of word processing on student writing in a highcomputer-access environment (Technical Report 91–3). North York, Ontario: York University, Centre for the Study of Computers in Education. Owston, R. D., Murphy, S., & Wideman H. H. (1992). The effects of word processing on students’ writing quality and revision strategies. Research in the Teaching of English, 26(3), 249–276. Owston, R. D. & Wideman, H. H. (1997). Word processors and children’s writing in a high-computer-access setting. Journal of Research on Computing in Education, 30(2), 202–220. Padgett, A. L. (2000). Journal Writing in the Elementary School: Word Processor vs. Paper and Pencil. Unpublished Master of Arts Research Paper, Johnson Bible College. Peacock, M. & Beard, R. (1997). “Almost an Invincible Repugnance?”: Word processors and pupil writers. Educational Review, 49(3), 283–294. Pennington, M. C. (1993). Exploring the potential of word processing for nonnative writers. Computers and the Humanities, 27(3), 149–163. Peterson, S. E. (1993). A comparison of student revisions when composing with pen and paper versus word-processing. Computers in the Schools, 9(4), 55–69. Philips, D. (1995). Study 6: Using the Word Processor To Develop Skills of Written Expression. Final Report. Evaluation of Exploratory Studies in Educational Computing. Wellington, NZ: New Zealand Council for Educational Research. Phoenix, J. & Hannan, E. (1984). Word processing in the grade 1 classroom. Language Arts, 61(8), 804–812. Pisapia, J. R., Knutson, K., & Coukos, E. D. (1999). The Impact of computers on student performance and teacher behavior. Paper presented at the annual meeting of the Florida Educational Research Association (44th, Deerfield Beach, FL, November 10–12, 1999). Pohl, V. & Groome, D. (1994). Happy hands: Pre-keyboarding facility for emergent writers. The Computing Teacher, 21(5), 16–18. Priest, N. B. (1995). Developing writing skills of fourth grade students using a variety of intervention strategies. M.S. Practicum, Nova University. Reed, W. M. (1996). Assessing the impact of computer-based writing instruction. Journal of Research on Computing in Education, 28(4), 418–437. Robertson, I., Calder, J., Fung, P., Jones, A., O’Shea, T., & Lambrechts, G. (1996). Pupils, teachers, & palmtop computers. Journal of Computer Assisted Learning, 12(4), 194–204. Roussey, J. Y., Farioli, F., & Piolat, A. (1992). Effects of social regulation and computer assistance on the monitoring of writing. European Journal of Psychology of Education; 7(4), 295–309.

Meta-Analysis: Writing with Computers 1992–2002

27

Roblyer, M. D. (1997). Technology and the oops! effect: Finding a bias against word processing. Learning and Leading with Technology, 24(7), 14–16. Seawel, L., Smaldino, S.E., Steele, J.L., & Lewis, J.Y. (1994). A descriptive study comparing computer-based word processing and handwriting on attitudes and performance of third and fourth grade students involved in a program based on a process approach to writing. Journal of Computing in Childhood Education, 5(1), 43–59. Shadish, W.R. & Haddock, C.K. (1994). Combining Estimates of Effect Size. In H.M. Cooper & L.V. Hedges (Eds.), The Handbook of Research Synthesis (pp. 261–281). New York: Russell Sage Foundation. Shadish, W.R., Cook, T. D., & Campbell, D. T. (2001). Experimental and QuasiExperimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin Company. Shaw, E.L., Nauman, A.K., & Burson, D. (1994). Comparison of spontaneous and word processed compositions in elementary classrooms: A three-year study. Journal of Computing in Childhood Education, 5(3), 319–327. Sitko, M.C. & Crealock, C. M. (1986, June). A longitudinal study of the efficacy of computer technology for improving the writing skills of mildly handicapped adolescents. Paper presented at at the Invitational Research Symposium on Special Education Technology, Washington, DC. Snyder, I. (1993a). The impact of computers on students’ writing: A comparative study of the effects of pens and word processors on writing context, process and product. Australian Journal of Education, 37(1), 5–25. Snyder, I. (1993b). Writing with word processors: A research overview. Educational Research, 35(1), 49–68. Snyder, I. (1994). Writing with word processors: The computer’s influence on the classroom context. Journal of Curriculum Studies, 26(2), 143–162. Technology and Assessment Collaborative (inTASC; in press). Introduction to the USEIT Study (Research Report No. 1). Chestnut Hill, MA: Author. U.S. Department of Education. (2002). No Child Left Behind Act of 2001. Retrieved May 29, 2002, from http://www.ed.gov/legislation/ESEA02/ U.S. Department of Commerce. (2002). A Nation Online: How Americans are expanding their use of the internet. Washington, DC: Author. Retrieved November 1, 2002, from http://www.ntia.doc.gov/ntiahome/dn/nationonline_ 020502.htm. Vacc, N. N. (1987). Word processor versus handwriting: A comparative study of writing samples produced by mildly mentally handicapped students. Exceptional Children, 54(2), 156–165. Waldman, H. (1995). The Effects of a Multimedia Literacy Tool on First Grade Reading and Writing Achievement. Unpublished Doctoral dissertation, The University of San Francisco. Walker, C. L. (1997). Computers, revision, and the developmental writer: A case study of on-screen versus off-screen revision processes. Research and Teaching in Developmental Education, 14(1), 23–34.

Meta-Analysis: Writing with Computers 1992–2002

28

Williamson, M. L. & Pence, P. (1989). Word processing and student writers. In B. K. Briten & S. M. Glynn (Eds.), Computer Writing Environments: Theory, Research, and Design (pp. 96–127). Hillsdale, NJ: Lawrence Erlbaum & Associates. Wolf, F. M. (1986). Meta-Analysis: Quantitative Methods for Research Synthesis. Thousand Oaks, CA: Sage Publications. Wolfe, E. W., Bolton, S., Feltovich, B., & Bangert, A.W. (1996). A study of word processing experience and its effects on student essay writing. Journal of Educational Computing Research, 14(3), 269–283. Yackanicz, L. (2000). Reluctant Writers and Writing-Prompt Software. Unpublished Master’s thesis, Chestnut Hill College. Zammuner, V. L. (1995). Individual and cooperative computer-writing and revising: Who gets the best results? Learning and Instruction, 5(2), 101–124. Zhang, Y., Brooks, D.W., Frields, T., & Redelfs, M. (1995). Quality of writing by elementary students with learning disabilities. Journal of Research on Computing in Education, 27(4), 483–499. Zoni, S. J. (1992). Improving Process Writing Skills of Seventh Grade At-Risk Students by Increasing Interest through the Use of the Microcomputer, Word Processing Software, and Telecommunications Technology. Unpublished Practicum Paper, Nova University.

Meta-Analysis: Writing with Computers 1992–2002

Appendix A 1)

Publication Type (one variable) • • • • • •

2)

Refereed journal article Conference presentation Manuscript under journal review Doctoral dissertation Master’s thesis Research organization study/technical report

Research Methodology (eight variables, unless otherwise indicated, dichotomous: yes/no) • • • • •

random assignment of students direct comparison to paper/handwritten writing presence of pre- and post-test standardized/controlled writing conditions intervention time/duration of study − less than six weeks − between six weeks and one semester − more than one semester • sample size − thirty or less − between thirty-one and one-hundred − more than one-hundred • In the case of handwritten samples: were they converted to computerized form to ensure blindness of scorers/raters? • other indicators of sound design (i.e., treatment vs. control groups, counterbalanced design, absence of confounding variables, etc.)

3)

Student Characteristics (six variables) • Grade level − elementary − middle − secondary • Gender description − heterogeneous − homogeneous • Race/ethnic description − heterogeneous − homogeneous

29

Meta-Analysis: Writing with Computers 1992–2002

• School-setting − Rural − Suburban − Urban • Type of students − Mainstream − SPED/At-risk − Gifted − ESL/ESOL • Writing ability of students − Low − Average − High

4)

Technology-related Factors (seven variables) • Type of hardware used • Type of software used • Description of students’ prior keyboarding skills − no mention − minimal − adequate − advanced • Description of students’ prior word-processing skills − no mention − minimal − adequate − advanced • Keyboarding training provided as part of study (yes/no) • Word processing training provided as part of study (yes/no) • Technological assistance provided to students during study (yes/no)

5)

Writing Environment Factors (two variables) • Writing within Language Arts/English discipline? • Type of student writing − Collaborative − Individual

6)

Instructional Factors (six variables; all dichotomous: yes/no) • • • • • •

Did students receive writing instruction during the intervention period? Receipt of teacher-feedback/editing Receipt of peer-feedback/editing Were students allowed to revise without any kind of feedback Internet or distance editors Did students make use of spell-checkers?

30

Meta-Analysis: Writing with Computers 1992–2002

7)

Outcome Measures (three variables) • Quantity of writing − Number of words − Number of t-units − Number of sentences • Quality of writing − Holistic, judgmental (no rubric) − Mechanics, rubric − Grammar, rubric − Style, rubric • Revision of writing − Number of revisions − Nature of revisions

31

Meta-Analysis: Writing with Computers 1992–2002

32

Appendix B: Results Computers and Writing: Quantity Fourteen independent effect sizes were extracted from fourteen studies that compared quantity of writing, as measured by word count, between computer and paperand-pencil groups. Below we present descriptive highlights of the fourteen studies followed by an analyses of effect sizes, and regression analyses that explore moderating variables.

Descriptive Highlights As detailed in Table B1, 64% of the studies (n = 9) were published in refereed journals, 14.3% (n = 2) employed random assignment, and more than half (n = 8) sampled from multiple classrooms. For 57% of the studies (n = 8), the research duration lasted between six weeks and one semester, and 86% (n =12) utilized standardized writing tasks across groups. In 43% (n = 6) of the studies, students were provided with keyboarding training. Individual writing (as opposed to collaborative writing) was the focus in all fourteen studies, and peer editing, teacher feedback, and technical assistance were available to students in 21% (n = 3) of the studies. It was inconclusive whether or not teacher feedback and/or technical assistance were study features in n =5 and n=9 studies, respectively. With respect to student demographics, only three studies (21%) provided sufficient information that indicated that the sample was gender diverse and four studies (29%) indicated that they had racially/ethnically-diverse student samples. Over half of the studies did not provide sufficient information about the participating students to classify their gender or racial/ethnic diversity. All but two studies (n =12) focused on mainstream education samples, and half (n =7) of the studies were conducted with elementary school students. Finally, two studies occurred in rural, three in urban, and four in suburban settings, while the three studies lack any geographic information.

Meta-Analysis: Writing with Computers 1992–2002

Table B1:

33

Characteristics of Studies Included in Quantity of Writing Meta-analysis n of studies (%)

Study Characteristics Refereed journal article Publication type

Doctoral dissertation

Master’s thesis

9 (64.3%)

3 (21.4%)

Yes

No

No information

Random assignment

2 (14.3%)

12 (85.7%)

—

Pre-Post design

6 (42.9%)

8 (57.1%)

—

12 (85.7%)

2 (14.3%)

—

Keyboarding training included in study

6 (42.9%)

6 (42.9%)

Peer-editing

3 (21.4%)

11 (78.6%)

—

Handwritten samples converted to WP format

3 (21.4%)

11 (78.6%)

—

Technical assistance provided to students

3 (21.4%)

2 (14.3%)

9 (64.3%)

Teacher’s feedback on provided to students

3 (21.4%)

6 (42.9%)

5 (35.7%)

Standardized writing sample

2(14.3%)

2 (14.3%)

n of studies (%)

Sample Characteristics Yes

No

Sample described demographically

6 (42.9%)

8 (57.1%)

Gender-diverse

4 (28.6%)

1 (7.1%)

9 (64.3%)

Racially/Ethnically diverse

3 (21.4%)

1 (7.1%)

10 (71.4%)

Prior keyboarding skill

7 (50%)

1 (7.1%)

6 (42.9%)

Sampling – school-level Sampling – classroom-level

Grade level

Multiple

11 (78.6%)

3 (21.4%)

—

6 (42.9%)

8 (57.1%)

—

School setting

No information

Between six weeks and one semester

One semester or longer

6 (42.9%)

6 (42.9%)

2 (14.3%)

Elementary

Middle

High

3 (21.4%)

4 (28.6%)

7 (50%) High

Student sample ability level

—

Single

Less than six weeks Length of study

No information

Average

Low

Mixed

3 (21.4%) 2 (14.3%) 1 (7.1%) 4 (28.6%) SubRural Urban urban Mixed 2 (14.3%) 3 (21.4%) 4 (28.6%) 2 (14.3%)

No Information 4 (28.6%) No Information 3 (21.4%)

Meta-Analysis: Writing with Computers 1992–2002

34

Publication Bias: Funnel Plot The funnel plot depicted in Figure B1 shows that nearly two-thirds of effect size findings are approximately .50 or greater. Smaller-sized studies demonstrated a wide range of effect sizes, from virtually no effect at all through upwards of 1.2 units, as do the five largest studies (those with sample sizes greater than 50; ranging from -.05 to .87). Striking from the funnel plot, however, is the dearth of studies that employed a sample size greater than 50; exactly half of the studies had sample sizes that were 30 or fewer.

Figure B1:

Funnel Plot for Quantitative Meta-Analysis 1.4 1.2

Adjusted Effect Size

1.0 .8 .6 .4 .2 0.0 -.2 0

10

20

30

40

50

60

70

80

Actual Sample Size

Weighted Effect Sizes and Homogeneity Analysis The overall effect of computers, as compared with paper-and-pencil, on quantity of student writing, based on twelve independent effect sizes, extracted from twelve studies, resulted in a mean effect size of .501. The weighted mean effect size, d+ = 4.5226, with a 95% confidence interval ranging from .1.8187 through 7.2265. Individual weighted effect sizes ranged from –1.62 through 11.97. The homogeneity analysis resulted in Qt = 4120.6571, df = 13, p < .0001. This significant Qt statistic indicates that the fourteen effect sizes comprising this analysis do not come from the same population and that there may be moderating variables that impact the magnitude and/or direction of the effect sizes

Meta-Analysis: Writing with Computers 1992–2002

35

In order to identify which, if any, of the coded study features have a significant moderating effect on the relationship between computers and quantity of writing, regression analyses were performed.

Weighted Effect Sizes and Regression Analysis To examine the extent to which effect sizes were moderated by various study features, a mixed model approach was employed. This approach assumes that some of the variance in the effect sizes is systematic, and thus can be modeled, while another portion of the variance in the effect sizes is random and, therefore, cannot be modeled (for a full discussion of mixed vs. fixed effects modeling in the context of meta-analyses, see Hedges & Olkin, 1985). The first step in the analysis was to dummy code all of the categorical variables. Dichotomous variables were left as is, variables with three or more levels were transformed into a series of k-1 (where k is the number of levels in the original variable) dichotomous variables. In the process of creating these variables, categories within the “school level” and “student ability” variable were collapsed. The dummy variable for “school level” was created to compare studies conducted in elementary schools with those conducted in middle and high school combined. Student ability was transformed to contrast the “average” and “mixed” groups with the remaining groups (low, high, no information provided). To identify those variables with sufficient variance between levels required by the matrix algebra used in regression analysis, frequencies of each dummy variable were examined. In general, if each level of a given variable had a frequency of three or more, then it could be successfully entered in the regression analysis. Of the coded study features, eight variables met the criterion for sufficient variance. These eight variables were grouped into two themes: • “student support,” which included: keyboard training, technical assistance, teacher feedback, and peer editing • “student sample characteristics” which included: keyboard experience prior to study, student ability, school setting, and school level. Additionally, a regression analysis that focused on the “study’s methodological quality” was conducted. For this analysis, methodological quality was calculated by summing assigned points across the 12 variables related to study quality, for a total possible score of 16. As presented below, the aggregate methodological quality rating was not a significant predictor. To explore whether individual aspects of study quality moderated the reported effects, the following study features were entered separately into a regression model: type of publication, presence of control group, presence of pre-post design, length of study, multiple vs. single classrooms and multiple vs. single schools. Additionally, publication year was dummy coded and included along with these methodological variables. To dummy code publication year, studies were divided into two groups: those published between 1992–1995, inclusive, and those published after 1995.

Meta-Analysis: Writing with Computers 1992–2002

36

Study Features with Moderating Effects on Quantity of Writing Regression Model: Student Support in Writing Table B2 shows the results of the “student support” multiple regression model. In total, the four predictors, entered in a single block, accounted for 6% of the variance. None of the individual predictors were significant. These results suggest that the various types of support provided to students during the course of each study did not systematically affect the amount of writing students produced.

Table B2:

Regression Analysis of Student Support Variables on Weighted Effect Sizes of Quantity of Writing

Constant Keyboard training Teacher feedback Peer editing Technical assistance

B 3.7541 1.9465 -.6918 -.5389 .9379

SE 2.4246 3.5734 6.3335 5.2453 5.0348

-95% CI -.9980 -5.0574 -13.1053 -10.8198 -8.9302

+95% CI 8.5062 8.9503 11.7218 9.7419 10.8061

Z 1.5484 .5447 -.1092 -.1027 .1863

P .1215 .5860 .9130 .9182 .8522

Mean ES 4.5250

R-Square .0598

Beta .0000 .2133 -.0629 -.0490 .0853

QM = .4615, p = .9771 N 14.0000

Regression Model: Student Sample Characteristics Table B3 presents the results of the “student sample characteristics” multiple regression model. Although the five variables collectively accounted for over 52% of the variance, no variables were found to be significant predictors. One variable, school level, approached significance. As the variable was dummy coded, the large, positive Beta indicates that studies employing middle and high school student samples tended to demonstrate greater effect sizes than did those studies employing elementary school samples. These results suggest that the characteristics of students participating in each study was not systematically related to the amount of writing students produced.

Table B3:

Regression Analysis of Student Sample Characteristics Variables on Weighted Effect Sizes of Quantity Of Writing

Constant Prior keyboarding skills Geographic setting School level

B 1.0722 -.1672

SE 3.3559 3.5419

-95% CI -5.5055 -7.1094

+95% CI 7.6498 6.7750

Z .3195 -.0472

P .7494 .9623

Beta .0000 -.0184

-1.1249 6.9368

3.7626 3.7644

-8.4996 -.4414

6.2499 14.3150

-.2990 1.8427

.7650 .0654

-.1235 .7616

Mean ES 4.3386

R-Square .5264

QM = 3.5892, p = .3094 N 9.0000

Meta-Analysis: Writing with Computers 1992–2002

37

Regression Model: Study Methodology Table B4 presents the results of the “study methodology” multiple regression model. Although the five variables collectively accounted for 33% of the variance, none of the predictors were statistically significant. These results suggest that the various features of the studies were not systematically related to the amount of writing students produced.

Table B4:

Regression Analysis of Study Methodology Variables on Weighted Effect Sizes of Quantity of Writing

Constant Publication year Type of publication Control group design Pre-post design Length of study School level Single or multiple classes

B 2.2700 .8193 -3.1169 3.2659 2.0379 1.9022 -.5315 1.2208

SE 4.1183 3.7754 4.3803 4.7969 5.2493 3.9179 5.3005 4.2170

-95% CI -5.8018 -6.5804 -11.7023 -6.1360 -8.2506 -5.7769 -10.9205 -7.0445

+95% CI 10.3418 8.2190 5.4686 12.6678 12.3265 9.5813 9.8574 9.4861

Z .5512 .2170 -.7116 .6808 .3882 .4855 -.1003 .2895

P .5815 .8282 .4767 .4960 .6978 .6273 .9201 .7722

Mean ES 4.5248

R-Square .3294

Beta .0000 .0898 -.3307 .3617 .2234 .2086 -.0483 .1338

QM = 18.9594, p < .0003 N 14.0000

Sensitivity Analysis When heterogeneity among effect sizes are found in a meta-analysis, the “robustness” of the main findings can be examined through sensitivity analyses (Lipsey & Wilson, 2001). The sensitivity analysis explores ways in which the main findings are either consistent or inconsistent in response to varying the ways in which the data have been aggregated or included in the overall meta-analysis. For example, to provide a sense of how sensitive the main findings are across subgroups (say of school level), sensitivity analyses focus on a particular level of a variable. A key variable of interest in this analysis is length of study. It can be reasonably argued that in studies of short duration (i.e., six weeks or less) measuring the impact of using computers on students’ writing is different than measuring computers’ impact on writing over a longer period of time. Studies conducted under longer time periods can result in students who are more adept at keyboarding, are more comfortable with features of word processing programs, and have sufficient time to adapt their writing strategies to exploit features of word processors. Considering this, a sensitivity analysis was conducted which focused only on those studies for which the length of intervention was greater than six weeks. This selection procedure eliminated six of the fourteen studies from the analysis.

Meta-Analysis: Writing with Computers 1992–2002

38

Sensitivity Analysis: Longer Study Duration and Student Support Multiple Regression Model Due to insufficient variance on the other student support variables, the sensitivity analyses focused on peer editing and keyboard training, only. The resulting model (R 2 = .038) consisting of these two independent variables was statistically insignificant. These statistics indicate that there is no relationship between the weighted effect sizes of quantity of writing and these variables, regardless of length of study.

Sensitivity Analysis: Longer Study Duration and Student Sample Characteristics Multiple Regression Model As shown in Table B5, the overall student sample characteristics multiple regression model, (excluding student academic ability) was significant (QM = 18.9594, p = .0003). One variable, school level, was significantly related to the weighted effect sizes. For those studies that lasted for more than six weeks, the significant, positive beta weight for school level indicates that there were larger effect sizes for quantity of writing that favored computers over paper and pencil for studies that occurred in middle and high school as opposed to elementary school.

Table B5:

Sensitivity Analysis: Regression Analysis of Student Sample Characteristics Variables on Weighted Effect Sizes of Quantity of Writing for Studies Lasting More Than Six Weeks

Constant Prior keyboarding Experience Geographic setting School level

B .1617 .8231

SE 2.2657 2.9326

-95% CI -4.2791 -4.9248

+95% CI 4.6025 6.5710

Z .0714 .2807

P .9431 .7790

Beta .0000 .0796

.5385 9.3045

2.6193 2.9334

-4.5954 3.5550

5.6724 15.0540

.2056 3.1719

.8371 .0015

.0521 .9020

Mean ES 4.0151

R-Square .9167

QM =18.9594, p