The WEB Project is a five-year Technology Innovation Challenge Grant that was completed in September 2000. The purpose of this project was to infuse standards-based instruction in multimedia, digital art, music composition, and online discourse into the general arts and humanities curricula of Vermont K-12 schools. Multimedia technology was incorporated within six academic content areas: art, music, technology, history/social studies, English/language arts, and interdisciplinary studies.

Students shared their works-in-progress with a virtual community consisting of other students, teachers, digital artists, traditional artists, musicians, composers, Web page designers and other experts. This was done via a virtual learning environment called The WEB Exchange, which resided on The WEB Project's server. Through threaded design conversations, students requested feedback on their works of art, music, and multimedia.

They then filtered the feedback they received and used it to improve their final artistic products, which many of the participating students then posted on The WEB Exchange.

Concurrently, language arts students, aided by language arts teachers and mentors from the Vermont Center for the Book, discussed curriculum-related texts. Moderating their own discussions, students engaged in deep, rich dialogue that focused on standards-based activities such as responding to text, substantiating arguments with evidence found in the text, informed decision-making, etc. These interventions were stable over the last two to three years of the five-year grant.

Measuring the Project's Impact

One of the research questions posed by the RMC Research Corporation evaluation team in evaluating this project was, "What is the impact of The WEB Project on student achievement?" Our intent was to generalize our methodology to other instructional technology grants in which student achievement must be reported. Findings from an online survey of The WEB Project teachers and administrators, repeated in spring 1998, 1999, and 2000, indicated that a connection between student motivation, metacognition, and learning processes, as outlined in a conceptual model developed by Sternberg (1998), might be applicable.

According to Sternberg, motivation drives metacognition, which, in turn, stimulates the development of thinking and learning skills. Thinking and learning skill development further stimulates metacognition, resulting in the development of expertise. The evaluation team extended the Sternberg Developing Expertise model to define "expertise" as student achievement measured by teacher-created rubrics. Participating teachers developed, refined, and benchmarked rubrics for student-created products over the past three years.

Teachers also selected a rubric for measuring student learning processes from Marzano et al.'s (1993) Dimensions of Learning model. The rubric addressed the depth and richness of revisions to student-created products and performances.

The Survey

Using mixed methods that consisted of the online survey, student pretest and posttest surveys, and scores on teacher-created/selected rubrics that assessed students' learning processes and final products, the evaluation team used structural equation modeling to correlate the various elements of the extended Sternberg model. The hypothesis was that motivation would drive metacognition, and that metacognition would drive thinking and learning processes (specifically, inquiry learning and application of skills; two scales derived from WEB Project-based activities). Thus, increases in thinking and learning processes would result in increases in teacher-scored measures of student achievement.

The student survey was pilot-tested in spring 1999, and three derived scales (metacognition, inquiry learning, and application of skills) had high internal consistency (alpha = .72 to .84). Two ten-item sets of questions for "in this class" and "in school in general" motivation were added to the survey in spring 2000.

In January 2000, the survey was administered to 165 students in nine cooperating schools. One-hundred and thirty-seven responses were from students who had not yet been exposed to the intervention, and could therefore be used as pretests. Internal consistency and reliability for all scales (class motivation, school motivation, metacognition, inquiry learning, and application of skills) ranged from alpha = .70 to alpha = .87. In May 2000, at the end of the spring term, the survey was re-administered as a posttest to the same group of students. As of August 2000, 131 completed surveys were returned by all nine schools.

About 75% of the students who responded were from high schools, and 25% were from middle schools. Gender was about equally distributed.

Seventy-six valid data sets were matched in order to conduct a true repeated measures methodology (pretest vs. posttest). Only the "application of skills" scale increased during the spring term (2-tailed significance = .0165).

For the path analysis, the posttest survey results were correlated with teacher assessments. Participating teachers assigned a "product" score of "0" (no evidence), "1" (approaches standards), "2" (meets standards), and "3" (exceeds standards) to their students' final products. Products were re-scored by a jury of experts to increase reliability, resulting in 91 reported "product" scores. One-hundred and seven teachers assigned a "process" score of "1" (low) to "4" (high) to each of their participating students for the quality and depth of revisions of their final products, which they construed as a measure of student learning processes. These data constituted two independent measures of student achievement, which served to complete the model.

Four separate simplified path analysis models were tested. The first pair addressed process and product outcomes for class motivation, and the second pair addressed school motivation. The statistically significant (p < .05) results were as follows:

Motivation was related to metacognition. The relationship between class motivation and metacognition was slightly stronger (R = .307, p < the relationship between school motivation and metacognition (R = .282, p < .0001). The relationship between metacognition and inquiry learning (Beta = .546, p < .0001) was stronger than the relationship between metacognition and application of skills (Beta = .282, p < .0001).

The relationship between inquiry learning and the student learning process outcome (Beta = .384, p = .001) was stronger than the relationship between application of skills and the student learning process outcome (Beta = -.055, not significant). The relationship between application of skills and the student product outcome (Beta = .371, p = .004) was stronger than the relationship between inquiry learning and the student product outcome (Beta = .063, not significant).

Clearly, correlation does not imply causality. However, when each of these elements was considered as an independent variable, there was a corresponding change in associated dependent variables. For example, there was a significant correlation between motivation and metacognition, indicating that students' enthusiasm for learning with technology may stimulate students' metacognitive (strategic) thinking processes. The significant correlations between motivation, metacognition, inquiry learning, and the student learning process score indicate that motivation may drive increases in the four elements connected by the first path. Similarly, the significant correlations between motivation, metacognition, application of skills, and the student product score indicate that motivation may drive increases in the four elements connected by the second path.

Based on the significant correlations of the two teacher measurements of student achievement with the student survey data, these data validated the evaluation team's extension of the Developing Expertise model to explain increases in student performance as a result of engaging in technology-supported learning activities. Moreover, nearly all students across the project met the standards for both the teacher-created student product assessment and the learning process assessment. This indicates that, in general, the project had a positive impact on student achievement.

Conclusions

These preliminary findings suggest that teachers should emphasize the use of metacognitive skills, application of skills, and inquiry learning as they infuse technology into their respective academic content areas. Moreover, these activities are directly in line with the Vermont Reasoning and Problem Solving Standards, and with similar standards in other states. The ISTE/NETS standards for assessment and evaluation also suggest that teachers:

Apply technology in assessing student learning of subject matter using a variety of assessment techniques. Use technology resources to collect and analyze data, interpret results, and communicate findings to improve instructional practice and maximize student learning.

Apply multiple evaluation methods to determine students' appropriate use of technology resources for learning, communication and productivity.

Rockman (1998) suggests that "A clear assessment strategy that goes beyond standardized tests enables school leaders, policymakers, and the community to understand the impact of technology on teaching and learning." RMC Research Corporation's extension of the Sternberg model can be used to organize and interpret a variety of student self-perceptions, teacher observations of student learning processes, and teacher-scored student products. It captures the overlapping kinds of expertise that students developed throughout their technology-related activities.

One of the greatest challenges facing the Technology Innovation Challenge Grants and the Preparing Tomorrow's Teachers To Use Technology (PT3) grants is to make a link between educational technology innovations, promising practices for teaching and learning with technology, and increases in student achievement. We believe that this model may be replicable in other educational institutions, including schools, districts, institutions of higher learning, and grant-funded initiatives. However, to use this model, participating teachers must be able to clearly identify the standards they are addressing in their instruction, articulate the specific knowledge and skills that are to be fostered by using technology, carefully observe student behavior in creating and refining their work, and create and benchmark rubrics that they intend to use to evaluate student work.

Lorraine Sherry
Shelley Billig
Daniel Jesse
Deborah Watson-Acosta
RMC Research Corporation, Denver

References

Marzano., R.J., Pickering, D., & McTighe, J. 1993. Assessing student outcomes:
Performance assessment using the Dimensions of Learning Model. Littleton CO: McREL Institute.

Rockman, S. 1998. Communicating our successes: Issues and tactics. Unpublished manuscript.

Sternberg, R.J. April, 1998. "Abilities Are Forms of Developing Expertise." Educational Researcher, 27 (3),11-20.

©2000 All Rights Reserved T.H.E. Journal, L.L.C.

Do Laptops Help Increase Test Scores?

This is an article taken from the Curriculum Administrator, February 2001 by Vicky Bigham.

The short answer is "yes", standardized test scores do seem to rise for students who use laptops regularly. Like so many things in teaching, it is hard to isolate one variable. Consider the following statistics:

· According to the US Department of Labor's SCANS report and the International Society for Technology in Education's National Educational Standards for Standards for Students, the higher order thinking skills are highly valued in academic as well as in business environments. The students who had 24-hour access to laptops were more likely to pursue more varied types of data.

· "Does It Compute?," by the Educational Testing Service, is perhaps the first study to document the relationships between student use of technology across the nation and higher scores on a national standardized tests. In general, the 8th grade students whose teachers used computers mostly for simulations and applications performed better on the National Assessment of Educational Progress. Students whose teachers used it primarily for drill and practice performed worse. The strongest message in this study was the unquestionable importance of professional development. In both grades studied in this report, students whose teachers had professional development regarding how to use computers outperformed students whose teachers did not.

· The Idaho Council for Technology in Learning was created in 1994 to oversee the state's technology initiative and spending of an annual appropriation of $10.4 million to state public schools. The ICTL commissioned a legislative study to assess the impact of funds spent to date and found that students with frequent access to computers were 2.4 months ahead, academically, than those with limited access.

· Since 1990, West Virginia's Basic Skills/Computer Education Program has placed more than 29,000 computers in K-6 classrooms. The study analyzes the link between the use of technology and higher student achievement.

Since implementation, student scores have risen steadily on both the state standardized testing instrument and the National Assessment of Educational Progress. Significant gains in reading, writing and math were achieved. The state's technology implementation program was found to be more cost-effective than other popular interventions, including class-size reduction.

· A recent study was released by Westat, a Maryland-based research firm, that was commissioned in 1998 by the Illinois State Board of Education and was primarily intended to assess the implementation and impact of learning technologies in K-12 classrooms. The broad research questions focused on seven primary areas; access; usage; competency; student learning; productivity; best practices; and factors that influence access, usage, competence, student learning and productivity. There was a relatively small but significant impact on student achievement recognized. Where teachers' use of technology to facilitate or enhance instruction was high, standardized test scores were also high.

REPORT ON

THE EFFECTIVENESS OF TECHNOLOGY IN SCHOOLS, ‘90-’97

This summary contains the numbered end notes from the original report. The page numbers refer to where the content appears in the report.

The report was commissioned by the Software Publishers Association; conducted by an independent educational technology consulting firm, Interactive Educational Systems Design Inc.

The report is based on 219 research reviews and reports on original research projects, from both published and unpublished sources. Of these 219 studies, 86 were published in professional journals, and 41 were doctoral dissertations.” p.1

Part 1 Effects of Technology on Student Achievement

The level of effectiveness of educational technology is influenced by:

specific student population

software design

educators' role

how students are grouped

level of student access to the technology p.2

Software is more effective (than some other instructional methods) because learner traits are more often being taken into consideration when software is being designed. p.2

Evidence suggests that interactive video is especially effective when the skills and concepts to be learned have a visual component and when the software incorporates a research based instructional design. p.2

Use of on-line telecommunications for collaboration across geographic locations has also shown to improve skills. p.2

Software Design Elements Which are Helpful

Students having some control over the amount, review & sequencing of instruction results in higher achievement and better student attitudes.

Low-achieving students and students with little prior content knowledge require more instructional guidance.

Learning Environment

District-level involvement and leadership of a school-level computer coordinator are key factors in developing a school environment conducive to effective use of technology p.3.

Educators are more effective after receiving extensive training in the integration of technology with the curriculum. p.3 Technology in the learning environment has been shown to make learning more student-centered, to encourage cooperative learning and to stimulate increased teacher/student interaction.

SECTION 1: EFFECTS OF TECHNOLOGY ON STUDENT ACHIEVEMENT

Note: Meta-analysis is a method of assessing the effects of technology-based instruction across many different studies, using a common measurement scale, called effect size (ES). An ES of 0.3 means that technology-based instruction is 30 percent more effective than the control group instruction. p.5.

An ES of 0.3 means approximately three (3) months gain. Thus, in three years, an instructional treatment with an ES of 0.3 per year would result in an additional gain of almost one year. p.5

Effectively used, technology-based instruction can increase student growth by 3 months per year, or an additional year of growth for every three years of technology assisted instruction. p. 5.

According to Kulik and Kulik,(8) in their meta-analysis of 254 controlled evaluation studies covering students from kindergarten through higher education, they found that computer-based instruction (CBI) had an average ES of 0.3. Where differences in achievement were statistically significant, the differences favored CBI in 94 percent of the cases. p.5.

In a follow-up meta-analysis of 97 studies, Kulik(9) found an average ES of 0.38 for CBI involving software classified as drill-and-practice and tutorial. p.5.

Ryan(11) found that the amount of technology related teacher training was significantly related to the achievement of students receiving CAI. p.6.

Nine studies that focused on word processing in the context of remedial writing instruction yielded an average ES of 0.49. p.6.

In two separate studies and five different measures of phonological awareness, the computer-based approach was found to be significantly more effective than regular instruction.(20) The average ES of 1.05 is considered significant.(19) p.7.

Stone(21) found that second grade students who had been receiving computer-assisted instruction (CAI) in reading and other areas since kindergarten, scored significantly higher in both reading comprehension and vocabulary than students in a nearby school with no CAI. (all other factors were equal). p.7.

Stine(22) found that Chapter 1 eligible students who used CD-ROM books demonstrated significantly greater gains in vocabulary and reading comprehension than students who did not. p.7.

Spelling

Two studies identified by Anderson-Inman suggest that keyboarding and keyboarding software can be used to improve spelling for low performing students. p.10.

Math

Two studies by researchers at the Stevens Institute of Technology(49), demonstrated the positive effects of high school mathematics software on retention. Ona delayed posttest the group using software demonstrated a retention rate 70 percent better than the group who did not use the software. p.11.

Funkhouser found that high school algebra and geometry students who used commercially-available problem solving software scored significantly higher on tests of mathematics content than a comparable group of students who did not use the software.(50) p.11.

In a study by Alexander,(53) college algebra students who used software significantly outperformed students receiving conventional instruction in their “understanding of the concept of functions and mathematical modeling.” The results in this study suggest the power of the computer to provide concrete visual support for the learning of abstract concepts. p.12.

Research by Svec suggests that college physics students can benefit from a microcomputer-based laboratory (MBL)(60).

Early Childhood Education

Two recent studies support the conclusion that well-designed computer-based activities, when presented with active participation of a teacher, can increase young children's cognitive abilities.

Godmacher and Lawrence(78) compared two groups of Head Start preschoolers. One group received a computer enrichment program and the other group engaged in standard Head Start activities. Students in the computer group demonstrated improvements in all academic skills tested, and their growth in memory and visual perception was significantly greater than that of the non-computer group. p.17

Haugland(80) suggests that the type of software young children are exposed to makes a difference in their cognitive development. Haugland and Shade designed a scale for identifying developmental software. Some of the characteristics of developmental software are as follows:

AGE APPROPRIATE...reflects realistic expectations of young children.

CHILD CONTROL...Children decide the flow and direction of an activity, not the computer.

EXPANDING COMPLEXITY...Once inside the software, the program continually expands to teach the child powerful ideas.

INDEPENDENCE...The software facilitates...independence by enabling children to master at least the initial components of the program quickly, with minimal instruction or prompting.

PROCESS ORIENTATION...The process of exploring the software engages the child and their motivation becomes intrinsic.

REAL-WORLD MODEL...The software is a simple, reliable model for children of the real world.

TRIAL AND ERROR... The software gives children unlimited opportunities for creative problem solving.

TRANSFORMATIONS...Software allows children to change objects and situations, which would be more difficult in daily life..children view hidden processes and learn the nature of cause and effect relationships.(82)

Four preschool classes were exposed to four different treatments throughout most of one school year: developmental software plus corresponding off-computer activities; developmental software only; non-developmental software only; and no exposure to software. The class that had access to non-developmental software demonstrated significant gains in concentration and short-term memory but significant losses in creativity. p.18

Software Design Characteristics and Student Achievement

Students benefit from software designed to address the particular instructional goal.

Instructional Control (Learner control versus program control)

Dalton(100) explored the effects of instructional pacing on a group of fifth and sixth graders viewing an interactive video on comets. Students who used the learner-paced version significantly outscored those using the program-paced version on a test of basic facts and definitions. p.22.

Several studies (p.22-23) indicate that students perform better on posttest when learner control is a part of the software program.

Three studies suggest caution before exposing low-achieving students or students with little prior content knowledge to software with a high degree of learner control. Either the software design or the teacher should provide sufficient structure for such students to succeed. p23.

Homogeneous small groups of low-ability students exposed to the learner-control version performed significantly worse than low-ability students exposed to the same software version, but placed in groups of mixed ability. Presumably, the low-ability students in the mixed-ability groups benefited from the guidance provided by their higher-ability partners.

Feedback

Low-ability students in grade 11 receiving KCR (knowledge-of-correct response) feedback during social studies reading comprehension practice, significantly out achieved students receiving AUC feedback. (AUC=Answer Until Correct)

Cognitive Strategies

Five recent studies suggest the learning effectiveness of software with embedded cognitive strategies. p.25.

The benefits of embedded cognitive strategies were more pronounced for low verbal learners than for high verbal learners. p.27

Scaffolding of Learner Support

Kao, Lehman, and Cennamo1(22) demonstrated the usefulness of scaffolding, a multitiered support structure for helping students build new skills, in a program. Students in the “scaffolded” version performed significantly better on a posttest than students in the “least support” and “full support” versions, suggesting that scaffolding can be an effective software design feature for improving student achievement. p.27.

Animated Graphics

Six studies found evidence for the benefits of animated graphics.

Calvert, Watson, Brinkley, and Penny(124) found that animated graphics “increased the poor readers” verbal recall to the level of their better reading peers.”

Researchers at Texas A&M University(126) working with physics students using animated graphics, still frame graphics and no graphics found there was no significant difference in achievement among the groups, however, students who had interacted with animated graphics required less time to answer posttest questions.

Rieber(128) used two groups of elementary students. One group saw still graphics the other animated graphics. With elementary students, using the animated graphics version resulted in significantly higher achievement with respect to both intentional and incidental learning. p.29.

ChanLin and Chan(129) compared six different versions of CAI on a DNA lesson. They found that students using the version that included both animated graphics and metaphors significantly outscored students using all other versions of the program.p29.

Student Grouping

Baron and Abrami(175 ) compared the effects of three grouping strategies when using language arts tutorial software. They found no significant differences in achievement among upper elementary students working individually, in pairs, or in groups of four. p.29.

Hooper(176) had different results when exploring the effects of computer-based mathematics instruction using various grouping strategies. One group of fifth-and sixth-graders received training in how to learn cooperatively and then worked in pairs and groups of four at the computer. Another group worked individually on computers. The students who worked cooperatively scored significantly higher on a post-test that included factual, application, generalization, and problem-solving questions. p.29.

Mevarech(177) had similar results to Hoopers. Mevarech compared pairs and individual, low-achieving third graders who used an integrated learning system as part of their mathematics instruction. The pairs, who were instructed on how to collaborate at the computer, progressed at the same rate as high achievers in their class. However, low achievers who worked alone progressed at a slower rate than the high achievers.p.39.

The learning advantage for cooperative groups trained in methods of cooperation was confirmed in subsequent research by Hooper, Temiyarkarn, and Williams,(178) who also found that cooperative groups were significantly more efficient learners. p.39

Section 2: Effects of Technology on Student Self-Concept and Attitude About Learning

Recent research confirms the potential of educational technology to improve students’ attitudes about themselves and about learning.

The results of several studies indicate that technology has beneficial effects on student self-concept.(193) In addition, a review of the literature finds positive effects on attitudes toward language arts, mathematics, science and social studies(194).

This section of the report features recent studies of educational technology that address student self-concept and student attitudes toward specific curriculum areas, and reviews research that relates changes in student attitudes to software design characteristics, specific technologies, and specific learner characteristics.

Three recent studies provide evidence of the positive impact of educational technology on student self-concept.

Rhyser(195) examined the effects of integrating computer-based instruction (CBI) in an urban elementary school. Students receiving CBI expressed stronger “feelings of success in school” than students in an equivalent school with out CBI. Such feelings are an important component of a positive self-concept. p43.

DeGraw(196) found that fourth graders grew in self-esteem and self-confidence when computers were placed in their homes and their school, as part of the Buddy System Project.

Previous research suggests that interaction with educational technology may lead to improved self-concept because:(1) Successful experiences with technology give students a feeling of control over their own learning.(198) (2) Such experiences may increase students sense of confidence in their abilities to perform in specific learning situations.(199)

In a study by Green,(201) inner-city third graders demonstrated significantly greater improvement in attitude toward writing after experiencing a writing process approach with word processing. p.43

Anderson-Inman(205) reported on two studies in which keyboarding was found to be a highly motivating method of practicing spelling for low-performing students. p.44.

Telecommunications

Riel(139) reviewed research on the use of networking for collaboration across classrooms indifferent geographic locations, she found evidence of improved academic skills. Six recent studies further illustrate the merit of using on-line telecommunications for educational purposes. p.32.

Students who had on-line access learned more in the Civil Rights Unit than students who did not go on-line. Their final projects were rated as stronger overall, and stronger in most of the specific competencies measured. p.33.

Students using the network received significantly higher course grades than equivalent students who did not use the network.p.33