This began as a paper for EDDE 802 - Research Methods in Education, and then became a chapter on peer assessment in online learning contexts
The venerable reputation of peer assessment. The ability of peer-to-peer interaction to enhance learning is well documented. Topping (2009) references George Jardine (1774-1826) who described the methods and advantages of peer assessment of writing at the University of Glasgow. In 1842, the French essayist Joubert is attributed with the quote, “To teach is to learn twice” (as cited in Vaughn, Garrison & Cleveland-Innes, 2014, p. 87). Over one hundred years later, McKeachie (1987) wrote, “The best answer to the question, ‘What is the most effective method of teaching?,’ is that it depends on the goal, the student, the content and the teacher. But the next best answer is, ‘Students teaching other students.’” McKeachie includes several studies regarding the effectiveness of peer instruction, such as Gruber and Weitman (1962) and Webb and Grib (1967), which show peer discussions without an instructor produced significant improvements in achievement tests and student interest in their discipline.
The Webb and Grib study involved 1,400 students in 42 courses, and the favorable results included gains in student responsibility, from memorization to comprehension and understanding. The authors note that more importantly, at the completion of their investigation of student led discussions, “a significant proportion of teachers and students had changed the conception of their roles and their ideas on how students learn,” but “since the objectives of the project did not focus on these changes, no provision was made for their measurement, if, indeed, measurement of them is possible” [emphasis added] (1967, p. 71). Still, what instructors discovered was a profound impression that students were more capable on their own, that students who had rarely spoken out in class not only participated, but expressed ideas of “good quality” (1967, p. 75). The other thing instructors discovered was their role as teachers had shifted. After listening to the discussions, instructors became aware of the student’s varied interpretations of lecture material, which caused them concern over the effectiveness of their communication, which led to a redefinition of role. The new role meant “becoming more attuned to student needs, being more aware of what their students were thinking, and recognizing the difficulties students were having with material” (1967, p. 75). This speaks to the primary benefit of peer assessment; it makes the learning process more visible in ways that traditional assessment strategies often cannot, but it also speaks to one of the pitfalls of peer assessment – assessor competence.
Students’ perception of their role also changed from a passive, receptive attitude to a more active, responsible one. As one of the students quoted in their study relates, “It finally puts the responsibility of educating one’s self on the student’s shoulders, where it should be, for it is the student who will learn and understand more when it is he who discovers for himself the truth or falsity of the course’s content (p. 76)” And, much like the instructors, students also became cognizant of the abilities of their fellow students. They gained greater respect for the ability and the divergent views of other students, and the more reticent students seemed to “blossom” in the more relaxed atmosphere of the peer group.” The peer led discussions strengthened social presence (e.g. “They help to develop a habit of listening as well as speaking. They help to teach respect for another person’s point of view.”), and cognitive presence, “students often realized while attempting to explain the material to others that they did not really comprehend it themselves” (p. 77). For some, this realization meant changing their study habits from memorizing to “seeking a more thorough understanding of the material” and applying that understanding to personal existence. In the end, Webb and Grib identified seven principle advantages of student led discussions:
1. Discussion places more emphasis on comprehension and understanding and less on memorization,
2. In the interaction, students came to see several other points of view,
3. The students own ideas were clarified in the process of discussing with others,
4. The discussions forced students to think and organize their ideas,
5. Students were more actively involved in their own learning,
6. The discussions forced more thorough preparation than regular class meetings, and
7. The discussions led to a greater interest in the subject matter.
Crowdsourcing feedback in online learning contexts. Beyond student-led discussions, more recent studies corroborate that peer marking results in statistically significant improvement in students’ subsequent work (Forbes & Spence, 1991: Hughes, 1995; Cohen et al., 2001; Rust, 2002 as cited in Rust, O’Donovan & Price, 2005). Peer assessment has also handled the digital shift and shows promise in online learning contexts; “one of the strategies that can improve the quality of education, particularly in web-based classes, is electronic peer review. When students assess their co-students’ work, the process becomes reflexive: they learn by teaching and by assessing” (Nagel & Kotze, 2010). This is the spirit of the Community of Inquiry’s conception of Teaching Presence, which “implies that everyone in the community is responsible for providing input on the design, facilitation, and direction of the teaching process” (Vaughn, et al., p. 87). Technological advances enable learning to transcend time and place and support peer assessment because it eliminates challenges students face in managing their day-to-day lives. “In a blended community of inquiry, one of the biggest challenges of peer assessment activities can be finding a convenient place and time for all students to meet outside the classroom,” a problem solved by leveraging collaborative tools providing 24-7 access in hyper-connected environments.
Feedback, conceptualized as information provided by an agent about aspects of one’s academic performance, is one of the single most important factors influencing learning (Hattie & Timperley, 2007). And yet, “it is often not possible to provide feedback that is both detailed and prompt, thus limiting its effectiveness in practice” (Sun, Harris, Walther & Baiocchi, 2015). Peer assessment has been suggested as a method to crowdsource feedback in large online learning contexts because crowdsourcing has “been applied to otherwise intractable problems with surprising success” and would provide “as many graders as students, enabling more timely and thorough feedback” in a number of settings, such as MOOCs, where it would otherwise be impossible (Sun, et al., 2015). As Sun, et al. (2015), plainly state:
peer assessment is a workable solution to the problem of feedback; it reduces the burden to the instructors with minimal sacrifice to quality. On top of this, it has been conjectured that students also learn in the process of providing feedback [emphasis added]. If true, then peer assessment may be more than just a useful tool to manage large classes; it can be a pedagogical tool that is both effective and inexpensive (p. 2).
MOOCs present a problem of scale and peer assessment could unify Peters’ vision of large-scale education with new interactive technologies which work best in smaller, more intimate environments. Guri-Rosenblit (2014) writes that the industrial mode of operation has not yet proven compatible with the new digital technologies.
Efficient online communication is, by its very nature, labour intensive. The industrial model is based on the notion of a small number of academics who are responsible for developing high-quality materials for large numbers of students. Obviously, small numbers of academic faculty are unable to interact with thousands or even with hundreds of students (p. 115).
The effective utilization of peer assessment may allow small numbers of faculty to design interactive learning environments with hundreds or thousands of students using learners themselves as the prime mode of interaction. The advantages of peer assessment include providing larger quantities of feedback than the instructor could provide, in a timely and authentic fashion that resembles professional practice where providing and receiving feedback from work colleagues is a common activity (van der Pol, van den Berg, Admiraal & Simons, 2008).
Not everyone believes this. Peer assessment has its detractors and its challenges. Downes (2013) describes some of the ugliest manifestations of peer assessment as “the blind leading the blind,” where students reinforce each other’s misconceptions, or “the charlatan”, where students who are not subject matter experts convince other learners of their expertise. Peer assessor competence, “the blind leading the blind,” presents two challenges. First, students may not possess subject matter expertise to fairly assess their peers. Secondly, learners may also possess a skill deficiency because students “typically have no experience” in peer assessment “which breeds inconsistent subjective evaluation” (Luaces, Alonso-Betanzos, Troncoso & Bahamonde, 2015). Instructing how to give and receive feedback is an important part of the teaching and learning process (Barber, King, & Buchanan, 2015), and the role of positive, affective feedback in peer assessment is inconclusive. Stiggins (1999) suggests that planning for positive feedback can help students “succeed early and often” and yet, other research shows students ignore positive feedback.
Designing peer assessment interactions for maximum impact.
Peer assessment is far from a novel concept, but to this point it has only been conjectured that peer assessment is a reflexive process. Peer assessment “is an arrangement for learners to consider and specify the level, value, and quality of a product or performance of other equal-status learners.” (Topping, 2009). In addition to the assessment of peer’s work, revising the learners' own work after engaging with peer feedback is regarded as the other important activity for learning reflection (Smith, Cooper, & Lancaster 2002), particularly in online environments where “a growing number of educators have tried to utilize Internet-based systems to facilitate the process of peer assessment” (Chen & Tsai, 2009).
Falchikov and Goldfinch (2000, p. 315 as cited in Falchikov, 2004) suggest peer feedback judgments are most effective when they are based on well understood criteria of academic products. The potential benefits of peer assessment are also maximized when there is a deliberate attempt to build assessor proficiency through instructional interventions, such as providing specific assessment criteria, and with examples for how to compose valuable peer feedback messages (Gielen & de Wever, 2015). Peer assessment is most effective when learners are prepared for assessment through the use of marking exercises (Rust, 2002). At a bare minimum, learners should be involved in a short intervention where they are exposed to the assessment criteria, model answers, and examples of meaningful feedback messages (Gibbs, 1992, p. 17 as cited in Rust, 2002).
As Falchikov (2004) has enumerated, there are several key variables known to affect the outcomes of peer assessment, including design, population characteristics, what is being assessed, the level of the course, how the assessment is carried out and the nature of the criteria used in the assessment process. All of these instructional variables suggest that learning is fundamentally situated, and peer assessment, as a pedagogical approach, is fundamentally constructivist in nature, where “meaning is understood to be the result of humans setting up relationships, reflecting on their actions, and modeling and constructing explanations” (Fosnot, 2005, p. 280). This important to keep in mind because not only will the variables enumerated above affect the outcomes of peer assessment, but so will the structure and type of feedback provided to the learner.
Cheng, Liang and Tsai (2015) divide peer feedback messages into three types, affective (comments providing support, praise, or criticism), cognitive (comments focusing on the correctness of the work or giving guidance for improvement), and metacognitive (comments about verification of knowledge, skills or strategies). In their investigation of writing performance in an undergraduate context, cognitive feedback messages were more helpful than affective or metacognitive feedback. Cognitive feedback provides explanation or elaboration of the problems identified or suggestions provided. Another way to view the quality of a feedback message can be determined by its content. The content of an effective feedback message should provide both verification and elaboration. Verification is described as “a dichotomous judgement to indicate that a response is right or wrong,” and elaboration is the “component of the feedback message which contains relevant information to help the learner in error correction” (Hattie & Gan, 2011, p. 253 as cited in Gielen & de Wever, 2015).
Assessees, the other half of the peer assessment transaction, need to be capable to question the assessor’s peer feedback and have the opportunity to make changes accordingly, where the assessee chooses to follow, or not follow, the assessor’s advice in order to augment the quality of the academic performance (Horvadas, et al., 2014). Strangely enough, previous research suggests both positive and negative feedback can stimulate negative outcomes; positive feedback may cause learners to “rest on their laurels,” and negative feedback may cause learners to give up rather than double their efforts (Sun, et al., 20145). This discussion highlights that, for peer assessment to achieve its impact, attention needs to be paid to the structure and support needed for an assessor to generate high quality peer feedback (Horvadas, Tsivitanidou & Zacharia, 2014), and for assessees to be able to able to engage, reflect and revise their work. Without accounting for the structural components of the peer assessment process, peer assessment will less likely produce a learning environment where students are teaching students, but more likely create the conditions where the blind are leading the blind.
The gray areas of peer assessment. Despite claims that peer feedback can be an effective and inexpensive pedagogical approach, there remains significant mystery about how, why, and when peer assessment works. Peer assessment is a complex phenomenon, and a literature review of peer assessment in online learning contexts highlights three intertwined research gaps. In peer assessment studies, there exists
1. an unclear understanding of what takes place during the peer assessment process that contributes to learning,
2. a general lack of quantitative studies analyzing the impact of feedback on academic performance, and
3. very little research as to how positive/negative feedback impacts learners during the learning transaction.
Successful peer feedback “is dependent on interrelated factors including feedback quality, competence of assessors, perceptions of the usefulness and importance of feedback” (Demiraslan Cevik, Haslaman & Celik, 2015). A mini-research agenda for peer assessment in communities of inquiry and MOOCs includes examining how “the types of feedback” (affective or cognitive) affects students’ performance (Nelson & Schunn, 2007), student motivation, and why some students cannot perceive the benefits of peer assessment (Cheng, et al., 2015). Another research direction includes how to build assessor competence in participatory learning environments. Much is unknown about how feedback affects motivational variables, which “deserve further exploration” (Van Zundert, Sluijsmans, Konings & van Merrienboer, 2012), and why students decide to use or ignore specific feedback comments. Demiraslan Cevik et al. (2015) suggest that “the relationships between the nature of group dynamics and the acceptance and use of feedback merit further exploration” in participatory learning environments where peer assessment is given from one learning group to another, rather than an on individual basis, exploring how group learning dynamics affects the acceptance or rejection of peer assessment.
The importance of understanding peer assessment in terms of learning analytics.
Gasevic, Rogers, Dawson & Gasevic, (2016) “posit that learning analytics must account for (instructional) conditions in order to make any meaningful interpretation of success prediction” [emphasis in the original]. In a 2016 study they conducted, feedback was a type of trace data collected in only one of the nine courses under investigation, and in that case, the feedback tool was used to ask students about their study habits online and the value of quizzes. It was primarily used for question and answer, not peer feedback, highlighting a lack of research on peer feedback as an instructional condition from a learning analytics research perspective, confirming their observation that there is a “need to consider instructional conditions in order to increase the validity of learning analytics findings.”
These potentially important differences in peer assessment design have not been fully explored from a learning analytics perspective. Gasevic, et al., (2016) also suggest that “learning analytics has only recently begun to draw on learning theory and there remains a significant absence of theory in the research literature that focuses on LMS variables.” Gasevic, et al., (2016) suggest there are distinctive elements in courses, such as peer feedback, that determine learning management system (LMS) use, and they ground learning analytics approaches in Winne and Hadwin’s constructivist learning theory, where learners construct knowledge using tools (e.g. cognitive, physical and digital) to operate on raw information (e.g. readings given by the course instructor or peer assessment artifacts) to construct products of their own learning (Winne, 1996; Winne, 2011; Winne & Hadwin, 1998 as cited in Gasevic et al, 2016). These products can be evaluated with respect to internal standards, such as time, or external standards (e.g. rubrics used for grading and/or structured peer feedback scripts). Gasevic, et al., (2016) point out that learners are active agents in their own learning:
As agents, learners make decisions about their learning in terms of choices of study tactics they will apply to evaluate their learning products against. Decisions made about learning are influenced by conditions, which can be internal (motivation) and external (learning task grading policy).
It will be essential to understand the instructional conditions in order to make any sense of the patterns that existing within the social learning analytics.
Social learning analytics
make use of data generated by learners’ online activity in order to identify behaviours and patterns within the learning environment to signify effective process. The intention is to make these visible to learners, to learning groups and to teachers, together with recommendations that spark and support learning. In order to do this, these analytics make use of data generated when learners are socially engaged. This engagement includes both direct interaction – particularly dialogue – and indirect interaction, when learners leave behind ratings, recommendations, or other activity traces that can influence the actions of others (Shum & Ferguson, 2012, p. 10).
Most modern learning management systems (LMS) come with a built-in peer assessment tool that automatically distributes anonymous student responses to peer graders, enabling easy crowdsourcing of an effective pedagogical approach, but the ability to fully make sense of this data will require structured instructional conditions, outlined above, in order to best understand learners’ online activity and learning behaviours.
Learning analytics has been defined as “measuring, collecting, analysing and communicating data about learners and their contexts with the purposes of understanding and optimizing learning in the context in which it takes place.” Learning analytics “should support students’ agency and development of positive identities rather than predict and determine them,” with the goal of providing a basis for effective decision making regarding pedagogical design (University of Bristol, 2013). The potential of learning analytics resides in its ability to “combine information from multiple and disparate sources, to foster more-effective learning conditions in real-time” (Booth, 2012). Learning analytics approaches typically rely on data emanating from a user's interactions with information and communication technologies (ICTs), such as LMS, student information systems and/or social media. For example, the trace data (also known as log data) recorded by the learning management system, such as Moodle or Blackboard, contains time-stamped events about use of specific resources, attempts, time spent in the production or interaction with peer assessment feedback, the number of discussion messages read and volume of online discussions posted. Data mining techniques, employing “large amounts of data to support the discovery of novel and potentially useful information” (Piatetsky-Shapiro, 1995 as cited in Shum & Ferguson, 2012), are commonly applied to identify patterns in these trace data (Baker & Yacef, 2009, as cited in Gasevic et al., 2016).
Identifying and understanding whatever patterns exist in the peer assessment or peer feedback trace data will only be enhanced by well-designed and well-structured peer assessment activities that account for the instructional conditions. It has been suggested that “machine ethics, including learning analytics, stand on the cusp of moral nihilism” (Willis, 2014) because the conduct of learning analytics is viewed legalistically rather than asking the question, “What does this mean for humanity?” As Willis (2014) suggests, “now is the time to act within frameworks of human autonomy and agency” to help redefine what is learned from past academic failures and “responsibly innovate knowing that competing values often pervade technological innovations” and push for learning analytics’ interventions that are in the best interests of learners. As the forces of massification move forward, the promise of peer assessment will only be realized if it is also firmly based in effective pedagogical practices, some of which are largely understood, while others are still unknown territory.
Detailed APA citations available upon request.
