The Eli Review Blog

The first peer review in a new term of the new year is a fresh start with a new community of learners. It’s exciting. And no doubt mostly unhelpful.

Don’t worry. It’s not you. It’s not what you said or didn’t say. It’s not what you had students read or write to prepare. It’s not the way you phrased your instructions for the peer review. It’s not how much time they had. It’s not related to the interface or the device.

The first review is mostly unhelpful because students rehearse what they’ve always said in peer review. They aren’t yet practicing giving helpful feedback.

That first review shows you what students already know how to say about drafts. It shows you what criteria they care about, how they connect criteria to parts of the draft, what suggestions occur to them, and how personable they are when leaving comments. Or, more accurately, it shows you that they don’t do things by habit already.

If peer learning is going to get better in your class, you’ve got to create those habits. Students have to shift from rehearsing to practicing helpful feedback.

Rehearsal vs. Practice: A Music Analogy

One member of the Eli team, Bill Hart-Davidson, has been thinking quite a bit lately about the difference between rehearsal and practice. For the last year, he’s been learning to play the bass guitar. Starting from scratch, he decided to keep a log of his practice so he could reflect on his learning goals and progress.

For Bill H-D, both practice and rehearsal are important to learning. But they do different things:

Practice involves routines that challenge your ability. In my own experience they take patience and persistence to do, because they necessarily involve a lot of mistakes. They are not comfortable repetitions. They are best done, as a result, with a deliberate and limited focus and in a compressed bit of time (followed by a rest or a change in activity).

Rehearsal is working through things you’ve already learned to make them smoother, faster, cleaner, tighter, etc. It reinforces what you know. Rehearsal is not nearly as much like work as practice is. It’s often quite fun, in fact. It’s valuable to distinguish between them and to pay attention to the time I allocate to each, differently. It is what keeps me progressing in my learning.

When you create peer review assignments for students, it can be useful to think about where you are asking them to rehearse and solidify what they already have some grasp of and where you are asking them to stretch, to find their limit and push beyond their current abilities.

Rehearsal to Practice: A Teaching Scenario

This scenario for practicing helpful feedback assumes that rehearsal will happen first. It’s unavoidable. But, the rehearsal stages a productive teaching moment. Here’s how:

1. In class, give your best spiel for the value of peer learning. Explain why it matters to the course, to you, and to students.
2. Assign “Feedback and Improvement,” one of Eli’s student modules, or a similar reading that explains helpful feedback.
3. Assign a short response paper, like the one included in “Framing Feedback and Revision,” one of the sample curricula instructors can load into their own Eli Review courses.
4. Assign a peer review with a specific instructions for students to follow a pattern when commenting such as “describe-evaluate-suggest.” The sample curriculum “Framing Feedback and Revision” includes an example.
5. Once reviewers have given feedback, ask writers to rate it according to this rubric:
   1. One star if the comment DESCRIBES their draft or the assignment in a way that helps the writer better understand their own goals.
   2. One more star if the comment EVALUATES how well the draft meets the criteria.
   3. One more star if the comment SUGGESTS a strategy or question that helps them revise.
   4. One more star if the comment is particularly kind or thoughtful.
   5. All five stars only if the comment transforms the writer’s thinking.
6. Analyze the comments by downloading the comment digest  or using the engagement data. Report your findings to students when you debrief the review.
   1. Show comments that follow “describe-evaluate-suggest,” and discuss the helpfulness ratings of those comments.
   2. Show how to transform weak comments into stronger ones.
   3. Talk with students about what it’s like to get helpful and unhelpful feedback, and help them feel comfortable being “stingy with the stars” as they rate comments.
7. Set goals for comments in the next peer review. A typical first goal is to challenge reviewers to “describe-evaluate-suggest”; those three moves usually require 2-3 sentences per comment. Another way to identify a class goal is through a peer norm for total word count given by reviewers, which is available in the review tasks’ engagement tab under the intensity data story.

Repeat these steps again. Usually, about 60% of the class gives helpful feedback in the second review, if it happens soon after the first one and with the direct instruction on comments described in steps 6-7 above. When debriefing with students about the second review, be sure to praise improvement in the quality of comments given by sharing models.

By the 3rd review, 80%+ of students should be consistently helpful. Celebrate students who are no longer writing comments shorter than 20 words and whose overall word count is consistent with the peer norms.

Rehearsal to Practice: An Evidence-Based Approach

You can watch students get better at giving feedback by checking the number of comments long enough to be helpful.

Now, paying attention to word count is just a very basic indicator. Think of it like practicing with a metronome. It gives you, as the conductor, a way of setting the tempo at a pace where everyone is a bit challenged to keep the beat. Over time, the music gets cleaner and closer to the desired tempo.

In our research, we’ve found that a comment shorter than 20 words is unlikely to have all three helpful feedback moves: “describe, evaluate, suggest.” Short comments tend to be praise or a sentence-level correction. Lots of short comments are unlikely generate enough helpful feedback to inspire global revision.

With these word count parameters in mind, the goal of the first three weeks is to increase the number of comments overall while reducing the number of short comments made by the class. The following chart shows how comment length and overall count varies across thirteen reviews in a first-year community college class. Notice the steep decline from 30% short comments in week 1 to 15% short comments in week 3; the percentage goes up again in week 4 because that was the penultimate review before grading (same for weeks 7, 11, and 13) where editing/praise comments are warranted.

Figure 1 demonstrates that, as students shift from rehearsing what they’ve always said in peer learning to practicing the three helpful feedback moves, they say more. Engaged reviewers offer more comments and more comments that are long enough to be helpful.

Instructors can gauge reviewers’ word count in Eli Review’s Engagement Data tab. The intensity data story provides a per student sum of word count and a count of comments under 20 words. By using the headers to sort the columns, instructors can identify students with the highest concentration of too short comments.

Figure 2: This table shows that reviewers writing the fewest words also wrote the most short comments.

To improve, low intensity reviewers may need more skills, motivation, confidence, and/or time. If reviewers aren’t making behavioral changes by week 3 following direct instruction, one-on-one conversations about the expectations and procedures for reviewer engagement may be needed. It can be rewarding to sit beside a student and help them realize that they can notice something in a draft and can craft a helpful, kind comment in two or three sentences.  A helpful comment is the quickest win in becoming a better writer.

Once students learn to practice helpful feedback, peer review improves. Reviewers gain from giving feedback. Writers get what they need to revise from comments. Soon, even rehearsal is a success.