Menu

Eli Review User Guide

Review Tasks

Review is the crucial step in Eli Review, where writers get the feedback they need on their writing tasks that will help drive their work in revision tasks. As an instructor, it’s important to think of yourself as the review coordinator – you establish what’s to be reviewed, you divide students into groups, and you define the criteria by which students respond to one another.

Once a review begins, you can use the real-time data generated by reviewers to model effective feedback and to guide further interventions. After reviews are complete, you can guide students in creating revision plans, which will outline their revision strategies and let you provide just-in-time coaching for their revision process.


1. Creating a Review

Designing effective reviews is the most complex work in Eli. Eli’s review features allow instructors to design reviews as simple or as complex as their learning goals require.

Step 1: Define Review Details

17a-review-task-step1

The first thing you’ll do is define some crucial contextual details about the review. Specifically:

This step only defines the structural characteristics of the review; later steps will allow you to specify what students review and how to structure their feedback.

Step 2: Review Materials

 

17b-instructors-review-task-create-step2

Once you’ve defined the descriptive details of your review, you’ll define what materials will be reviewed. These are the texts students will respond to during a review. From here you can select one or more of the writing tasks that have been assigned to the students in this course to be reviewed. Note that you won’t be able to create a review until you’ve created at least one writing task that can be reviewed.

Step 3: Reviewer Groups

The last step in creating a review is to give some structure to how you want students to respond to one another. Divide students into groups. Keep in mind that in Eli, writers and reviewers are 1:1; a student assigned to a group with 3 other reviewers will review the 3 texts of each of those reviewers. There are 3 ways to create groups:

 

step3-reviewer-groups

 

Important things to keep in mind about groups:

Late writers are a particular challenge for coordinating feedback; see our tutorial for strategies on managing late writers.

Step 4: Add Response Types

The last step in creating a review is to give some structure to how you want students to respond to one another.

17b-review-task-step4

There are five different response types you can add to any review:

trait-set-1-create

Important things to keep in mind about response types:

For more about the affordances of each response type, and some tips on designing your prompts, see our response type tutorial.

Saving as Draft / Assigning / Editing

When you’re done working on your review, you can proceed in two different ways:

17b-review-task-buttons

It is possible to edit or delete a review once it’s been assigned to students, provided that students have not already begun responding to it. Once students have begun submitting feedback to one another, Eli will not allow you to change the review because:

  1. any feedback students already submitted would be lost, and
  2. students may have already seen and utilized feedback submitted by reviewers

Be sure that the review is as designed as you’d like it before students begin responding; after that, the system won’t allow edits, with the exception of:

Loading Existing Review Tasks

The repository display will allow you to browse all of the review tasks you’ve created in the past, even as part of another course. When you’ve found the  task you want to clone, just click the “Load” link and the task form will be populated with the settings for that task.

You’ll be asked to enter a new due date, review groups (if not reused in the same course), and review materials. You’ll also be able to edit any of the settings from previous task. Once you’re done editing, you can save the task as a draft or assign it to students, as normal. For more information on loading tasks, see the Task Repository section of the user guide.

What Students See When Reviewing

To get a sense of how reviews are displayed for students, see the Responding to Writing section of the Student User Guide.


2. Review Report – Class-Level Data

One of the crucial areas in which Eli stands apart from other review technologies is the data Eli produces during the review process. Once a review is available to students, Eli will prepare a series reports that both communicate reviewer progress toward completion and surface useful data about writer and reviewer performance. This section outlines the review report summarizing performance for the entire class and the next section describes the report for individual students.

Each section of these reports is designed to provide instructors with:

Task Overview Display

18d-instructors-review-task-report-details

The first display in the review report is an overview display that will help you coordinate the review, including tools that allow you to do the following:

You can navigate between the other sections of the report using the navigation tabs.

Responses to Writing

The “Responses to Writing” tab provides an overview of how the entire class is responding to the writing being reviewed.

18e-instructors-review-task-report-writing

This is an aggregate display, tracking not individuals but class-level performance. From here, you’ll be able to:

This report will continue to evolve as students complete their reviews. Projecting it during a review and using it as a status board, including calling reviewers or writers out by name, can have very positive results and can help improve feedback.

More about the “Responses to Writing” report:

Responses to Reviews

Where the “Responses to Writing” report is a display of reviewer responses to writing, the “Responses to Reviews” report is a display of how writers have responded to the reviews they’ve received. This report is meant to help identify the most helpful reviewers as well as the most helpful individual comments given by reviewers.

18f-instructors-review-task-report-review

As with the “Responses to Writing” report, this report will evolve as students complete their reviews. It is also made more useful if students rate their feedback, which can be encouraged by sharing this data with them – the more feedback is rated, the better the report.

More about the “Responses to Reviews” report:

Engagement Data

The Engagement Data section of the review report provides data about reviewer activity that can be helpful when coaching them in how to give better feedback. Instructors may choose to use this data in their evaluations, but this quantitative data is meant to give insight into reviewer behavior that might help improve the helpfulness of their feedback.

18g-instructors-review-task-report-engagement

Included in this report:

* helpfulness calculations ignore comments that haven’t been rated – Eli does not assume the lack of a rating by the recipient implies anything about their perceived value of a comment

This report is truncated to fit into a standard browser window and is focused specifically on comments. The Engagement Data Download provides significantly more data about reviewer engagement.


3. Review Report – Student-Level Data

Where the class-level reports provide data about student performance in the aggregate, the individual student report provides a breakdown of a single student and their work in a review. Clicking on a student’s name (or their anonymized label) anywhere in the review report will open that student’s individual report, displaying two different views of that student’s work:

18i-instructors-review-task-report-student-feedback-received

The data in these reports resembles the data found in the class-level reports, but here it is focused specifically on student performance.  Both reports – whether listing feedback the student received or feedback the student gave, will be displayed in a similar format. In both cases, you’ll see:

18i-instructors-review-task-report-student-feedback-endorse

Endorsing Feedback

Endorsing an individual comment is a way for you, as the instructor, to send two messages at one:

 

endorsement-blog

Students will only see when comments have been endorsed, not when they haven’t been endorsed, so use of this feature is at the instructor’s discretion. See our endorsement tutorial for detailed examples of using endorsements strategically.

What Students See in Their Reports

To get a sense of what students see after a review, see the Review Report: Feedback from Reviewers section of the Student User Guide.


4. Downloading Review Data

While most data generated by reviewers is accessible through this review report, teachers and teacher researchers often prefer to work with raw data when conducting research. In some cases, the typical browser window is too small to allow a satisfying / usable display of all the data Eli makes available.

To that end, there are two download options for review data: the comment digest and the engagement data report.

Report Formatting – .csv files

Both data reports will download as CSV (comma-separated values) files. This type of file can be imported into almost all spreadsheet and database applications, making it possible to sort and query the data in any number of ways.

You can learn more about CSV files here:

Report #1 – Comment Digest

The comment digest is a report compiled as reviewers complete their reviews. It is a collection of all of the comments exchanged between reviewers and the quantitative data about those comments*. Instructors can download digests of individual reviewers or of all the reviewers in a course.

A comment digest includes one record for every comment exchanged between reviewers, and each record contains the following data:

* The Comment Digest is only available for reviews in which contextual comments were enabled.

Sample: you can download a sample comment digest to preview this feature.

Report #2 – Engagement Data

This report is an extension of the data available in the Engagement Data tab of a review report. In includes one row per  review participant, and each row contains the following data:

Things to note about how Eli assembles the Engagement Data report:

Sample: you can download sample engagement data to preview this feature.

upward arrow button