For the past two years, our team at Eli Review has been making a distinction between educational technologies and learning technologies. Most recently, Bill Hart-Davidson explained it this way:
- Educational technologies help students, teachers, and administrators accomplish goals associated with schooling, but not necessarily associated with learning and teaching. Examples:
- Student-information systems
- Learning Management Systems
- Automated Writing Assessment.
- Learning technologies help students, teachers, and administrators accomplish goals associated with learning, often focused on formative assessment to enable a particular kind of learning that would otherwise not be possible or very time intensive. Examples:
- assistive technologies
- clicker tools
- some adaptive technologies
We’ve explored this distinction several times:
- Bill asked the question in January 2015, “Can Technology Make us More Human?”
- His Ignite talk in April 2015 explained how Eli’s scaffold for peer learning makes real-time decision-making easier for teachers as they coach and writers as they revise.
- Jeff Grabill’s keynote to Zeeland ISD faculty in August 2016 contrasted robots that replace teachers with technologies that enhance teacher work.
- Bill explained how Eli’s algorithms “aid the human work of teaching writing.”
- Bill, Mike McLeod, and I extended the contrast between educational technologies and learning technologies in our #4C16 panel by discussing the implications for vetting technologies, considering UX, and planning professional development.
We describe Eli Review as a learning technology that makes it easier for students and teachers to give better feedback and improve writing. Fundamentally, Eli is a pedagogy. Our professional development modules explain how teachers can use the digital scaffold to coach reviewers and writers. We know that Eli’s efficiencies make it possible for feedback to be a routine and powerful part of every class.
Modus Operandi, Not a Sales Pitch
We run our company differently because we see ourselves as learning technology company invested in the professional development of our users. We’re a small team, but we average about 5 hours of personal, real-time support—excluding email, sandbox development, or other prep work—with new customers who form a relationship with us (not everyone does). While Eli Review is our product, and we take great pride in it, the app facilitates the types of pedagogies we value, and those pedagogies spread through relationships. Besides keeping the lights on and our social media engines on Twitter and Facebook humming, conversations about teaching and learning are our real work.
Our commitment to being a learning technology company also means we approach our system usage data at the end of the academic year with big questions:
- Are we changing business as usual for the writing students do in school?
- Are we helping instructors do more of what works to improve learning?
Below, we share several metrics around the quantity of revision plans and reviews assigned in Eli that help us know if we’re really moving the needle on spreading a pedagogy of rapid feedback and revision.
More Instructors Assign Revision Plans
Eli makes it possible for instructors to assign write-review-revise cycles so that students get feedback, use it to compose a revision plan, and resubmit their writing. Revision plans were introduced in early 2012, but this is the first year when more than 1,000 revision plan tasks have been assigned.
Out of the 974 course taught this year, 448 (46%) included at least 1 revision plan. Moreover, 86 courses (9%) included 5+ revision plan tasks. For more instructors, revision plans are becoming routine.
We are really proud to facilitate 10,000 students selecting, prioritizing, and reflecting on peer feedback before revising. Eli creates a space in the revision plan for metadiscourse about writing, and we know that’s where learning happens.
More Instructors Assign More Reviews
Eli exists because the common timeline for writing doesn’t create enough opportunity for feedback.
It’s core to our mission for more instructors to assign more reviews. It’s core because, as Jeff says in the opening video to the first professional development module, “one of the most powerful movers of the needle in any learning environment is feedback.”
We are actively encouraging instructors to increase the numbers of reviews they assign each term because that increases the likelihood that writers are getting more feedback, which leads to more learning. Our data shows shows K-12 and higher education teachers in our system have slightly increased the frequency of review.
Average Number of Reviews Assigned (in Courses) by Institutional Type per Academic Year
| Institutional Type | 2012-2013 | 2013-2014 | 2014-2015 | 2015-2016 | |
| K-12 | Average Review Tasks | 5 | 3.9 | 3.74 | 3.82 |
| Courses with at least 1 | 7 | 163 | 213 | 150 | |
| Higher Education | Average Review Tasks | 4.92 | 4.57 | 4.38 | 5.06 |
| Courses with at least 1 | 65 | 206 | 547 | 782 | |
Caption: 2012-213 was a banner year for quantity of review tasks because there were several teacher-researcher groups working on peer learning pedagogy. This year, averaging almost 4 reviews across 150 classes in K-12 and averaging 5 reviews across 782 courses in higher education represents a gain over last year.
We can see that the average number of reviews is a bit skewed when we look more closely at the frequency of reviews per instructor. That is, a few instructors are doing a lot more reviews. The table below shows review frequency by range across academic years. In 2015-2016, there’s slight bump in the percentage of instructors assigning 9-24 reviews. Last year, only 8.7% of instructors were in that range; this year, 10.6% are.
We’re focusing on the positive gains, but we see too that half of all courses are assigning 4 or fewer reviews. At this rate, instructors aren’t using peer learning often enough to expect improvement; at least, Eli isn’t capturing enough peer learning to expect improvement. This data tells us how hard it is for teachers to change the rhythm of feedback in their courses. We have work to do to help more teachers increase feedback opportunities.
Power Users Are More Diverse
But, we’ve got a growing and more diverse group of power users. This table shows that 15% of instructors teaching courses enrolling 5+ students have designed 26 or more reviews over the course of their use of Eli.
| Reviews Assigned All Courses Ever | Instructors | Percent |
| 1-25 | 476 | 84% |
| 26-50 | 44 | 8% |
| 51-75 | 17 | 3% |
| 76-100 | 7 | 1% |
| 101-125 | 5 | 1% |
| 126-150 | 7 | 1% |
| 151-175 | 2 | 0% |
| 176-200 | 1 | 0% |
| 201-225 | 1 | 0% |
| 251-275 | 1 | 0% |
| 276-300 | 1 | 0% |
| 326-350 | 1 | 0% |
| 401-425 | 1 | 0% |
| Grand Total | 564 | 100% |
Ranked from highest number of reviews to lowest, the top 25 power users include some predictable and a few unpredictable types of instructors:
- an administrator who started in 2015-2016 setting up 30 courses per term (of course, given scale)
- 13 Michigan State University writing instructors (of course, given that Eli has the longest history at MSU)
- 5 high school teacher-researchers
- a hybrid writing instructor who started in 2014-2015
- an online writing instructor at community college who started in 2015-2016
- Me (of course)! Twice actually, #9 using the account from my time at Bedford/St. Martin’s and #15 in my role as Director of Professional Development.
And for kicks, Bill is #26, Jeff #29, and Mike #59. In the industry, it’s called “eating your dogfood.” We actually do it because we’re teachers, and Eli helps our students learn to give better feedback and revision.
Is Eli changing teaching and learning?
First, let’s start by acknowledging that changing teaching and learning is an audacious goal. Since it’s bold, any success is impressive.
Second, let’s be clear about how we’ve defined changing teaching and learning. Our data show that assigning 5+ review or revision tasks separates users into meaningful groups. So, we’re assessing our impact on changing teaching and learning by tracking two metrics:
- Percentage of courses assigning 5+ revision plans
- Percentage of courses assigning 5+ reviews
Given that we’re aiming for a hard goal and setting a fairly low bar, we’re excited to report among all courses enrolling 5+ students:
- 3% increase in courses assigning 5+ revision plans
- 13% increase in courses assigning 5+ reviews
This analysis of the quantity of revision plans and review tasks suggests that 40% of Eli Review courses in 2015-2016 included 5 or more opportunities for students to get feedback. Does doing these things more represent a change? In a sense, yes. The patterns of interaction in these classes is changing. We think that is a very good thing, and our thinking here is backed up by quite a bit of research.
We know that more opportunities for feedback and revision are our best hope for changing business as usual for the writing students do in school and helping instructors do more of what works to improve learning:
- When reviewers can describe what they see as reader, evaluate it according to criteria, and suggest changes, they are learning to give better feedback.
- When writers can select, prioritize, and reflect on feedback, they are learning to be better writers.
- When instructors can see the decisions students are making as they give and use feedback, they are better able to coach learning.
Still, these two discrete data points don’t tell us everything we need to know:
- This data doesn’t tell us if coaching is happening. We can only see assigning. But, wow, we can see assigning of write-review-revise tasks across six academic years and 167 institutions (some repeat customers)!
- This data doesn’t tell us if students are getting better at giving feedback. We can only see how often they had opportunity to practice. But, wow, we can see that some classes included a lot more practice.
- This data doesn’t tell us if students’ developed better habits of mind or even better drafts because of this practice. But, wow, the data in Eli allows us to describe the opportunities students were given to practice and how engaged they were in that work with more precision.
From the bird’s eye view across all courses over the last six academic years, we moved the needle a little bit in some courses in 2015-2016. There’s more to do to make feedback and revision routine and powerful in every class. But, what you can see, you can teach.
We’re a learning technology company. Contact us if you want to change the frequency and effectiveness of peer learning in your classroom.
