Our own Bill Hart-Davidson recently traveled to Austin, Texas where he was invited to talk about Eli Review by the Digital Writing & Research Lab. Bill talked about Eli’s history, its pedagogy, and the specific problems it was designed to address.
The full video of Bill’s talk is available here, along with the full text, images from his slide deck, and some real-time tweets from DWRL during the talk (see their full story). Thanks to the folks at DWRL for the live stream!
My name is Bill Hart-Davidson and I want to thank you for having me here to talk a bit about the peer learning service we created at the Writing, Interaction, and Digital Experience research center at Michigan State. It’s called Eli Review.
I’d like to start by explaining where Eli came from, and then I’ll talk about what it is and what it does. We started working on Eli about nine years ago. I was working a lot with your colleague Clay Spinuzzi back then, investigating how people in various kinds of knowledge work organizations were coping with an increasingly distributed environment for writing. We were looking at how teams of people, spread out in time and space, managed to write successful grant proposals. And to do it with some consistency. My colleague and fellow WIDE co-director at the time Jeff Grabill was doing something similar. He was looking at how folks in community organizations came together to write in scientific genres. We were all interested in how folks were learning to write in challenging situations outside of school.
One reason we shared this interest is that writing is something most people never stop learning. Jason Swarts has written a great book that makes this case with precision, showing how people in various kinds of knowledge-work settings continue to learn to write on the job. What they do not do is take a lot of writing courses. Rather, they learn largely by practicing, and by giving and getting feedback from co-workers.
So the questions we started asking back in 2004 were: is peer learning this important in formal learning as well? A lot of us use it, but not always well or with a lot of care. How well prepared are folks to do that?
And, assuming we want to set students up well for their lives beyond our classrooms, how well prepared are they to help each other learn? To give good feedback and to use the feedback they get?
I want to ask you to picture in your head what peer learning in writing looks like. Can you conjure such an image?
It looks little like this. Ok, most of the time it’s not this graceful.
What you see are people performing and learning together. Their greatest resource is one another. This is Vygotsky’s powerful idea of peer scaffolding.
The ability to identify and learn, in a moment, from a more capable peer is a powerful thing. And it is no accident that we teach both dancing and writing in studios. We want those moments to happen in both spaces.
There are some things that I think are worth lingering over when we ponder this idea of peer scaffolding. First, it is about the interactions that happen among learners, first and foremost. This idea of a more capable peer isn’t some hierarchical relationship that is fixed. It is a dynamic, shifting set of relationships. It happens, moment to moment.
You know the moments I mean. When, just for second perhaps, one student identifies that “more capable peer” – the one on the right foot – and then, even in the next measure, maybe those roles change. Now instead of sneaking a peek, I might be the one on the right foot, helping one of my fellow learners.
The feedback that flows among peers makes everyone better. This is “the zone of proximal development.” The idea that when we work and learn together we are each capable of much more than one of us working alone.
But dancing and writing are different too, aren’t they?
Writing happens in the mind of the learner and on papers in screen in front of her. So it’s harder to see. And that means it’s much harder to do the kind of on-the-fly peer calibration that makes the Zone of Proximal Development work.
It’s hard for teachers, too, to see and intervene (or not) like a dance instructor can. Sweeping across the chorus line, I can see who is on the beat, whose toes are pointed, and whose top line is a model for the others. And I can do it quickly. Fast enough to point it out to the group and get everybody on track.
Feedback like that – formative feedback – is powerful, but only when it is timely. That’s what makes writing studios challenging.
In the picture I just showed, could you tell who is writing? Who is addressing the key criteria in their writing? Who is offering feedback to a peer? Who is offering feedback that is helpful? Aligned with the assignment criteria?
Can you see who is struggling to understand the criteria? Can you tell who the others in the group are looking to as a model for their learning?
Can you see you is stuck and can’t figure out what to do next?
Look again.
So much of the data available to the dance instructor – a function of the visible moves a dancer makes – is unavailable to me in the moment as a writing teacher in a standard writing classroom.
And if it is unavailable to me, the teacher, it’s also unavailable to my students.
The most powerful learning resource we have, the most transformative influence on learning…is largely missing here.
Until now.
Eli Review is a peer learning service that knows what you know about how students learn. That’s because it is writing pedagogy – and learning theory – made into software.
Eli was engineered from the ground up by writing teachers to help learners harness the power of feedback. Eli puts a powerful set of coordination tools and reports in students hands. The result: more time spent doing the things that matter most. Especially revision, review, and reflection.
And for teachers, Eli offers a “live feed” of student learning. This means that power of formative feedback is working for you too. In your classroom, in your program, in real time.
The goal of Eli, broadly speaking, is to transform classrooms into the kinds of feedback rich spaces that are the best environments for learning. What works?
Students improve most when they are engaged in deliberate practice – practice that they understand to be oriented towards some specific goal – and when this practice is both guided by an expert AND scaffolded by peers.
What kinds of practice is best?
Revision is best. Revision focused on higher-order concerns (e.g. strengthening evidence to accompany claims). To do that, developing writers need good feedback.
More practice is better. And…and this is important, intermittent or inconsistent practice is bad.
So…
Let’s take a look at the kind of transformation Eli makes possible. You might be aware of a few meta-analyses that have shown what kinds of activities produce the most significant gains for writers.
They show that of the big three – writing (or drafting), review, and revision – it’s those last two that really pay off the most.
Trouble is, for most teachers, this is what a typical paper looks like in terms of time spent.
The research suggests it should look more like this. And not only that, but that review and revision should be focused on higher-order concerns and should be driven by high-quality feedback.
More, and more frequent review and revision. More and better quality feedback.
And this write-review-revise cycle would ideally repeat. Maybe several times. So each paper would unfold in time something like this.
Let’s think about why, for just a second, that makes sense.
Writing is the solitary move here.
Review, planning, and revising (based on others’ feedback) permit the kind of peer scaffolding we see in a dance studio. That yellow and green area? You can think of that as “the zone” – the zone of proximal development – the place where peers have a rich set of resources to learn from each other.
It’s also the place where I, as a teacher, can get to the things that really make a difference in the way I use my time.
[LIVE DEMO] Let me show you a sequence now in Eli and how it helps me to see where I can best intervene to help students.
It’s not magic. We engineered Eli to help do what every writing teacher can, does, or would do if she had enough time: pay attention at every key moment in the writing process.
Because we know the most informative moments about student learning don’t happen at the end. They happen along the way. Eli knows what you know: where to watch, and what to gather so you can see your students learning and help them.
And between you and me: it’s exhilarating. The first time you can read a reviewer’s comments that show you, clearly and unequivocally, that they “get it” because they are giving great advice. Even better, the moment when you can say: psst…hey, take you own advice!
We see eight reviews with a class average score of 2 (out of 5) represented by a black line on the graph. Green and Orange lines show where the top 30% and the bottom 30% of students fall on the graph. One specific student, represented by a blue line, is well below the average and below the bottom 30% on review number 3, and a large arrow points to this with the words “Now is the time to intervene”. The chart demonstrates how teachers can see who needs help and act quickly before students struggles become worse.
With more review and revision cycles, more time spent doing the things that benefit students most, and faster results…another remarkable thing happens.
Eli lets me intervene in students learning in plenty of time for it to make a difference. I don’t have to guess who’s struggling. I can see it early. With enough time and in enough detail to make the comments, have the conference, or switch peer response groups in ways that will help.
Folks often ask us if this formative data means we don’t read their writing anymore. Heck no. Of course we read it. The difference is that we now read it in the context of their learning process. And it makes all the difference.
This is why a writing classroom with Eli is truly more like a yoga studio than a dance studio. We aren’t always trying to get everyone to do the same choreography in a writing class after all. There can be many right ways to do the steps, as many right ways as there are students.
But we do want to get each student to somewhere new in their individual practice. We want them all to learn something new, to expand their repertoire, to build strength and confidence.
Eli’s data displays make this kind of individualized instruction possible from day one of your course. If that sounds hard to believe, you should talk to teachers who have used it. It’s transformative. Not because it gives you more control, but rather because it lets you give that control back to students where it belongs.
This is a word cloud, made up of the most commonly used words in student reviews for a technical communication course I taught last year. A few words stand out as frequently used as they are larger than others: think, good, information, make, students, maybe, customer, help, like…
There are numbers in Eli. But more importantly, there are words. There is writing. There is reflection. There is evidence for you to understand how your students are thinking about their writing and how they are thinking about the criteria and learning goals you’ve asked them to engage.
Eli has served 200+ schools around the world since 2011.That includes courses created by more than 1000 unique instructors. And more than 60,000 students. 500,000 pieces of student writing and more than 1.2 million comments, nearly 40 million words worth!, of writing about writing. One of my favorite stats is that as of this spring, we had 12,800 revision plans in Eli that no teacher had assigned. These are plans students created on their own, no teacher needed. J
So…what’s next? More help for teachers and students in the form of feedback. Including some experiments with machine learning and machine-assisted feedback. Let me show you an example…
The Hedge-o-Matic is a simple, though not uncomplicated, application built into an article that was recently published in the journal Enculturation. We want to thank the editors here on the panel as well as our patient reviewers who helped us bring this “appticle” to life. You can go try it out right now, thanks to their help.
The Hedge-o-Matic takes a passage of text as input and parses it into sentences. It then classifies each sentence as a hedge or non-hedge, or perhaps as more or less hedge-y. We are looking for propositional hedges in the HoM – the sort of moves a rhetor makes when she wants to adjust a claim to suit the strength of the available evidence. The HOM then presents results as feedback to users in two views:
- The first is a visual: a hedge distribution chart showing where the hedgey & non-hedgey sentences are in the passage. Reading left to right on the X axis, the dots show each sentence in order and where it falls on the hedge/non-hedge continuum.
- The second display is a simple list of the classified sentences with the label – hedge or non-hedge – and the confidence values reported by the app.
