Monday, 13 November 2017

Try this, it works! Written Error Correction



I've come across a few posts on written error correction recently. ELT research Bites took on the topic in a two part post (2) and earlier in the year Gianfranco Conti (PhD applied linguistics, MA TEFL, MA English lit, PGCE modern languages an P.E.) wrote one. Conti claims that marking students books should be the 'least of a language teacher's priorities' but is he right?


Conti's post starts with a reference to Hattie who has suggested that feedback is very effective. Conti notes that giving corrective feedback on writing has now been given top priority in many state schools. He then goes on to write that his post is a response to the numerous teachers who have written to him asking whether the time and effort they put into marking is justified. Conti states:
I intend to answer this question by drawing on thirty years of error-correction research, my personal experience as a learner of 14 languages and teacher of five and, more importantly, neuroscience and common sense.
Impressive stuff. 14 languages! 30 years of error correction research! AND neuroscience! However when we get to the research we run into a problem. 


What jumps out initially is the age of the references. Conti promises 'thirty years of error correction research' but sadly those 30 years seem to be 1974-2004. The most recent reference, Conti 2004, is to his own writing. In fact, the only post 2000 references are to his own writing. I would have liked to read the works in question to evaluate the claims made but as Conti doesn't provide a reference list or hyperlink to the texts referenced in the post, this wasn't possible. 

Now, references don't have best before dates, and to this day E still equals MC squared. That said, the age of Conti's references does present an issue in this case. For instance, Dana Ferris, possibly the world's leading expert on written corrective feedback (WCF) is only mentioned in relation to a 1999 paper. She has, since then, written extensively on the subject including three books (Response to student Writing 2003, treatment of error in second language 2002, 2011, and with BitchenerWritten Corrective Feedback 2012). None of these are mentioned in the section called "What L2 error-correction research says". 

What's more, the research findings show a distinct change in the period Conti leaves out. For instance, Ellis and Shintani note that whereas in 1996 it was possible for Truscott to argue that the effectiveness of WCF could not be supported, this position is no longer tenable (2013:271). And as if spookily preempting Conti,  Ferris, in a 'state of the art' paper from 2004 notes that 'since 1999, I have done a considerable amount of both primary and secondary research work on the issues surrounding error correction in L2 writing' (2004:50). 

A lot is missed if we miss out the last 15 years of research. In a recent meta-analysis looking at WCF, of the 21 studies that met the inclusion criteria, only four were published before 2004. Conti's post does not include any of the 17 remaining studies. This is important as the research design of 'early (pre-Truscott, 1996) studies' contained design and execution flaws (Bitchener and Ferris 2012:50) perhaps indicating why 'studies published after the year 2000 showed a significantly higher effect size...than that of the studies published before 2000' (Kang and Han 2015:99). 

So what does the research say about corrective feedback? 


Research tends to suggest that error correction is effective. Ellis and Shintani state that 'both oral and written CF can assist L2 acquisition.' (2014:268) It has a positive effect on student writing (Ferris 2011, Bitchener and Ferris 2012). Kang and Han conducted a meta analysis of current research and concluded that "written corrective feedback appears to have a moderate to large effect on the grammatical accuracy of L2 students(2015:7)Research by Van Beuningen et al (2012) also points to the efficacy of WCF noting that it can improve subsequent pieces of research. This contrasts Conti's claims  that 'both direct and indirect correction do not impact students’ accuracy more effectively than no correction at all' (though it is perhaps possible that the bold font cancels out the research evidence).

It isn't clear from his post, but Conti may be talking about lower level students. As Schmidt notes on the ELT research bytes webpage, the Kang and Han meta Analysis found that '[WCF's] efficacy is mediated by a host of variables, including learners’ proficiency, the setting, and the genre of the writing task' (2015). Notably, Kang and Han's analysis suggests WCF is less beneficial among lower level learners. 

And what type of feedback is best? 

Conti claims that direct correction is 'pretty much a waste of time'  and 'Indirect correction, on the other hand, is not likely to contribute much to acquisition as the learner will not be able to correct what s/he does not know' (section 2) But what does the research say about types of correction? 

Direct or indirect? 
direct*

Direct correction, that is telling the students exactly what is wrong, and what they ought to write, 'is more effective than indirect' and direct feedback alone 'resulted in gains in grammatical accuracy' (Ellis and Shintani 2014:271). According to Shintani and Ellis 'Bitchener and Knoch (2010), Hashemnezhad and Mohammadnejad (2012) and Frear (2012) all reported direct feedback to be more effective than indirect' (2015:111In older studies no difference was detected, or indirect CF appeared superior  (Ferris 2011:32) but 'recent studies report a clear advantage for direct forms of feedback.' (Bitchener and Ferris 2012:74). As an interesting side note, teaching guides tend to promote indirect feedback (Ellis and Shintani's 2014:279). 

In conclusion, we can say fairly confidently that feedback of some kind is, in most cases, better than no feedback. Research suggests that even a 'single treatment', particularly if focused on a grammar point with a clear rule, is effective. (Ellis and Shintani 2014:271). 


indirect coded 
Coded or uncoded? 

Coded feedback is using some kind of code like 'V' for verb or 'S/V' for subject verb problems. These are usually accompanied by some kind of meta-linguistic explanation. Uncoded feedback, on the other hand, would just be highlighting that an error had occurred but not providing an indication as to what it was. The theory behind correction codes is that students will have to work a bit harder to work out what their errors are. 


indirect uncoded 
Interestingly, there is no evidence that coded feedback is superior to uncoded (Ferris 2011:34). Both teachers and students, however,  believe that coded feedback is more effective. (Ferris and Bitchener 2012:93) and there is some research supporting the idea that meta-linguistics explanations make feedback more effective (Ferris 2011:100). 

Focused or unfocused?

Focused just means concentrating on one type of error, verb forms or articles for example, rather than picking up different types of errors. The research is not that clear here. According to Ferris most researchers now believe focused feedback is more effective than unfocused (Ferris 2011:51, 2010:182). Shintani and Ellis (2015:111) are more cautious, noting that research has shown focused feedback to be effective 'in the sense that it results in gains in grammatical accuracy in new pieces of writing' and adding that it is more effective than unfocused feedback 'in some cases'. 

So the jury is seemingly out on focused vs unfocused WFC. However, whereas a study that compared focused and unfocused feedback found no difference between the two (Ellis et al., 2008) both were superior to the 'no feedback' group. A finding which seems to contradict Conti's bold statement. 

Doesn't error correction demotivate students? 


Finally, a common complaint is that error correction demotivates or humiliates students. This is certainly possibleConti quotes research from 1998 noting that 'an excessive concern with error treatment may affect students’ motivation negatively'. Well yes, it may, but (ready the bold font) it also may not. Ellis and Shintani argue that the case for this is perhaps overstated, pointing to the fact that 'learners typically state that they want to be corrected' (2014:275) a point Ferris (2011:51)  and Conti himself (see point 1) concur with. In my context (academic English writing) a study by Weaver (2006, N=44) suggests, like much research on this subject, that when students are asked, they say they like and want feedback. In fact, 96% of business students surveyed by Weaver agreed that 'tutors don't provide enough feedback'. Unless they actively enjoy humiliation (a hypothesis I'm sure someone could investigate,) then it seems unlikely that students mind WCF.  

Conclusion 


Conti has written a great deal on this subject. His blog includes posts explaining how current essay feedback practices are questionable, '7 reasons why traditional error correction doesn't work', 'why asking students to self correct is a waste of time' and 'why teachers should not bother correcting errors in their students writing'. Clearly, there is a theme here (and no, it's not starting blog posts with the word 'why'). Conti doesn't think error correction is all that worthwhile. To be clear, he doesn't think it is worthless either, just that it shouldn't be given as much importance as it currently is. It would be really useful though, when making statements like "There is no conclusive empirical evidence that EC can be effective" (2.7), if he could explain why he chooses to only discuss evidence that is 15 or more years old. I don't know Conti's teaching context so can't comment on whether or not there is an overemphasis on WCF there. What I can say is that, on my reading of the evidence at least, 'there is a clear case for correcting learners written errors' (Ellis and Shintani 2014:276). 




*I realise 'I like dogs and I like cats' isn't a great sentence. 

23 comments:

  1. Nice...of course, you've just doubled my working load, but hey...

    ReplyDelete
  2. Very informative post. Thanks for it. I remember reading a few of Conti's posts and being swayed by his arguments. So good to get a reality check :)
    Keep up the good work!

    ReplyDelete
  3. Cheers for this Russ! I echo Marek's comments. I was recently feeling guilty for my rather ad hoc use of correction codes and lack of any particular focus to the errors I was correcting so this post has given me lots to think about. Funny how something like coded feedback feels so right, both from the teacher's and student's point of view, yet the research says no. If they're having to think about it, it must work! Coded feedback also comes highly recommended on CELTA or Trinity certs, doesn't it? I shall be interested to see how Conti responds.

    ReplyDelete
    Replies
    1. research often contradicts 'common sense', doesn't it? :)

      Delete
  4. Interesting post. You might consider that Conti's use of Hattie is problematic due to Hattie's work not being overly focused on language instruction but education generally. So teachers highlighting content issues might be a bit more pointless in geography than in an EFL or even EAP.

    PS. Shintani not Shitani.

    ReplyDelete
  5. I am fairly new to this debate, but based on Conti's most recent blog, https://gianfrancoconti.wordpress.com/, the perspective seems to be that error correction is less useful than other techniques: "better investing your time in planning and resourcing your teaching more effectively."(Conti, 2017)

    ReplyDelete
    Replies
    1. That often seems to be his message, yes.

      Delete
  6. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  7. Nice piece. However, evidence-based and STRONG-evidence based are two very different things.

    The evidence you mentioned is hardly compelling and substantive including Ellis and Shintani's. Whilst it is true that I did not mentione later studies, having reviewed them recently as a result of your tweet a week or two ago I would not change my position at all.

    The meta-analysis you mention compares studies that measured accuracy in their informants using instruments and reliability measures of questionable validity that (a) do not provide a strong causality between independent variable (treatment) and observed outcome as they do not convincingly exclude maturation as a variable; (b) inter-rater reliability measures were 'dodgy' to say the least; (c) they do not delve sufficiently in the qualitative aspect of the treatment (correction means all and nothing, just as grammar instruction means different things to different people; one of the problems found by researchers in error correction is how poorly and inconsistently it is administered). It is no use to say direct correction works - the success of any instruction is affected by so many factors! (e.g. learner intentionality being key).

    Finally, Ferris has made a living off making a case for correction; she is a one-trick pony. If you read the C.I. people's literature you will find tons of equally biased papers and studies dismissing Ferris' assertion as nonsense.

    Truth is: the evidence for correction in the literature you quote has not sufficient external validity (one may argue that even the internal validity isn't) as cannot exclude maturation as a factor and it is often from studies which cannot really be compared in terms of context, MEASURES OF PROFICIENCY ADOPTED and, most important: the interrater reliability procedures and measures are not discussed in sufficient detail or, in certain cases, are not discussed at all. I don't know how much research you have carried out in your life, but in this kind of studies, such instruments and measures are crucial in establishing proficiency gains and statistical significance.

    Not all evidence is valid or substantive just because it is evidence. Evidence-based (your blog's claim) means all and nothing in this sense. It is the quality of the evidence that matters.

    Your post is largely based on secondary sources. If you looked at the primary sources as I have done in the last two weeks you would notice so many flaws in the procedures and design that you would not second Ferris and Ellis and Shitani's claims as strongly as you seem to do.

    Finally, I have often said correction can indeed work. My point is that it is not worth the hassle; in my experience, more effective scaffolding, input-provision and recycling constitute a better use of teacher time yielding better results in contexts where contact time is very limited.

    ReplyDelete
  8. Hi,

    Thanks for reading, and thanks for taking the trouble to reply. I appreciate it.

    You make a number of points and I'll try to respond to them all here. I do just have a quick trivial question though. You open saying 'nice piece' but in light of all the seeming flaws you point out, which parts did you think were 'nice'?

    1. 'However, evidence-based and STRONG-evidence based are two very different things'

    I presume here that you mean the research I have relied on is just 'evidence based' but the research you have in mind is 'strong'. Is that right?

    That might be true, my mind is open. That said, how does excluding 15 years of research make your evidence any stronger? Do you not think the last 15 years of research have been 'strong[ly] evidence based'?

    2. Thanks for your comments on the Meta-Analysis, I will look into what you say.

    3. 'Ferris has made a living off making a case for correction; she is a one-trick pony.'

    Ferris is a one-trick pony? Do you mean that she only researches error correction (hardly seems like a crime?). She has written on other topics, like L2 composition, so I'm not sure why she is a 'one trick pony'. Seems a bit unfair to call a researcher focusing on one area a 'one trick pony'. Perhaps I'm misunderstanding you? If you can link to papers showing Ferris' errors I'd love to read them.

    4. 'I have often said correction can indeed work. My point is that it is not worth the hassle' I think I said that in the final paragraph, no? 'Conti doesn't think error correction is all that worthwhile. To be clear, he doesn't think it is worthless either, just that it shouldn't be given as much importance as it currently is'

    5. 'Your post is largely based on secondary sources.'
    Well yes there are quite a few secondary sources, but the authors are often the same people who carried out the primary research.For instance, Ferris and Ellis are often reporting their own research. You claim had I looked at the primary sources I would have noticed a lot of errors and problems. This is certainly possible! Are there response papers highlighting all of these design flaws? Could you link to them? At the moment you are saying 'I've read these papers and they are flawed' but how can I check that without knowing which papers you've read and what the flaws are?

    Your basic position seems to be that I have misread the research and missed out large parts of it. Ok, I'm here to learn. Why don't you write a blog post, I'd be happy to post it on this blog, detailing which 'primary' sources I have missed and how they change things. I'm sure everyone would benefit from reading something like that, especially from someone who has a PhD in the subject. :)

    ReplyDelete
    Replies
    1. Conti makes some bold claims above about validity without explaining the issues in internal or external validity. He just assumes he can claim they are invalid without pointing to any evidence. This is despite them being published by highly respected language researchers in a peer reviewed journal, which, under the review process, usually checks for things like that.

      Delete
  9. Hi Russ,

    Conti says “The evidence you mentioned is hardly compelling and substantive”.

    1. He gives no reasons for objecting to Ellis and Shintani’s (2014) summary of research findings. Since the book was published, not one reviewer has challenged Ellis and Shintani’s summary, which I think is balanced and fair.

    2. There is also the evidence from Shintani and Ellis (2015) that deals exclusively with written corrective feedback. Again, Conti says nothing.

    3. Conti’s ad hominem remarks about Ferris carry no weight whatsoever.

    4. Regarding Kang & Han’s (2015) meta-analysis, Conti complains that the studies included in the meta-analysis “do not provide a strong causality between independent variable (treatment) and observed outcome as they do not convincingly exclude maturation as a variable”. Baloney! Kang & Han explain their method, their data coding, the moderator variables, the reliability of coding, and how effect size calculations were made. Taken as a whole, the selected studies show what Russ and Anthony Schmidt in his piece in Research Bytes said they show. Conti’s demand for the studies to provide “ a strong causality between independent variable ..and observed outcome” is meaningless until he explains which studies and which independent variables and observed outcomes he’s referring to, as is the assertion that “they do not convincingly exclude maturation as a variable”. This is posturing of the very worst kind; a hopeless attempt to sound learned.

    Conti’s second complaint is that “inter-rater reliability measures were 'dodgy' to say the least”. He gives no explanation of why they were “dodgy”. In contrast, Kang & Han say this: “To ensure coding reliability, all of the primary studies were coded multiple times until saturation was reached. Then, an outside coder was brought in to code a random subset of five studies. The overall interrater agreement rate was 97%, and the differences were resolved through discussion” (Kang & Han, 2015, p.6).

    Finally, Conti complains about the limitations of empirical studies, which don’t “delve sufficiently in the qualitative aspect of the treatment” and fail to take into account that “the success of any instruction is affected by so many factors! (e.g. learner intentionality being key)”. More baloney. In one paragraph Conti boasts of his experience in quantitative research methodology, allowing him to lecture Russ on the importance of ensuring that instruments and procedures, and measures of things such as interrater reliability are reliable, if proficiency gains and statistical significance are to be correctly calculated, and in the next he’s protesting that quantitative research is fatally flawed.

    As if this desperate ploy were not enough, Conti ends with the clincher: not all evidence is valid or substantive just because it is evidence - it is the quality of the evidence that matters! Amen to that.

    Geoff Jordan

    ReplyDelete
  10. This is a really interesting topic, and thanks for doing all this research. Like you, I teach Academic Writing in a UK university. My (admittedly anecdotal) experience is that using a correction code using metalinguistic explanation works, but only if students are encouraged to engage with it by revising their work. It is satisfying to see this borne out in Shintani and Ellis (2015, p. 117: “In the ME feedback condition, however, LAA was strongly implicated when learners were required to revise as the learners needed to work out how to apply the explanation deductively to correcting their article errors”). (Incidence of my confirmation bias duly noted…

    Yes, it is double the work to check their revised work, but it pays dividends in terms of student performance at the end of the course, and I tell my students this. Those students that just stuff their corrected writing in their bags and never think about it again don’t do as well. It’s all very well the teacher taking the time to correct the error, but if the student doesn’t take any notice of it, it will have no effect.

    The topic of error correction has been debated in my place of work recently, but with regard to oral correction of pronunciation when the focus of the lesson is not pronunciation. It would be interesting to know what the evidence on this is. You have inspired me to head to the library!

    Alison Evans

    ReplyDelete
    Replies
    1. Thanks for reading. I'd be interested in hearing what you do as we are thinking of revising our EAP marking soon. Can I get in touch with you?

      We have got a post on oral error correction. It's here http://malingual.blogspot.co.uk/2016/05/try-this-it-works-error-correction-for.html

      Delete
    2. That's really helpful, thank you! Good to know that I should be correcting pronunciation much more than I perhaps do...

      Yes, you can contact me. I've just found your email address on the About page, so I'll drop you a line.

      Alison

      Delete
  11. hi all
    backseatlinguist (Jeff McQuillan) has an interesting negative evaluation of some studies which attempted to go all out and correct every mistake http://backseatlinguist.com/blog/all-correction-all-the-time-is-written-error-correction-worth-the-effort/
    ta
    mura

    ReplyDelete
  12. Didn't Bitchner and someone else thrash this all out in a series of articles going back a decade or more? Seemed like a kind of pointless, flogging to death of the issue once you had read through 3 or 4 of the exchanges and I think it got bitter and personal towards the end....Dare I quote Kissinger on academic debates being so bitter because they matter so little?

    ReplyDelete
    Replies
    1. Truscott and Ferris I think. But research has been done since then. Truscott seems to have been wrong.

      Delete