Legitimate concerns about writing rubrics

There is no doubt that some criterion-referenced assessment procedures and related documents such as the Australian ESL Scales can be extremely useful; one example of the latter in use comes from The New England (NSW) Girls School. A “cut-down” form I use myself is here, and it has proven very accurate.

However, there are legitimate concerns about the use of writing rubrics. See for example The Trouble with Rubrics by Alfie Kohn (ENGLISH JOURNAL March 2006 — vol. 95, no. 4).

Once upon a time I vaguely thought of assessment in dichotomous terms: The old approach, which consisted mostly of letter grades, was crude and uninformative, while the new approach, which included things like portfolios and rubrics, was detailed and authentic. Only much later did I look more carefully at the individual floats rolling by in the alternative assessment parade — and stop cheering…

Consistent and uniform standards are admirable, and maybe even workable, when we’re talking about, say, the manufacture of DVD players. The process of trying to gauge children’s understanding of ideas is a very different matter, however. It necessarily entails the exercise of human judgment, which is an imprecise, subjective affair. Rubrics are, above all, a tool to promote standardization, to turn teachers into grading machines or at least allow them to pretend that what they’re doing is exact and objective. Frankly, I’m amazed by the number of educators whose opposition to standardized tests and standardized curricula mysteriously fails to extend to standardized in-class assessments.

The appeal of rubrics is supposed to be their high interrater reliability, finally delivered to language arts. A list of criteria for what should be awarded the highest possible score when evaluating an essay is supposed to reflect near-unanimity on the part of the people who designed the rubric and is supposed to assist all those who use it to figure out (that is, to discover rather than to decide) which essays meet those criteria…

I worry more about the success of rubrics than their failure. Just as it’s possible to raise standardized test scores as long as you’re willing to gut the curriculum and turn the school into a test-preparation factory, so it’s possible to get a bunch of people to agree on what rating to give an assignment as long as they’re willing to accept and apply someone else’s narrow criteria for what merits that rating. Once we check our judgment at the door, we can all learn to give a 4 to exactly the same things.

This attempt to deny the subjectivity of human judgment is objectionable in its own right. But it’s also harmful in a very practical sense. In an important article published in 1999, Linda Mabry, now at Washington State University, pointed out that rubrics “are designed to function as scoring guidelines, but they also serve as arbiters of quality and agents of control” over what is taught and valued. Because “agreement among scorers is more easily achieved with regard to such matters as spelling and organization,” these are the characteristics that will likely find favor in a rubricized classroom. Mabry cites research showing that “compliance with the rubric tended to yield higher scores but produced ‘vacuous’ writing.”…

Maja Wilson has written a book on the subject: Rethinking Rubrics in Writing Assessment . The fact it won a NCTE James Britton Research Award attracts me, and the chapter I downloaded looks very promising indeed. The gist of her argument may be seen here:

So if you really do want to think about what rubrics, for all their uses, might miss, and whether what they miss could after all be very important, I suggest you follow these ideas and keep thinking…

See Maja Wilson (PDF) Chapter One

RELATED

Right-wing education critique is historically inaccurate and perpetuates myths on my personal site quarrels about some relevant issues. It was written around the same time as this post. Teachers may be interested. As the title indicates, it is a bit of a rant…

 

Literacy controversies

When you have been teaching as long as I have, and all up that’s over forty years, you should become immune to recurrent panics, usually political, media-driven, or coming from “educationists” with an axe to grind. But you don’t…

The latest tea-leaf reading and teeth-gnashing has followed the publication of the PISA Report 2007 (PDF). I posted on that here. Recently too we’ve had the Australian Adult Literacy and Life Skills Survey 2.

However, there are, as always, many reasons to think about how we teachers could do better. Young People Seen Losing Love of Reading by Kathleen Kennedy Manzo appeared in the November 19 edition of the US online magazine Teacher Week.

American youths are reading less in their free time than a generation ago, a statistic that bodes poorly for their academic performance, job prospects, civic participation, and even social well-being, a report by the National Endowment for the Arts says.

Increasing use of electronic media is largely to blame for a decline in pleasure reading among young people, says the report, released today. But the failure of schools to instill a love of reading is also a contributing factor, according to endowment Chairman Dana Gioia.

“The study shows that reading is endangered at the moment in the United States, especially among younger Americans … and not merely the frequency of reading, but the ability to read as well,” Mr. Gioia said in a telephone conference call with reporters before the report’s release. The emphasis in many schools on bolstering reading skills and preparing students for tests, he added, is insufficient for nurturing an appreciation of reading.

“This functional approach to reading,” he said, “is not adequate to instill a lifelong love of the subject.”

In a subsequent online ideas exchange a number of US teachers made suggestions, many of them similar to ideas you might hear in staff rooms or at conferences here in Australia. (By the way, in all those international comparisons, whatever it is they measure, Australia still outperforms the USA in literacy.) For example:

Claudia, a secondary teacher:

I have watched this conversation with interest, because for the past several years I’ve taught nothing but an elective, “Reading for Pleasure,” at my high school. We started small, one section per semester. Now, we run 12 sections, and could fill others if we had the teachers. In my school, there is a strong culture of reading—our media center is active and kids talk about their books. For the first nine weeks of this semester, 131 students in my classes read a total of 269,157 pages. (This includes books for English classes, not necessarily for pleasure.)

My class is designed to share books with students, and to give them a place and time to read, uninterrupted by other demands. Then, everyone writes about what they’ve read. I read with my students every day (tough gig, I know), and I respond to every entry, as a fellow reader.

I’ve noticed the same ‘dip’ in interest among our students that others have noticed. Over and over in their literacy autobiographies students tell me how much they loved reading in elementary, and then “something” happened. They can’t even articulate what it was. I wonder how much can be attributed to new demands on kids’ time, new interests, or peer groups that may not value reading. I spend lots of time in class talking about books, fiction and nonfiction, young adult, classics, popular adult fiction. Once kids learn (relearn?) what they like, they DO read for pleasure, because they have the power to choose what they read.

I want to throw one more idea into the mix. Nowhere in all these articles do I see a serious acknowledgement of the reading kids do the most: online. I would argue kids are interacting with text much more often than the “experts” think when we factor in computer use.

What do you think of this whole question? How do you encourage reading? How widely do your students read?

If you are a student, what is your attitude to reading? Do you think you are “good enough”? What do you read? How often?

 

Email about the Educational Testing Service

This arrived this morning.

Hello,

Because you write about ESL and learning the English language, I thought you and your readers might be interested in the Educational Testing Service’s [link] new TOEFL Tips & News feed [link]. TOEFL Tips and News is a free way for students to keep updated on the latest TOEFL developments, share English learning tips, and receive helpful insights from ETS on how to pass their exam.

Here are some recent headlines from TOEFL’s Tips & News:
• What Should You Read to Build Your Reading Skills?
• How To Recognize a Pronoun Instantly
• By 2008, seats for TOEFL iBT will increase by 80% in China

You can subscribe to TOEFL Tips & News by visiting the ETS website. Please help us tell students about ETS’ newest educational tool by copying and pasting this code to your blog: Subscribe to TOEFL Tips and News from ETS.

We appreciate any help you can offer to spread the word!

On ETS see Wikipedia.

The Educational Testing Service (or ETS) is the world’s largest private educational testing and measurement organization, operating on an annual budget of approximately $1.1 billion on a proforma basis in 2007. ETS develops various standardized tests primarily in the United States for K-12 and higher education, but they also administer tests such as TOEFL and GRE internationally. Many of the assessments they develop are associated with entry to US tertiary (undergraduate) and quaternary education (graduate) institutions…

ETS has been criticized for being a “highly competitive business operation that is as much multinational monopoly as nonprofit institution”.

Due to its legal status as non-profit organization, ETS is exempt from paying federal income tax. Furthermore, it does not need to report financial information to the Securities and Exchange Commission…

It is worth looking at their free magazine Innovations.

See also Fairtest.org.

The National Center for Fair & Open Testing (FairTest) Works to end the misuses and flaws of standardized testing and to ensure that evaluation of students, teachers and schools is fair, open, valid and educationally beneficial.

Australian connection

According to the Summer 2007 issue of Innovations, ETS produced the Adult Literacy and Life Skills survey used in Australia. I have a post on that here.

 

Worried about outcomes based education?

Following a story in today’s Australian, I have posted a rant on the subject, Pop songs are weird science — The Australian , on Lines from a Floating Life. Having read a comment there, and its associated blog, I added to the post. I now place those theses here too.

1. Outcomes-based education has nothing at all to do with seeing the teacher as a “facilitator”; in fact the reverse is true, as OBE insists that teachers actually teach and holds them accountable for what students can actually do at the end of the process. The woolly idea of “facilitating” is 1970s feel-good stuff; OBE is hard-headed post-Thatcherite “quality assurance”.

2. OBE has nothing to do with the teacher as “entertainer”.

3. OBE is not a left-wing plot; initially it was more probably a right-wing plot. See Jim Belshaw, “Changes in Public Administration – the New Zealand Model”.

4. OBE does not predicate any particular style of teaching, traditional or otherwise.

5. OBE does not cause the curriculum to become less traditional. You can have an OBE course in Latin which will differ from Latin as taught in 1955 only in the way the program specifies what students must be able to do in Latin after a unit of work and then tests that they are in fact able to use those bits of Latin — in other words, will have achieved/not achieved outcomes. Any variation in the way Latin might now be taught in 2006 comes from changes in language teaching from a whole range of influences, many of them actually well considered.

6. OBE has nothing to do with the rejection of the pass/fail mentality. It is at least 30 years, long before OBE, since anyone passed or failed the NSW Higher School Certificate. What the candidate gets is a result and a rank which may be good enough to open certain doors or not.

7. In good well-written outcomes reporting parents get a plain English description of what their children have been assessed as actually achieving. This may or may not be accompanied by a letter and/or a number, the semantic content of which is, and always has been, a great mystery. At least a clear statement of outcomes has some meaning, even if it may make comparing your child with Susie next door a little more difficult.

8. The fact there is so much confusion about OBE (whether it is desirable or not is a separate matter) is the result of illiteracy on the subject perpetrated by the media and certain commentators who really should know better.

The main down-side of OBE, from a teacher’s point of view, is that it makes the teacher only too conscious of his or her success or failure as a teacher, with the proviso, of course, that is it absurd to expect all students from IQ too low to assess on the one hand (and I have taught such) to bloody genius on the other (and I have taught them too, if “teach” is the right word then) to achieve identical outcomes. Needless to say, no curriculum makes such an absurd assumption, but the general public often seem to.

A second down-side, which will not attract much sympathy, is that it makes programming and lesson planning more rigorous and onerous. No longer can we just write down the name of the text studied and vague wish lists.

A third possible down-side, and one which worried good English teachers a great deal, is that “outcomes” which can’t easily be defined or measured, such as “enjoyment of poetry”, might be squeezed out of the course in favour of what can be defined. I feared this, but it does not seem to have happened.

A fourth problem with OBE is that the number of real outcomes any given unit of work may have could well be legion; what appears on the program is a selection arrived at during the planning stage of that unit. On the other hand, when one considers the sum of units taught in a year, having to determine outcomes for each unit does at least offer some guarantee that the year’s work has targeted a range of things it is hoped students will master. In the past where the emphasis was on the content of the course rather than on what students might be able to do it was more likely that the program might become unbalanced.

Read Professor Roy Killen, “OUTCOMES-BASED EDUCATION: PRINCIPLES AND POSSIBILITIES” and then believe very little of what passes for “knowledge” on the subject through the media.

See also for richer models of teaching Quality Education in NSW which includes but takes us beyond OBE and further from the more dreamily romantic or hippie-like aspects of “progressive education”, and my post on scaffolding.

Yes, I get frustrated… 😉

Later

Further to this post, see, mostly on New Lines from a Floating Life:

HSC English Paper 2 Advanced 2006 03Nov06.

Julie Bishop’s third-hand knowledge of English teaching 08Oct06.

Three magazines and an amazing AIDS story… 11Jul06.

Right-wing non-news story #20563 10Jun06.

Dumbed down syllabuses 30Apr06.

My past catches up 25Apr06.

Reactionary myth-makers and education 22Apr06.

Literacy.

And just one item from my old blog on Blogspot (if you can see it):

History debate rages over loss of narrative – Top stories – Breaking News 24/7 – NEWS.com.au.

See too Jim Belshaw’s kind response. That blog of his is a goldmine of background information on public administration, very revealing on the broader context from which current administrative approaches to accountability in education have sprung.