Legitimate concerns about writing rubrics

There is no doubt that some criterion-referenced assessment procedures and related documents such as the Australian ESL Scales can be extremely useful; one example of the latter in use comes from The New England (NSW) Girls School. A “cut-down” form I use myself is here, and it has proven very accurate.

However, there are legitimate concerns about the use of writing rubrics. See for example The Trouble with Rubrics by Alfie Kohn (ENGLISH JOURNAL March 2006 — vol. 95, no. 4).

Once upon a time I vaguely thought of assessment in dichotomous terms: The old approach, which consisted mostly of letter grades, was crude and uninformative, while the new approach, which included things like portfolios and rubrics, was detailed and authentic. Only much later did I look more carefully at the individual floats rolling by in the alternative assessment parade — and stop cheering…

Consistent and uniform standards are admirable, and maybe even workable, when we’re talking about, say, the manufacture of DVD players. The process of trying to gauge children’s understanding of ideas is a very different matter, however. It necessarily entails the exercise of human judgment, which is an imprecise, subjective affair. Rubrics are, above all, a tool to promote standardization, to turn teachers into grading machines or at least allow them to pretend that what they’re doing is exact and objective. Frankly, I’m amazed by the number of educators whose opposition to standardized tests and standardized curricula mysteriously fails to extend to standardized in-class assessments.

The appeal of rubrics is supposed to be their high interrater reliability, finally delivered to language arts. A list of criteria for what should be awarded the highest possible score when evaluating an essay is supposed to reflect near-unanimity on the part of the people who designed the rubric and is supposed to assist all those who use it to figure out (that is, to discover rather than to decide) which essays meet those criteria…

I worry more about the success of rubrics than their failure. Just as it’s possible to raise standardized test scores as long as you’re willing to gut the curriculum and turn the school into a test-preparation factory, so it’s possible to get a bunch of people to agree on what rating to give an assignment as long as they’re willing to accept and apply someone else’s narrow criteria for what merits that rating. Once we check our judgment at the door, we can all learn to give a 4 to exactly the same things.

This attempt to deny the subjectivity of human judgment is objectionable in its own right. But it’s also harmful in a very practical sense. In an important article published in 1999, Linda Mabry, now at Washington State University, pointed out that rubrics “are designed to function as scoring guidelines, but they also serve as arbiters of quality and agents of control” over what is taught and valued. Because “agreement among scorers is more easily achieved with regard to such matters as spelling and organization,” these are the characteristics that will likely find favor in a rubricized classroom. Mabry cites research showing that “compliance with the rubric tended to yield higher scores but produced ‘vacuous’ writing.”…

Maja Wilson has written a book on the subject: Rethinking Rubrics in Writing Assessment . The fact it won a NCTE James Britton Research Award attracts me, and the chapter I downloaded looks very promising indeed. The gist of her argument may be seen here:

So if you really do want to think about what rubrics, for all their uses, might miss, and whether what they miss could after all be very important, I suggest you follow these ideas and keep thinking…

See Maja Wilson (PDF) Chapter One

RELATED

Right-wing education critique is historically inaccurate and perpetuates myths on my personal site quarrels about some relevant issues. It was written around the same time as this post. Teachers may be interested. As the title indicates, it is a bit of a rant…

 

Teaching in tandem

These videos document the co-planning and teaching process of an ESOL teacher and a high school history teacher.

Thanks to Jonathan Chambers and Shanghai American School for these.

 

Worried about outcomes based education?

Following a story in today’s Australian, I have posted a rant on the subject, Pop songs are weird science — The Australian , on Lines from a Floating Life. Having read a comment there, and its associated blog, I added to the post. I now place those theses here too.

1. Outcomes-based education has nothing at all to do with seeing the teacher as a “facilitator”; in fact the reverse is true, as OBE insists that teachers actually teach and holds them accountable for what students can actually do at the end of the process. The woolly idea of “facilitating” is 1970s feel-good stuff; OBE is hard-headed post-Thatcherite “quality assurance”.

2. OBE has nothing to do with the teacher as “entertainer”.

3. OBE is not a left-wing plot; initially it was more probably a right-wing plot. See Jim Belshaw, “Changes in Public Administration – the New Zealand Model”.

4. OBE does not predicate any particular style of teaching, traditional or otherwise.

5. OBE does not cause the curriculum to become less traditional. You can have an OBE course in Latin which will differ from Latin as taught in 1955 only in the way the program specifies what students must be able to do in Latin after a unit of work and then tests that they are in fact able to use those bits of Latin — in other words, will have achieved/not achieved outcomes. Any variation in the way Latin might now be taught in 2006 comes from changes in language teaching from a whole range of influences, many of them actually well considered.

6. OBE has nothing to do with the rejection of the pass/fail mentality. It is at least 30 years, long before OBE, since anyone passed or failed the NSW Higher School Certificate. What the candidate gets is a result and a rank which may be good enough to open certain doors or not.

7. In good well-written outcomes reporting parents get a plain English description of what their children have been assessed as actually achieving. This may or may not be accompanied by a letter and/or a number, the semantic content of which is, and always has been, a great mystery. At least a clear statement of outcomes has some meaning, even if it may make comparing your child with Susie next door a little more difficult.

8. The fact there is so much confusion about OBE (whether it is desirable or not is a separate matter) is the result of illiteracy on the subject perpetrated by the media and certain commentators who really should know better.

The main down-side of OBE, from a teacher’s point of view, is that it makes the teacher only too conscious of his or her success or failure as a teacher, with the proviso, of course, that is it absurd to expect all students from IQ too low to assess on the one hand (and I have taught such) to bloody genius on the other (and I have taught them too, if “teach” is the right word then) to achieve identical outcomes. Needless to say, no curriculum makes such an absurd assumption, but the general public often seem to.

A second down-side, which will not attract much sympathy, is that it makes programming and lesson planning more rigorous and onerous. No longer can we just write down the name of the text studied and vague wish lists.

A third possible down-side, and one which worried good English teachers a great deal, is that “outcomes” which can’t easily be defined or measured, such as “enjoyment of poetry”, might be squeezed out of the course in favour of what can be defined. I feared this, but it does not seem to have happened.

A fourth problem with OBE is that the number of real outcomes any given unit of work may have could well be legion; what appears on the program is a selection arrived at during the planning stage of that unit. On the other hand, when one considers the sum of units taught in a year, having to determine outcomes for each unit does at least offer some guarantee that the year’s work has targeted a range of things it is hoped students will master. In the past where the emphasis was on the content of the course rather than on what students might be able to do it was more likely that the program might become unbalanced.

Read Professor Roy Killen, “OUTCOMES-BASED EDUCATION: PRINCIPLES AND POSSIBILITIES” and then believe very little of what passes for “knowledge” on the subject through the media.

See also for richer models of teaching Quality Education in NSW which includes but takes us beyond OBE and further from the more dreamily romantic or hippie-like aspects of “progressive education”, and my post on scaffolding.

Yes, I get frustrated… 😉

Later

Further to this post, see, mostly on New Lines from a Floating Life:

HSC English Paper 2 Advanced 2006 03Nov06.

Julie Bishop’s third-hand knowledge of English teaching 08Oct06.

Three magazines and an amazing AIDS story… 11Jul06.

Right-wing non-news story #20563 10Jun06.

Dumbed down syllabuses 30Apr06.

My past catches up 25Apr06.

Reactionary myth-makers and education 22Apr06.

Literacy.

And just one item from my old blog on Blogspot (if you can see it):

History debate rages over loss of narrative – Top stories – Breaking News 24/7 – NEWS.com.au.

See too Jim Belshaw’s kind response. That blog of his is a goldmine of background information on public administration, very revealing on the broader context from which current administrative approaches to accountability in education have sprung.

 

Scaffolding

You can view the Powerpoint slideshow used at the presentation to the final conference at UTS in August 2003. You may also read a 2005 article on the project by Jennifer Hammond and Pauline Gibbons: Putting Scaffolding to Work.

The term “scaffolding” is a metaphor taken from the building industry. There it is a temporary structure that is taken away when the building can stand alone. Good teachers often do it “automatically”. On the other hand, some “scaffolding” (if done in a mechanical fashion) may be unnecessary, even counter-productive.

Starting in 2001 and continuing in 2002 with 7E and 7F, a research project on scaffolding was conducted at Sydney Boys High School (and other secondary and primary schools), a collaboration between the Department of Education and Training (Multicultural Branch) and the University of Technology Sydney. Research team leader was Dr Jennifer Hammond of UTS.

7F 2002 was the original target class. The researchers looked at English, Geography and ESL. In 2002 the classes were in English and French.

In 2003 the data was studied and conclusions drawn, culminating in a presentation at UTS in August 2003.

Among the findings of interest were:

— The metaphor of “scaffolding” is misleading in being too rigid. Pauline Gibbons (UTS) suggests we might rather think of the bamboo scaffolding seen in South-East Asia, which can bend with the wind.

— There are at least two levels of scaffolding. First DESIGNED-IN — strategies built into the unit and lesson planning which take into account where students currently are and seek to enable them to move in the direction required. Second CONTINGENT — the teacher responds to the flow of the lesson and offers scaffolding as need arises.

— The object of successful scaffolding is to make itself unnecessary. Learning takes place best in an environment of high challenge with high support, but as tasks are mastered the teacher hands over to the student and the scaffolding is removed. For more, see a seminal article by Luciano Mariani.

— The common distinction between teacher-centred and learner-centred is unhelpful; what in fact happens is a constant interplay between the two. Classes where students are seen as worthy conversational partners do tend to be marked by successful scaffolding, however, as in such classes teachers are more attuned to the degree to which students are taking on new knowledge.

It is within the students, of course, that the learning occurs, but it is within the teacher, who sits at the juncture of the forces above, below and sideways that the learning situations are framed. (Bazerman, C. (1994), ‘Where is the classroom?’, from A. Freedman & P. Medway (eds), Learning and teaching genre, Portsmouth NH, cited in Johns, A. M. (1997), Text, Role and Context: Developing Academic Literacies, Cambridge, C.U.P.)

— Students prosper best in an environment of MESSAGE ABUNDANCE. That is, as many channels of communication as possible are used to support a variety of learning styles.

— Teachers of students from language backgrounds other than English cannot take too much background English language/cultural knowledge for granted.

— Key terms should always be written up on the board, shown on overhead or in some kind of presentation program, and carefully defined and demonstrated in context.

— Much use should be made of diagrams, flow charts, mindmaps and other means of visualising concepts.

— Group and pair work should be used frequently as this gives more opportunities for contingent scaffolding, for students to articulate ideas for themselves, and for handover to take place. In Ms Walles’s class especially, drama methods such as hotseating proved powerful strategies for developing understanding and creativity.

— At a Bondi District ESL Information Network Meeting in late 2003 the topic was ESL in Science. Close study of answers from the HSC Standards Package for Physics reveals clearly that certain text types and a certain level of language are critical for best results in that subject, more now than in the past. The text types needed for success in each subject should be explicitly modelled, analysed and practised. For example, see Teaching Science to ESL students. There is excellent material on the ESL Page of the Victorian Department of Education Curriculum @ Work Pages: also links to Art, English, Health and PE, LOTE, Mathematics, Technology, Social Sciences, with mainstream ESL ideas for all of them.

# Good teachers often do all of the above intuitively or through experience. Ms Guthrie (the Modern Languages teacher involved in the project in 2002; Ms Guthrie passed away in 2007, I am sorry to report) commented on how affirming it was to see just why certain procedures worked. Ms Ross, a new teacher at the time, found the thinking involved of great benefit; being able to see on the subsequent videos just how scaffolding was working out in her lessons was a very positive piece of professional development. To be more conscious of scaffolding can also assist any teacher in more effective unit and lesson preparation.

More ideas on scaffolding

Some definitions and guidelines, adapted from Anita J Woolfolk, Educational Psychology (Edition 7, Allyn and Bacon, Boston 1998).

Scaffolding: Support for learning and problem solving. The support could be clues, reminders, encouragement, breaking the problem into steps, providing an example, or anything else that allows the student to grow in independence as a learner.
Assisted learning: Providing strategic help in the initial stages of learning, gradually diminishing as students gain independence.
Zone of proximal development: Phase at which a student can master a task if given appropriate help and support. (It can be argued that it is this zone that all learning occurs.–NW) See also Vygotsky’s Zone of Proximal Development.

Some strategies to scaffold complex learning:

* Procedural facilitators: e.g. encouraging students to use “signal words” such as who, what, when, where, why and how after reading a passage.
* Modelling: the teacher might model the generation of questions about the reading.
* Thinking out loud: this models the teacher’s expert thought processes.
* Anticipating difficult areas: during the modelling phase, the teacher anticipates student errors or difficulties.
* Providing prompt or cue cards: these may be used by the students in an early phase of learning and withdrawn as the students master the material.
* Regulating the difficulty: tasks involving implicit skills are introduced by beginning with simpler problems, providing the students with extra practice after each step, and gradually increasing the complexity of the task.
* Providing half-done examples: giving students half-done examples and having them work out the conclusions can be an effective way to teach students how to ultimately solve problems on their own.
* Reciprocal teaching: having the teacher and students rotate the role of teacher. The teacher provides support to students as they learn to lead discussions and ask their own questions.
* Providing checklists: students can be taught self-checking procedures (see an example by another ESL teacher below) to help them regulate the quality of their responses.

Guidelines:

Tailor scaffolding to the needs of students.

* When students are beginning new tasks and topics, provide models, prompts, sentence starters, coaching and feedback. As the students grow in competence, give less support and more opportunities for independent work
* Give students choices about the level of difficulty or degree of independence in projects; encourage them to challenge themselves but to seek help when they are really stuck.

Make sure students have access to powerful tools that support thinking.

* Teach students to use learning and organisational strategies, research tools, language tools (dictionaries or computer searches), spreadsheets, and word processing programs.
* Model the use of tools; show students how to use an appointment book or electronic notebook to make plans and manage time, for example.

Capitalise on dialogue and group learning

* Experiment with peer tutoring; teach students how to ask good questions and give helpful explanations.
* Experiment with cooperative learning strategies, small group work, and so on.

Some teachers will be familiar with scaffolding as a phase in the “learning-teaching cycle” promoted, quite rightly, in much of the material on literacy.

There is a danger, however.

* First, ask yourself if the scaffolding is really necessary. If the text type (for example) is already familiar, scaffolding may not be necessary.
* Second, ask yourself if the scaffolding is in the most appropriate form or terminology. I have read examples that even I have trouble understanding, and which are in fact more difficult than the base task. It may be the fault of curriculum documents and instructional materials which, in their desire to open up new approaches and concepts, stray way beyond the zone of proximal development of staff, let alone students!
* Third, ask yourself if the scaffolding/modelling is unnecessarily restrictive. Experience has shown that a too rigid adherence to text-type models can inhibit students, and may indeed not reflect the real range of possibilities in addressing a particular context, purpose or audience. See the DET document Focus on Literacy: Writing [2000] for more discussion:

The main message of Focus on literacy: Writing is that our focus should be on the social purpose of anything we ask students to write. Often our purpose is to engage, move, persuade or delight a reader. The social purpose should dictate, not only how something is written, but also how it is assessed.

Our main focus should be on social purpose, not text type. Text types are only typical ways of fulfilling certain writing purposes. Often, a writer’s purpose demands that an atypical approach has more impact, such as beginning a narrative with the ending, or not describing a character until later.

We now have evidence that too much emphasis on the text types of the primary syllabus, namely narrative, recount, explanation, discussion and exposition, can lead to poor writing, because when we write something our purpose is not to reproduce a text type but to engage a reader’s attention.

— Paul Hardage, Chief Education Officer, English, in Curriculum Support for Teaching in English 7-12, 2001 Vol 6 No 1.

Go Forth and Correct Thyself

Written by another experienced ESL teacher who prefers to remain anonymous.

Remember the luxury of writing when there was time: being able to reflect and meditate on ideas and words; editing to erase the things you really didn’t want to say; appreciating your growth in written communication?

When I was teaching adult learners, students wondered how they would ever do those things in English and often asked me, “How will I improve? How will I know how to write correctly when the teacher isn’t with me?”

So I introduced a systematic approach to writing analysis that would encourage and develop the students’ ability to self-correct. We discussed the important elements of writing, and categories for analysis were selected. For example: meaning, text structure, many facets of grammar, spelling, punctuation, handwriting…whatever the students were able to understand at their stage of writing. An abbreviation was assigned to each writing component. When the students handed in a piece of writing, I identified any errors with the appropriate abbreviation. The students recorded their types of errors, then attempted self-correction. The purpose of keeping a diary of their errors was so they could focus on particular weaknesses in subsequent writing tasks. As their experience of writing grew, they became more skilled at identifying their problems and amending them.

I also encouraged students to read their texts aloud to themselves. Often they could more easily hear anomalies than recognise them in writing.

My comments

* This is a highly scaffolded approach to the difficult question of error. Wherein is the scaffolding? How does this method locate the student’s zone of proximal development? How does it balance challenge and support?
* A modification of the method I have used is to focus on one class of error–subject-verb agreement for example–and highlight (rather than mark out) instances of the error. The student then attempts self-correction.

See also: Literacy and ESL on the AIS Literacy Site; Scaffolding Literacy; Roslyn Arnold, Empathic intelligence in pedagogy.

# Here’s a good education blog, and a clever pun: The Blog of Proximal Development. The writer, a Canadian, has a fine understanding of pedagogy (there is NOTHING wrong with that word) and literacy teaching. I recommend it very warmly.