[Rasch] qualitatively described, quantitatively responded

Theo Dawson theo at devtestservice.com
Sat Nov 10 08:48:59 EST 2007


On Nov 9, 2007, at 2:44 PM, shirin shirazi wrote:

> Hi Theo,
>
> I have some questions to ask if you don't mind:
>
> 1. How do raters score complexity? How is complexity operationally  
> defined ? Complexity of the message, complexity of lexis, complexity  
> of syntax? How is level of abstraction determined? If you say  
> complexity of ideas, Can you elaborate on complexity of ideas and  
> how is it possible to make differentation between different papers  
> as to this characteristic? Moreover why are the structure and level  
> of abstraction of the argument in focus?

We assess the hierarchical complexity of the texts with the Lectical  
Assessment System. In a nutshell, we examine the logical structure of  
arguments and meanings in a text, "looking through" the particular  
content. Scoring is described in many of the articles downloadable  
from http://devtestservice.com/index-4.html. There is also a web site  
dedicated to scoring: http://lectica.info

> 2. Essays are scored differently for different purposes: for example  
> we have primary trait scoring, holistic scoring, multiple-trait  
> scoring, analytic scoring. Which one are you going for and why?  
> Genre specifies the context for primary trait scoring where we look  
> for a particular trait in essays, for instance coherence. The  
> criterion for scoring is based on how we define the construct of  
> writing. Can we define the construct as consisting of coherence plus  
> complexity?

For the purpose of this work, coherence is regarded as one trait and  
complexity is regarded as a second trait. No assumptions are made  
about their relationship. Neither do we assume that these two traits  
describe everything we need to know about an essay. I started the  
project because I noticed a possible trend in the essays written by my  
students.

> 3. Coherence is made up of many subcomponents or subcategories. I  
> wonder how each one is scored. Is each dimenstion weighted equally?  
> The question I asked a few months ago from the list.

We are not yet satisfied with our approach to evaluating coherence. In  
this area we are relative amateurs. Does anyone know of a good scale  
for coding coherence that does not confound it with complexity?

> 4. Supposing that the best way is to break down different dimensions  
> and allocate a score to each. Are they of the same importance in  
> scoring? I mean complexity is more important or coherence? It should  
> be based on theory. Is there one to support the construct we are  
> measuring?

I think importance is at least somewhat dependent upon context.  
Certainly, we want students to come to reason in ways that  
increasingly encompass the complexity of subject-matter. But when  
writing is used to communicate with others, coherence is clearly as  
important as complexity. (And, to complicate things further, one may  
need to be able to adjust the level of complexity for one's audience.)  
But writing is also a means for scaffolding one's own thinking, which  
makes it a powerful support for cognitive development. This means that  
there may be situations in which complexity should trump coherence.  
Another wrinkle is that some students have never learned the skills  
required to produce a coherent argument at any level of complexity. We  
need a way to differentiate between these students and those who are  
temporarily disorganized as a consequence of stretching.

As an employer, I want to know (1) the highest level of complexity at  
which a potential hire can write coherently on subjects relevant to  
his or her work, and (2) how well that person can adjust the  
complexity level of his or her writing to accommodate an audience. If  
I was on an admissions committee, I'd want to know that a student  
could write coherently at some level, but I'd also want to know how  
far that student could stretch in terms of complexity. In some  
situations I'd choose evidence of more complex thinking over evidence  
of coherence.

I hope I've answered the right question...

>
> Date: Fri, 9 Nov 2007 06:16:33 -0800
> From: ici_kalt at yahoo.com
> Subject: Re: [Rasch] qualitatively described, quantitatively responded
> To: theo at devtestservice.com; rasch at acer.edu.au
> CC:
>
> Hi Theo: Of course an essay cannot be scored as a single dimension.  
> When you say complexity, coherence, and many more: sintax, logic,  
> order or the text (introduction/development/conclusion), relevance  
> of the message, etc. EACH one is a single dimension, and we have to  
> try to measure each one separately.
> I think the beginning of this discussion, following your example, is  
> to treat an essay as a multidimensional variable that can be  
> measured with a single number. Many in this listserve say no to this  
> approach, that is the reason why we have items trying to measure one  
> dimension each one: certainly there are several items measuring  
> "coherence" (not a single item to measure coherence), and other set  
> of items to measure "complexity", and so forth. We don't like to see  
> a set of items measuring the multidimensional essay as a hole, with  
> my regards to people who likes holistic scoring.
> Yours
> Agustin
>
>
> Theo Dawson <theo at devtestservice.com> wrote:
> Rating the quality of essays on a single dimension is a common  
> practice in educational assessment---not good. This is because the  
> quality of an essay is dependent upon multiple factors that are  
> related to one another in complex ways. To name two:
>
> Complexity (structure and level of abstraction of the argument)
> Coherence
>
> These are not only different dimensions; under certain common  
> conditions, a higher level of performance on one can cause a lower  
> level of performance on another. For example, when a student reaches  
> beyond his or her modal level of reasoning complexity on a  
> particular subject (when she or he is engaged in constructing new  
> knowledge), coherence is likely to suffer---even if the writer is a  
> highly competent in these areas when writing at a lower level of  
> complexity. This creates the unacceptable consequence that a student  
> who is working with more sophisticated ideas can be awarded a lower  
> score than a student working with less sophisticated ideas. Worse,  
> clever students learn that they will not be rewarded for stretching,  
> creating a situation in which deep learning is discouraged by  
> grading practices.
>
> My colleagues and I are currently accumulating a database of essays  
> scored independently for complexity level and coherence. If you have  
> essays (in electronic form) that you can contribute to this  
> database, please contact me.
>
> Theo
>
> On Nov 6, 2007, at 6:28 PM, Agustin Tristan wrote:
>
>
>
> shirin shirazi <shirin71_shirazi at hotmail.com> wrote:
>
> Hi
>
> Thanks for your concern.
>
> I have thirty essays to rate. There is a rating scale consisting of  
> nine descriptors. The first one describes the writing ability of the  
> writer as a "fully competent". If the writer suits this level, he  
> gets nine. If he adequately responds to the task, it brings him down  
> to eight and the same process continues up until zero where there is  
> no response to the task. I wonder how these numbers have been given  
> to these qualitative words. Fully 9, adequately 8, to some extent 7,  
> partially 6, ...
>
> The second question is that why do we have a range of descriptors  
> from nine (at most) to three (at least)?
>
> Third why don't we have qualitative words to describe levels of  
> performance since descriptors are describing the proficiency in  
> qualitative manner, for example instead of earning a 9 in IELTS, we  
> allocate a word such as competent or incompetent or effective or  
> ineffective which are in harmony with qualitative words appearing in  
> descriptors. Do we use numbers since they are more tangible (People  
> come to grips with them easily)?
>
> Thanks in advance and your help is greatly appreciated.
>
> Best
> Shirin
> Date: Mon, 5 Nov 2007 15:43:55 -0800
> From: ici_kalt at yahoo.com
> Subject: Re: [Rasch] qualitatively described, quantitatively responded
> To: shirin71_shirazi at hotmail.com
>
> hello! what is exactly the problem you are trying to solve? describe  
> your test please.
> regards
> agustin tristan
>
> shirin ahmadishirazi <shirin71_shirazi at hotmail.com> wrote:
>
> Hi List members,
>
> I would be extremely grateful if you help solve the following  
> controversies:
>
> 1. Qualitative words are not appropriate to describe a quantitative  
> construct.
> 2. Qualitative words are used to differentiate descriptors of levels  
> of proficiency (although it is difficult to make distinctions  
> between "some" and "a few", and "several" and "many").
> 3. Productive skills (Speaking and writing) are qualitatively  
> described but quantitatively scored. How can qualitative words  
> (e.g., fully, sufficiently, partially, minimally, largely,  
> adequately, effectively, ...) get quantified?
>
> All the Best,
> Shirin
>
>
> Connect to the next generation of MSN Messenger  Get it now! 
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> http://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
>
>
> FAMILIA DE PROGRAMAS KALT.
> Mariano Jiménez 1830 A
> Col. Balcones del Valle
> 78280, San Luis Potosí, S.L.P. México
> TEL (52) 44-4820 37 88, 44-4820 04 31
> FAX (52) 44-4815 48 48
> web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
> Help yourself to FREE treats served up daily at the Messenger Café.  
> Stop by today!
>
>
>
> FAMILIA DE PROGRAMAS KALT.
> Mariano Jiménez 1830 A
> Col. Balcones del Valle
> 78280, San Luis Potosí, S.L.P. México
> TEL (52) 44-4820 37 88, 44-4820 04 31
> FAX (52) 44-4815 48 48
> web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> http://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
>
>
>
> FAMILIA DE PROGRAMAS KALT.
> Mariano Jiménez 1830 A
> Col. Balcones del Valle
> 78280, San Luis Potosí, S.L.P. México
> TEL (52) 44-4820 37 88, 44-4820 04 31
> FAX (52) 44-4815 48 48
> web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
> Windows Live Hotmail and Microsoft Office Outlook – together at  
> last. Get it now!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071109/a44e3cc6/attachment.html 


More information about the Rasch mailing list