[Rasch] qualitatively described, quantitatively responded

shirin shirazi shirin71_shirazi at hotmail.com
Sat Nov 10 06:44:54 EST 2007


Hi Theo,
 
I have some questions to ask if you don't mind:
 
1. How do raters score complexity? How is complexity operationally defined ? Complexity of the message, complexity of lexis, complexity of syntax? How is level of abstraction determined? If you say complexity of ideas, Can you elaborate on complexity of ideas and how is it possible to make differentation between different papers as to this characteristic? Moreover why are the structure and level of abstraction of the argument in focus?
 
2. Essays are scored differently for different purposes: for example we have primary trait scoring, holistic scoring, multiple-trait scoring, analytic scoring. Which one are you going for and why? Genre specifies the context for primary trait scoring where we look for a particular trait in essays, for instance coherence. The criterion for scoring is based on how we define the construct of writing. Can we define the construct as consisting of coherence plus complexity?  
 
3. Coherence is made up of many subcomponents or subcategories. I wonder how each one is scored. Is each dimenstion weighted equally? The question I asked a few months ago from the list.
 
4. Supposing that the best way is to break down different dimensions and allocate a score to each. Are they of the same importance in scoring? I mean complexity is more important or coherence? It should be based on theory. Is there one to support the construct we are measuring?
 
Cordially,
shirin
 


Date: Fri, 9 Nov 2007 06:16:33 -0800From: ici_kalt at yahoo.comSubject: Re: [Rasch] qualitatively described, quantitatively respondedTo: theo at devtestservice.com; rasch at acer.edu.auCC: 
Hi Theo: Of course an essay cannot be scored as a single dimension. When you say complexity, coherence, and many more: sintax, logic, order or the text (introduction/development/conclusion), relevance of the message, etc. EACH one is a single dimension, and we have to try to measure each one separately.
I think the beginning of this discussion, following your example, is to treat an essay as a multidimensional variable that can be measured with a single number. Many in this listserve say no to this approach, that is the reason why we have items trying to measure one dimension each one: certainly there are several items measuring "coherence" (not a single item to measure coherence), and other set of items to measure "complexity", and so forth. We don't like to see a set of items measuring the multidimensional essay as a hole, with my regards to people who likes holistic scoring.
Yours
Agustin
Theo Dawson <theo at devtestservice.com> wrote:
Rating the quality of essays on a single dimension is a common practice in educational assessment---not good. This is because the quality of an essay is dependent upon multiple factors that are related to one another in complex ways. To name two: 

Complexity (structure and level of abstraction of the argument)

Coherence

These are not only different dimensions; under certain common conditions, a higher level of performance on one can cause a lower level of performance on another. For example, when a student reaches beyond his or her modal level of reasoning complexity on a particular subject (when she or he is engaged in constructing new knowledge), coherence is likely to suffer---even if the writer is a highly competent in these areas when writing at a lower level of complexity. This creates the unacceptable consequence that a student who is working with more sophisticated ideas can be awarded a lower score than a student working with less sophisticated ideas. Worse, clever students learn that they will not be rewarded for stretching, creating a situation in which deep learning is discouraged by grading practices.

My colleagues and I are currently accumulating a database of essays scored independently for complexity level and coherence. If you have essays (in electronic form) that you can contribute to this database, please contact me.

Theo


On Nov 6, 2007, at 6:28 PM, Agustin Tristan wrote:

shirin shirazi <shirin71_shirazi at hotmail.com> wrote: 
Hi  Thanks for your concern. I have thirty essays to rate. There is a rating scale consisting of nine descriptors. The first one describes the writing ability of the writer as a "fully competent". If the writer suits this level, he gets nine. If he adequately responds to the task, it brings him down to eight and the same process continues up until zero where there is no response to the task. I wonder how these numbers have been given to these qualitative words. Fully 9, adequately 8, to some extent 7, partially 6, ... The second question is that why do we have a range of descriptors from nine (at most) to three (at least)? Third why don't we have qualitative words to describe levels of performance since descriptors are describing the proficiency in qualitative manner, for example instead of earning a 9 in IELTS, we allocate a word such as competent or incompetent or effective or ineffective which are in harmony with qualitative words appearing in descriptors. Do we use numbers since they are more tangible (People come to grips with them easily)? Thanks in advance and your help is greatly appreciated. BestShirin


Date: Mon, 5 Nov 2007 15:43:55 -0800From: ici_kalt at yahoo.comSubject: Re: [Rasch] qualitatively described, quantitatively respondedTo: shirin71_shirazi at hotmail.com
hello! what is exactly the problem you are trying to solve? describe your test please.
regards
agustin tristanshirin ahmadishirazi <shirin71_shirazi at hotmail.com> wrote:

Hi List members,
 
I would be extremely grateful if you help solve the following controversies:
 
1. Qualitative words are not appropriate to describe a quantitative construct. 
2. Qualitative words are used to differentiate descriptors of levels of proficiency (although it is difficult to make distinctions between "some" and "a few", and "several" and "many").
3. Productive skills (Speaking and writing) are qualitatively described but quantitatively scored. How can qualitative words (e.g., fully, sufficiently, partially, minimally, largely, adequately, effectively, ...) get quantified?
 
All the Best,
Shirin 

Connect to the next generation of MSN Messenger  Get it now!_______________________________________________Rasch mailing listRasch at acer.edu.auhttp://mailinglist.acer.edu.au/mailman/listinfo/rasch
FAMILIA DE PROGRAMAS KALT.
Mariano Jiménez 1830 A
Col. Balcones del Valle
78280, San Luis Potosí, S.L.P. México
TEL (52) 44-4820 37 88, 44-4820 04 31
FAX (52) 44-4815 48 48
web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com__________________________________________________Do You Yahoo!?Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com 

Help yourself to FREE treats served up daily at the Messenger Café. Stop by today!
FAMILIA DE PROGRAMAS KALT.
Mariano Jiménez 1830 A
Col. Balcones del Valle
78280, San Luis Potosí, S.L.P. México
TEL (52) 44-4820 37 88, 44-4820 04 31
FAX (52) 44-4815 48 48
web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com
__________________________________________________Do You Yahoo!?Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________Rasch mailing listRasch at acer.edu.auhttp://mailinglist.acer.edu.au/mailman/listinfo/rasch
FAMILIA DE PROGRAMAS KALT. 
Mariano Jiménez 1830 A 
Col. Balcones del Valle 
78280, San Luis Potosí, S.L.P. México 
TEL (52) 44-4820 37 88, 44-4820 04 31 
FAX (52) 44-4815 48 48 
web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com
__________________________________________________Do You Yahoo!?Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com 
_________________________________________________________________
Windows Live Hotmail and Microsoft Office Outlook – together at last.  Get it now.
http://office.microsoft.com/en-us/outlook/HA102225181033.aspx?pid=CL100626971033
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071109/ae7a2140/attachment.html 


More information about the Rasch mailing list