[Rasch] Re: consistency of markers across years

Ian Wilson I.Wilson at uws.edu.au
Tue Jul 22 16:08:26 EST 2008

HI Jason

A little over 12 months ago a colleague of mine in New Zealand presents
data on stringency and leniency in assessors in a medical assessment
over a 5 (maybe 3) year period.  He used z-scores above and below the
mean assessor score for a given station to categorise people as hawks
and doves.


Across time he was able to show that the majority of people were
consistency hawks or doves - however there was a small but significant
group that went from hawk to dove and back (or vice versa).  The swings
in this group were quite wide.


So for the majority the hawkishness or doveishness were consistent
across time.  





Ian Wilson

Professor of Medical Education

University of Western Sydney

Locked Bag 1797

Penrith South DC NSW 1797


T: +61 2 9852 4642  F:+61 9852 4701

E: i.wilson at uws.edu.au


From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of iasonas lambrianou
Sent: Tuesday, July 22, 2008 3:46 PM
To: rasch at acer.edu.au
Subject: [Rasch] Re: consistency of markers accross years


Dear friends, 

I have data from 13 high-stakes examinations ion English as a foreign
language. I want to find a way to predict the performance of markers
(i.e. raters/readers) on many aspects (e.g. fit statistics, harshness,
restriction of range, etc) from their past performance, so as to be able
to choose only the best markers. However, I am not sure if using data
from one examination is a good predictor for the next. 
For this purpose, and since there is no similar research in the
literature, I was 'encouraged' to conduct a study for the organisation
for which I work. Although it was a lot of work, I analyszed with Facets
all the 13 datasets separately, using MFRM. I left the students float
and centered the items and the markers. Each examination has from 10-25
markers. Some of the markers are the same for each exam. So, I have 2
markers who have participated in 10 exams, 4 markers who participated in
9 exams .... and around 40 who only participated in one exam. 
I now want to do the following: 
(a)check if there is a trendline in the performance of the markers (it
seems that there is none) 
(b) check if the markers (as a group) show a trend accross time. 

It seems that the [observed average score] -[fair average score] follows
a diminishing line. But these are not comparable from year to year,
right? I mean, how can I make the marker statistics comparable? Note
that the candidates and the tests are not the same from year to year. 



Dr. Iasonas Lamprianou 
CFAS, School of Education 
The University of Manchester 
Oxford Road, Manchester M13 9PL, UK 
Tel. 0044 161 275 3485 
iasonas.lamprianou at man.ac.uk 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20080722/2f3726aa/attachment.html 

More information about the Rasch mailing list