[Rasch] Estimating Rasch Measures for Extreme Scores
Adams, Ray
adams at acer.edu.au
Mon May 2 11:57:32 EST 2011
The paper:
Adams, R.J., Wilson, M.R. and Wu, M.L. (1997) Multilevel item response
modelling: An approach to errors in variables regression. Journal of
Educational and Behavioral Statistics, 22, 47-76.
is probably a good place to start.
The central problem is that the distribution of person parameter
estimates is often not a good estimate of the "true" distribution of the
latent variable. The shorter the test, the bigger the discrepancy. In
the case of complete data, the fact that there is only one ability
estimate for each raw score makes this immediately obvious.
To convince yourself with a simple illustration generate a data set that
fits the Rasch model using a short test. Then estimate the parameters
and compute the variance of the person parameter estimates (EAP, WLE,
MLE, whatever). Compare that variance to the true variance of the
person parameters. You can easily get more complicated to convince
yourself further.
Ray
-----Original Message-----
From: liasonas at cytanet.com.cy [mailto:liasonas at cytanet.com.cy]
Sent: Saturday, 30 April 2011 7:40 AM
To: Adams, Ray; rasch
Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
Do you happen to have any accessible papers (maybe with an example) to
see how to do it and what the advantages are?
Sent from my BlackBerry(r) smartphone
-----Original Message-----
From: "Adams, Ray" <adams at acer.edu.au>
Date: Sat, 30 Apr 2011 07:20:55
To: <liasonas at cytanet.com.cy>; rasch<rasch at mailinglist.acer.edu.au>
Subject: RE: [Rasch] Estimating Rasch Measures for Extreme Scores
There are quite a few ways to do the latent regression with a Rasch
model, which would be the proper approach here. Yes, ConQuest can do
it, so to can the likes of Mplus.
-----Original Message-----
From: liasonas at cytanet.com.cy [mailto:liasonas at cytanet.com.cy]
Sent: Saturday, 30 April 2011 7:20 AM
To: Adams, Ray; rasch
Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
How can this be done? Is it doable via conquest? Basically, she is using
the ability estimate as a dependent for linear regressions
Sent from my BlackBerry(r) smartphone
-----Original Message-----
From: "Adams, Ray" <adams at acer.edu.au>
Date: Sat, 30 Apr 2011 06:55:48
To: Iasonas Lamprianou<liasonas at cytanet.com.cy>;
rasch<rasch at mailinglist.acer.edu.au>
Subject: RE: [Rasch] Estimating Rasch Measures for Extreme Scores
Did the student take the measurement error in the Rasch estimates into
account when doing the ANOVA/OLS regression?
Ignoring the error and using ability estimates, be they EAP, MAP, MLE or
WLE is almost never the correct thing to do.
-----Original Message-----
From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of Iasonas Lamprianou
Sent: Friday, 29 April 2011 6:39 PM
To: rasch
Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
Dear colleagues,
I rarely submit requests in this list unless it is urgent and important
because I respect the time of the people who tend to reply most often. I
would like to thank them. This time, it is important and that is why I
politely request you to help me. The post is long, but it has to do with
the problems Rasch-users face in the harsh world of academia. I think
that the post concerns most of us.
I am trying to help a student with her PhD thesis (so I am writing on
her behalf). She submitted her thesis and her examiners spotted some
problems and she has to address them.
The problem: The PhD thesis is about the performance of students. For
each student participating in the study (N>1000), the researcher has
his/her score on four subjects: language, science, maths and history.
For each subject, each student has three teacher assessments which were
awarded in January, March and June. Each score runs from E (Failure) to
A (Excellent). So, overall, each student has three ordinal teacher
assessment measures for each of four subjects. It is a typical repeated
measures case for four variables/subjects with three measures per
variable/subject.
Design: Since the data are ordinal (E=1=Failure to A=5=Excellent) the
researcher used a Partial Credit Rasch model with three "items" to build
four Ability scales, one for each subject (the Rating Scale did not have
good fit). Also, the student used all 12 scores (4 subjects X 3
measures) to produce one overall Ability 'Academic Performance' measure.
Then, the researcher used these Rasch ability measures as dependent
variables to run OLS regressions.
Issue 1:
A serious problem spotted by the examiners is that a large proportion of
students (around 20%) has perfect scores (three 'A's) on some of the
four subjects. The researcher used a Winsteps routine to find measures
of ability for those students with extreme scores. The examiner has
major reservations about the validity of this decision and asks whether
these data (extreme scores) should be dropped. The examiner says: "If a
Rasch analysis is to be used to derive attainment scores, the final
distribution must provide a realistic representation of attainment. This
means that the large group of candidates who achieve perfect scores (on
the extreme right of the histograms) need to be properly represented.
These scores need to be appropriately dealt with by Rash (if this is
possible), or they need to be removed from the analysis (with an
assessment made about the impact of the resulting loss of data). "
To the defense of the researcher, the distance between the "perfect
score" and the "perfect-1" estimate is neither huge nor unreasonable: it
is around 1.4 logits on a scale which extends from around -11 to 11
logits. When the researcher draws the scatterplot between raw scores and
logits, the sigma-curve looks beautifully smooth and the estimates of
the extreme scores look neither "too extreme" nor out of tune with the
rest data points on the scatterplot. The distance between the "perfect
score" and the "perfect-1" estimate is not grossly out of line compared
to the other distances between raw scores estimates (for example, the
distance between the "perfect-1" and the "perfect-2" scores is only
around 0.3 logits smaller).
(a) The researcher needs strong references to defend her decision NOT to
drop the extreme data estimates. Can anyone please provide strong
peer-reviewed papers to support the decision to keep the extreme score
estimates as valid representations of the ability of the participants?
Issue 2:
Stemming from the previous comment, one of the suggestions of the
examiners is that the researcher could ditch the Rasch model and instead
sum the three measures in one subject (e.g. A+B+B=5+4+4=13) and then use
this sum for an OLS regression. The examiner says "A serious discussion
needs to be held about the benefits, if any, the Rasch analysis provides
over a more direct analytical path (e.g. ... a linear regression of
results averaged over three ... [teacher assessments]". We all know that
this is simply wrong to do because we cannof average ordinal measures
and the student already explains this in her Methodology section, but
she probably needs more references.
(b) Can anyone please provide a list of (recent, if possible) papers in
good peer-reviewed journals which explain that this is not the right
thing to do?
Issue 3:
Another suggestion of the examiners is that the researcher could ditch
the Rasch model and just use the ordinal measure (E=1=Failure to
A=5=Excellent) as a dependent variable in a proportional odds models.
This means that the researcher should run three different models for
each subject (for the Teacher Assessment awarded in January, March and
June).
(c) Can anyone pleased provide a list of (recent, if possible) papers in
good peer-reviewed journals which explain that this is NOT better than
using the Rasch model to get one linear measure instead of three
ordinal?
I feel that the examiners did a very good job overall and were very fair
and consistent. They spent too much time to read every little detail in
a long thesis, they spotted some important issues and we need to credit
them for this. I feel that we may want to help the student address these
interesting issues to the full satisfaction of the examiners.
Thank you for your time
In anticipation of your help
Jason Lamprianou
University of Cyprus
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
Unsubscribe:
https://mailinglist.acer.edu.au/mailman/options/rasch/adams%40acer.edu.a
u
More information about the Rasch
mailing list