[Rasch] Estimating Rasch Measures for Extreme Scores

Patrick B. Fisher pfisher at SportsMeasures.com
Sun May 1 00:29:46 EST 2011


If I understand the original question Jason, it seems to me that you 
actually have a classic Facets set up. I don't call you specifying which 
program (Facets, Winsteps, RUMM, etc) your student used. It would appear 
to me that the data set is Teachers grading/rating Students on 
Subjects/classes/projects. If I remember correctly, the only way a 
student would have an extreme score in this instance is if all teachers 
always gave perfect scores to every student on every subject. So, as 
soon as a teacher gives less than a perfect score to one student, that 
teacher no longer represents an extreme rater and would keep 'perfect' 
students from being extreme. I think Facets solves the problem.

Please forgive me if I've misunderstood something.

Patrick Fisher

On 4/30/2011 9:03 AM, Agustin Tristan wrote:
> Dear Iasonas:
> I think you are facing some problems using the Rasch model, that would 
> be the same using raw scores or IRT or any other model. I am certain 
> that the problem does not concern the Rasch model but the origin of 
> your data, I mean the validity of the scores that your student is 
> using for the analysis, and hence the problem concerns the test design.
> I understand that a judge sets a score from A to E according to 
> specific criterion (including technical, ethical and other elements), 
> but you only have ONE score that may represent a set (big or small) of 
> items. In cases like yours I think it is better to define an 
> analytical criterion where every aspect or trait will be translated 
> into one item and instead an A you may have 5, 6, 10, 20 or more 
> items. This eliminates the fact that several persons will have extreme 
> scores without meaning because it is very improbable that all the As 
> mean exactly the same ability, knowledge, competency, etc. among 
> different persons.
> To give you an example, we proceeded with an analytical criterion to 
> score performances of children (6 to 8 years old) in several 
> activities at school. In this case where the teachers used to assign a 
> letter i a competencey (say spatial competency, verbal competency, and 
> so on), we got several items (position up-down, side left-right, size 
> big-small, angle vertical-horizontal-inclined, selection of color, etc.).
> It is possible that you are not able to recode or translate the 
> holistic values into analytical items, and your problem will still be 
> there, nevertheless I am sure that you can be more than sure that it 
> is not a Rasch model defect but a test design defect. I consider that 
> statistics or CTT or Rasch model should not be used to solve the test 
> design problem, nevertheless the other proposals by colleagues in this 
> listserve are very wise.
> Regards
> Agustin
>
>
> *INSTITUTO DE EVALUACION E INGENIERIA AVANZADA.*
> **
> *
> *Ave. Cordillera Occidental No. 635
>
> Colonia Lomas 4ª San Luis Potosí, San Luis Potosí.
>
> C.P. 78216 MEXICO
>
> (52) (444) 8 25 50 76 / (52) (444) 8 25 50 77 / (52) (444) 8 25 50 78
>
> Página Web (en español): http://www.ieia.com.mx <http://www.ieia.com.mx/>
>
> Web page (in English): 
> http://www.ieesa-kalt.com/English/Frames_sp_pro.html
>
>
> --- On *Sat, 4/30/11, liasonas at cytanet.com.cy 
> /<liasonas at cytanet.com.cy>/* wrote:
>
>
>     From: liasonas at cytanet.com.cy <liasonas at cytanet.com.cy>
>     Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
>     To: "Luis Carlos Orozco" <lcorovar at gmail.com>
>     Cc: "rasch list" <rasch at acer.edu.au>
>     Date: Saturday, April 30, 2011, 12:17 AM
>
>     Thank you,
>
>     indeed, it seems that there are too many people reaching the
>     ceiling effect with a 3x4x (1-5 scale) design. Maybe this is one
>     of the weak points of the current system...but the point is that
>     somebody argues that the use of raw scores would solve the problem
>     of the raw scores, and we may need to see if there is a published
>     answer to this...
>     Sent from my BlackBerry® smartphone
>     ------------------------------------------------------------------------
>     *From: *Luis Carlos Orozco <lcorovar at gmail.com>
>     *Date: *Fri, 29 Apr 2011 19:17:34 -0500
>     *To: *Iasonas Lamprianou<liasonas at cytanet.com.cy>
>     *Cc: *rasch list<rasch at acer.edu.au>
>     *Subject: *Re: [Rasch] Estimating Rasch Measures for Extreme Scores
>
>     Maybe this from winsteps  be of some "help".
>     There is no "correct" answer to the question: "How large should
>     ExtremeScoreAdjustment= be?" The most conservative value, and that
>     recommended by Joseph Berkson, is 0.5. Some work by John Tukey
>     indicates that 0.167 is a reasonable value. The smaller you set
>     EXTRSC=, the further away measures corresponding to extreme scores
>     will be located from the other measures. The technique used here
>     is Strategy 1 in www.rasch.org/rmt/rmt122h.htm
>     <http://www.rasch.org/rmt/rmt122h.htm>.
>
>     Another thing to say is that your measure obtained from (4x3) is
>     not good enough to measure some people.
>
>     Luis C. Orozco V.
>
>     MD MSc Epidemiologia
>
>     EScuela de Enfermería
>
>     Universidad Industrial de Santander
>
>     Colombia
>     2011/4/29 Iasonas Lamprianou <liasonas at cytanet.com.cy
>     <http://us.mc1115.mail.yahoo.com/mc/compose?to=liasonas@cytanet.com.cy>>
>
>
>         Dear colleagues,
>         I rarely submit requests in this list unless it is urgent and
>         important because I respect the time of the people who tend to
>         reply most often. I would like to thank them. This time, it is
>         important and that is why I politely request you to help me.
>         The post is long, but it has to do with the problems
>         Rasch-users face in the harsh world of academia. I think that
>         the post concerns most of us.
>
>         I am trying to help a student with her PhD thesis (so I am
>         writing on her behalf). She submitted her thesis and her
>         examiners spotted some problems and she has to address them.
>
>         The problem: The PhD thesis is about the performance of
>         students.  For each student participating in the study
>         (N>1000), the researcher has his/her score on four subjects:
>         language, science, maths and history. For each subject, each
>         student has three teacher assessments which were awarded in
>         January, March and June. Each score runs from E (Failure) to A
>         (Excellent). So, overall, each student has three ordinal
>         teacher assessment measures for each of four subjects. It is a
>         typical repeated measures case for four variables/subjects
>         with three measures per variable/subject.
>
>         Design: Since the data are ordinal (E=1=Failure to
>         A=5=Excellent) the researcher used a Partial Credit Rasch
>         model with three  items  to build four Ability scales, one for
>         each subject (the Rating Scale did not have good fit). Also,
>         the student used all 12 scores (4 subjects X 3 measures) to
>         produce one overall Ability  Academic Performance  measure.
>         Then, the researcher used these Rasch ability measures as
>         dependent variables to run OLS regressions.
>
>         Issue 1:
>         A serious problem spotted by the examiners is that a large
>         proportion of students (around 20%) has perfect scores (three
>          A s) on some of the four subjects. The researcher used a
>         Winsteps routine to find measures of ability for those
>         students with extreme scores. The examiner has major
>         reservations about the validity of this decision and asks
>         whether these data (extreme scores) should be dropped. The
>         examiner says:  If a Rasch analysis is to be used to derive
>         attainment scores, the final distribution must provide a
>         realistic representation of attainment. This means that the
>         large group of candidates who achieve perfect scores (on the
>         extreme right of the histograms) need to be properly
>         represented. These scores need to be appropriately dealt with
>         by Rash (if this is possible), or they need to be removed from
>         the analysis (with an
>         assessment made about the impact of the resulting loss of data).
>         To the defense of the researcher, the distance between the
>          perfect score  and the  perfect-1  estimate is neither huge
>         nor unreasonable: it is around 1.4 logits on a scale which
>         extends from around -11 to 11 logits. When the researcher
>         draws the scatterplot between raw scores and logits, the
>         sigma-curve looks beautifully smooth and the estimates of the
>         extreme scores look neither  too extreme  nor out of tune with
>         the rest data points on the scatterplot. The distance between
>         the  perfect score  and the  perfect-1  estimate is not
>         grossly out of line compared to the other distances between
>         raw scores estimates (for example, the distance between the
>          perfect-1  and the  perfect-2  scores is only around 0.3
>         logits smaller).
>         (a) The researcher needs strong references to defend her
>         decision NOT to drop the extreme data estimates. Can anyone
>         please provide strong peer-reviewed papers to support the
>         decision to keep the extreme score estimates as valid
>         representations of the ability of the participants?
>
>
>         Issue 2:
>         Stemming from the previous comment, one of the suggestions of
>         the examiners is that the researcher could ditch the Rasch
>         model and instead sum the three measures in one subject (e.g.
>         A+B+B=5+4+4=13) and then use this sum for an OLS regression.
>         The examiner says  A serious discussion needs to be held about
>         the benefits, if any, the Rasch analysis provides over a more
>         direct analytical path (e.g.   a linear regression of results
>         averaged over three   [teacher assessments] . We all know that
>         this is simply wrong to do because we cannof average ordinal
>         measures and the student already explains this in her
>         Methodology section, but she probably needs more references.
>         (b) Can anyone please provide a list of (recent, if possible)
>         papers in good peer-reviewed journals which explain that this
>         is not the right thing to do?
>
>
>         Issue 3:
>         Another suggestion of the examiners is that the researcher
>         could ditch the Rasch model and just use the ordinal measure
>         (E=1=Failure to A=5=Excellent) as a dependent variable in a
>         proportional odds models. This means that the researcher
>         should run three different models for each subject (for the
>         Teacher Assessment awarded in January, March and June).
>         (c) Can anyone pleased provide a list of (recent, if possible)
>         papers in good peer-reviewed journals which explain that this
>         is NOT better than using the Rasch model to get one linear
>         measure instead of three ordinal?
>
>
>         I feel that the examiners did a very good job overall and were
>         very fair and consistent. They spent too much time to read
>         every little detail in a long thesis, they spotted some
>         important issues and we need to credit them for this. I feel
>         that we may want to help the student address these interesting
>         issues to the full satisfaction of the examiners.
>
>         Thank you for your time
>
>         In anticipation of your help
>         Jason Lamprianou
>         University of Cyprus
>
>
>
>         _______________________________________________
>         Rasch mailing list
>         Rasch at acer.edu.au
>         <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
>         Unsubscribe:
>         https://mailinglist.acer.edu.au/mailman/options/rasch/lcorovar%40gmail.com
>
>
>
>     -----Inline Attachment Follows-----
>
>     _______________________________________________
>     Rasch mailing list
>     Rasch at acer.edu.au
>     <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
>     Unsubscribe:
>     https://mailinglist.acer.edu.au/mailman/options/rasch/ici_kalt%40yahoo.com
>
>
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe: https://mailinglist.acer.edu.au/mailman/options/rasch/pfisher%40sportsmeasures.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20110430/36750184/attachment.html 


More information about the Rasch mailing list