[Rasch] Negative pt-bis and fit of 1.0? How can this be?

Mark Moulton markm at eddata.com
Wed Mar 7 06:28:02 EST 2012


Gregory,

I've kind of come around to the same point of view as you.  Fit statistics
are much more complex than point-biserials (or their Rasch equivalent) and
are driven by lots of factors, making them harder to interpret.  Does an
item fit the construct of a test?  A fit statistic may or may not give a
clear answer to that question.  Point-biserials almost always do.

I've also been using a home-grown "item reliability" statistic (somewhat
like test reliability, but applied at the item level using a residual-based
"real RMSE"-like denominator), and I have found this to be very helpful as
a quick and consistent indicator of item quality.

I guess the central role of fit statistics derives from the theoretical
desire to formalize what it means to fit data to the Rasch model.  But for
some reason the resulting fit formulae aren't as practical as I would like.

Mark
Educational Data Systems


2012/3/6 Stone, Gregory <Gregory.Stone at utoledo.edu>

>  Sending this to you and not the listserv.  I have come to the conclusion
> that apart from using it as one consideration for dimensionality, fit is a
> useless statistic when it comes to assessing reasonability of item
> performance.  While I have no empirical evidence of this, gathered
> deliberately, it has appeared to me that time after time, decisions made
> with point biserials are consistently more reasonable than any made with
> fit, regardless of whether we accept the largely arbitrary range of fit so
> popularized in books like Bond and Fox, or the more rational fit calculated
> by folks like Smith. This appears to hold true for small datasets, where
> many items demonstrate fit concerns, and large datasets where everything,
> naturally, fits perfectly. Thoughts?
>
>  Gregory
>
>
>  On Mar 6, 2012, at 1:46 PM, Mark Moulton wrote:
>
> Stuart,
>
>  Belay my previous message (coffee hadn't kicked in yet).
>
>  You're right.  Variances -- p * (1 - p) at the extreme of p (1 or 0)
> will be smaller, which should make the misfits bigger.
>
>  Could those cell probabilities p[ni] actually be close to 0.50?  This
> would happen if the person measures tended to collapse to the center of the
> scale, an artifact of randomness.
>
>  Mark
>
>
>
> On Tue, Mar 6, 2012 at 7:58 AM, Stuart Luppescu <slu at ccsr.uchicago.edu>wrote:
>
>> Hello, I'm analyzing items suggested for a course final exam. The
>> problem is that the students the items are tested on have not taken the
>> course yet. This results in very poor performance. The average person
>> measure is -0.78, and the average item p-value is 0.33 (for 4-choice
>> multiple choice items).
>>
>> What is confusing me is that all the items have mean-square fit
>> statistics (infit and outfit) near 1.0, while many of them have negative
>> point-biserial correlations. According to my understanding, the fit
>> statistics are calculated from an aggregation of the squared
>> standardized residuals, which are calculated from the raw residual
>> divided by the score variance. In this case, the expectation of an
>> individual response would be low, so raw residuals would be large. And
>> the score variance at the extremes are lower than in the middle, so you
>> divide a large raw residual by a small score variance and you get a very
>> large standardized residual, right? So, how come the fits are close to
>> expectation? Especially since the pt-bis are low or negative? I don't
>> get it. Can someone explain this to me?
>>
>> Thanks.
>> --
>> Stuart Luppescu -=- slu .at. ccsr.uchicago.edu
>> University of Chicago -=- CCSR
>> 才文と智奈美の父 -=-    Kernel 3.2.1-gentoo-r2
>> What we have is nice, but we need something very
>>  different.    -- Robert Gentleman
>>  Statistical Computing 2003, Reisensburg (June
>>  2003)
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Rasch mailing list
>> Rasch at acer.edu.au
>> Unsubscribe:
>> https://mailinglist.acer.edu.au/mailman/options/rasch/markm%40eddata.com
>
>
>  _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe:
> https://mailinglist.acer.edu.au/mailman/options/rasch/gregory.stone%40utoledo.edu
>
>
>
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe:
> https://mailinglist.acer.edu.au/mailman/options/rasch/markm%40eddata.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20120306/574734c7/attachment.html 


More information about the Rasch mailing list