[Rasch] Negative pt-bis and fit of 1.0? How can this be?

Mark Moulton markhmoulton at gmail.com
Wed Mar 7 05:35:34 EST 2012


Hi Stuart!

Outfit = mean(z[ni]^2)

where z[ni] is the raw residual divided by the variance

z[ni] = res[ni] / var[ni]

High misfit is therefore caused by either:

   1. large residuals (or a large number of large residuals)
   2. small cell variances

The size of the cell variance is driven primarily by poor targeting (and, I
believe, misfit in the case of "real RMSE" (Mike?)).  So the more out of
range the items are, the higher the cell variances.

It sounds like your sample is badly off-target due to final exam items
being on a pretest.  Therefore, even though there are a lot of large
residuals (causing low point-biserials), the cell variances are equally
large, and the result would be misfit around 1.0.  Large misfits require
small errors.

On a more general note, I recently did an analysis like yours where
end-of-course items were given as a pretest, and the results were
systematically awful.  Kids are basically faced with items they can't
engage with, so no measurement is actually occurring.  It would be cheaper
to use a random number generator.

Mark Moulton
Educational Data Systems



On Tue, Mar 6, 2012 at 7:58 AM, Stuart Luppescu <slu at ccsr.uchicago.edu>wrote:

> Hello, I'm analyzing items suggested for a course final exam. The
> problem is that the students the items are tested on have not taken the
> course yet. This results in very poor performance. The average person
> measure is -0.78, and the average item p-value is 0.33 (for 4-choice
> multiple choice items).
>
> What is confusing me is that all the items have mean-square fit
> statistics (infit and outfit) near 1.0, while many of them have negative
> point-biserial correlations. According to my understanding, the fit
> statistics are calculated from an aggregation of the squared
> standardized residuals, which are calculated from the raw residual
> divided by the score variance. In this case, the expectation of an
> individual response would be low, so raw residuals would be large. And
> the score variance at the extremes are lower than in the middle, so you
> divide a large raw residual by a small score variance and you get a very
> large standardized residual, right? So, how come the fits are close to
> expectation? Especially since the pt-bis are low or negative? I don't
> get it. Can someone explain this to me?
>
> Thanks.
> --
> Stuart Luppescu -=- slu .at. ccsr.uchicago.edu
> University of Chicago -=- CCSR
> 才文と智奈美の父 -=-    Kernel 3.2.1-gentoo-r2
> What we have is nice, but we need something very
>  different.    -- Robert Gentleman
>  Statistical Computing 2003, Reisensburg (June
>  2003)
>
>
>
>
>
>
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe:
> https://mailinglist.acer.edu.au/mailman/options/rasch/markm%40eddata.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20120306/a02abbe4/attachment.html 


More information about the Rasch mailing list