[Rasch] DIF size and significance

Rense renselange at earthlink.net
Tue Jul 8 12:21:13 EST 2008


Usually, the effects are statistically significant well before this matters.
Because, even when DIF in some item is both significant and large, the net
effect thereof on the person measures is generally far less than one might
think. Just plot the raw-to-rasch tables for the subgroups separately and
you will see that DIF effects of 0.25 Logits are totally obliterated by the
other items, especially in longer tests. 
 
To experiment, one might generate simulated raw-to-rasch tables via Winsteps
by anchoring items differently and treating the results as if coming from
"different groups".
 
Rense Lange

  _____  

From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf
Of Mike Linacre (RMT)
Sent: Monday, July 07, 2008 9:02 PM
To: rasch at acer.edu.au
Subject: [Rasch] DIF size and significance


Joe,

There are two concerns with DIF:
1. That is big enough to make a substantive impact on the measures.
2. That it is statistically improbably enough ("significant enough") that it
is not likely merely to be an accident of the natural randomness in the
current dataset.

So, the ETS criteria are "both ... and ..."

Yes, you always want to investigate the reason for DIF because it may
indicate a flaw in test construction or scoring.

If the DIF effect is roughly uniform within the focus group, then changing
the sample size will not alter the DIF size, but will alter its significance
predictably. 

OK?

Mike L.

At 7/7/2008, you wrote:


My question is this: are the ETS size/significance criteria 'either-or'? I'm
finding many items exhibiting DIF where the DIF contrast is small in
absolute size (0.25 logits or less) but nonetheless statistically
significant at the 0.01 level. 
 
I'm not sure whether I should...
 
-simply use a cut-off point based on size, such as ETS's 0.43 'moderate DIF'
standard
-investigate the causes of the high DIF significance/low DIF size items
-randomly select a smaller sample of, say 1000 students per test, and see if
the items still show DIF.
 
You can see the ETS criteria at http://www.rasch.org/rmt/rmt32a.htm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20080707/538868ef/attachment.html 


More information about the Rasch mailing list