[Rasch] DIF size and significance

Mike Linacre (RMT) rmt at rasch.org
Tue Jul 8 12:01:45 EST 2008


There are two concerns with DIF:
1. That is big enough to make a substantive impact on the measures.
2. That it is statistically improbably enough ("significant enough") that 
it is not likely merely to be an accident of the natural randomness in the 
current dataset.

So, the ETS criteria are "both ... and ..."

Yes, you always want to investigate the reason for DIF because it may 
indicate a flaw in test construction or scoring.

If the DIF effect is roughly uniform within the focus group, then changing 
the sample size will not alter the DIF size, but will alter its 
significance predictably.


Mike L.

At 7/7/2008, you wrote:
>My question is this: are the ETS size/significance criteria 'either-or'? 
>I'm finding many items exhibiting DIF where the DIF contrast is small in 
>absolute size (0.25 logits or less) but nonetheless statistically 
>significant at the 0.01 level.
>I'm not sure whether I should...
>-simply use a cut-off point based on size, such as ETS's 0.43 'moderate 
>DIF' standard
>-investigate the causes of the high DIF significance/low DIF size items
>-randomly select a smaller sample of, say 1000 students per test, and see 
>if the items still show DIF.
>You can see the ETS criteria at 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20080707/a4d5d605/attachment.html 

More information about the Rasch mailing list