[Rasch] Empirical order & Theoretical order
mschulz at pacificmetrics.com
Tue Nov 2 00:47:32 EST 2010
If the empirical difficulty order of items does not corresponde with the theoretical difficulty in terms of coginitive demand of the items, what can we conclude about the test?
Does this invalidate the additivity of the scores, even if we have perfect fit?
I think both Tom Conner and Augustin Tristan provided very good answers to this question. Another idea you should consider is that too much importance is being placed on individual test items. Your typical test item is no more reliable as a representative of a developmental stage in some theory than it is as a measure of a student. Guttman scales are the exception, not the rule in psychometrics. In most real world applications where items are theoretically ordered in difficulty, you need many items to represent a particular stage of difficulty just as you need many to measure where a student is on the measurement scale. We need to treat levels of item difficulty in our theories as "domains" and recognize that item membership in these domains is not deterministic and exact--it is statistical, somewhat like the stochastic item response. So when we classify items into theoretically ordered domains of difficulty, there will be overlap. Some items from a domain of theoretically higher difficulty will be easier than items from a domain of theoretically lower difficulty. What we *should* see is that our expected order of difficulty is confirmed on average, at the domain level. Our attention to individual test items should be focused on outliers in the distributions of item difficulty by domain.
More information about the Rasch