[Rasch] Question about poor reliability
Gustaf Bernhard Uno Skar
gustaf.b.skar at ntnu.no
Thu Dec 26 04:29:35 AEDT 2019
Dear Rasch Community,
I turn to you with a reliability question. I have conducted a MFRM-analysis (in the FACETS software), using the rating scale model. I had three facets: students, raters, and dimensions of text quality (there were eight dimensions). Each dimension was rated by at least two raters on five-point scales. The separation reliability for students was good: .94.
In a subsequent analysis, I pulled one dimension out to create two measures. The first (call it a limited writing proficiency measure) still showed high enough separation reliability (.94), but the second (spelling) indicated too low reliability (0.00 according to the Software output). Now here’s the question: what should I do with the spelling measure? To be sure, there is a lot of variation in the data (some kids are excellent spellers, some not so much).
Using the raw scores is one option, using the logit values another. Using raw scores will have some unwanted consequences (such as having a measure limited to the steps 1, 1.5, 2, 2.5 etc). Using logits, however, may not seem justifiable because the analysis was not able to reliably separate the students.
Any food for thoughts or hints about relevant papers/readings will be much appreciated.
Gustaf B. Skar, Ph.D.
Norwegian University of Science and Technology (NTNU)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Rasch