[Rasch] underfitting items and response rate
sgracia at ric.edu
Tue Sep 18 06:33:00 EST 2007
I administered an online survey last year as a pre-test for an educational intervention. It's almost time for me to administer the post-test. The survey consists of 11 scales that I want to analyze using Rasch methods. Ultimately, I want to look at any change that took place over the past year on each of the 11 variables.
Before administering the post-test, I have looked at the fit statistics for the items in the various scales. I would like to eliminate any items that do not fit the model and are not part of the variables I'm interested in. There's also some appeal to me in shortening the survey and reducing burden on the respondents.
I have approximately 150 respondents to the presurvey. On each of the scales, there's no problem with underfitting items. Some scales, however, do have overfitting items with very low mean squares (e.g., ,38, .02, etc.). Looking more closely, I see that those items are the ones that most people skipped. They might have 4 responses. So, they obviously did not "fit" in some way from respondents' points of view.
I could eliminate those items now. However, I suspect that some of these items actually do belong on the variable(s) but they represented situations that respondents had never encountered or did not understand. After a year in the program, I think their responses might be different.
So, I am torn between reducing the number of items based on this preliminary items (and probably increasing my response rate because the survey will be shorter) versus leaving the survey as is. If I leave the survey as is and the same items don't fit the second time, I will eliminate them from any analyses of change. If the items fit the second time around, I will learn more about the variables I'm interested in, but I realize that I won't be able to speak much about change in respondents on items since almost noone responded to them the first time.
How would you recommend that I proceed? Any assistance would be most appreciated.
Susan Gracia, PhD
Associate Professor, Educational Leadership Program
Dept. of Counseling, Educational Leadership, and School Psychology
Director of Assessment, Feinstein School of Education and Human Development
Rhode Island College
600 Mount Pleasant Avenue
Providence, RI 02908-1991
email: sgracia at ric.edu
More information about the Rasch