[Rasch] Rasch Analysis on Vignette Instruments

Dare Institute dareinstitute at gmail.com
Fri Aug 12 02:37:16 EST 2011


Hello,


The Dare Institute has been running instruments that involve the rating of
vignettes, and we are uncertain as to whether Rasch analysis is appropriate
for this situation. Here is an outline of the instruments:


In each instrument, five or more vignettes are presented. Each vignette
presents different people in very similar situations acting in similar ways,
but in ways deemed to be at different developmental levels of hierarchical
complexity. For instance, one vignette will show someone solving a problem
in a way indicative of a very high stage of development, another will show
someone solving the same problem in a way deemed less hierarchically
complex, etc. The participant is then asked a number of questions regarding
the vignettes, such as "Rate this person's method of solving this problem."
The answers to these items are always on a six-point scale, with 6
indicating the highest possible rating and 1 indicating the lowest possible
rating.

Our goal in analyzing these vignettes is to determine the stage of the
participant (i.e. at what developmental stage he/she is, as determined by
the model of hierarchical complexity). We assume that the participant will
rate highly the vignette that corresponds to the stage he/she is at, and
that the participant will give lower ratings to the other vignettes. We have
been coding the responses into Winsteps to run Rasch analysis.

The problems we are considering are as follows:

1) The "best" a person could do on the instrument is giving a 6 to the
highest stage vignette and then progressively lower ratings for the lower
stage vignettes. However, Rasch generally determines that the "best" someone
can do is give 6's to every question, a response that we would deem
inconclusive (since it indicates that the person corresponds to every
stage).

2) The questions do not measure developmental stage independently of one
another. In this regard, these vignette instruments do not parallel a
standard test where Rasch analysis would be used. On a standard instrument,
how a person responds to a single question (i.e. a right/wrong answer) tells
us something clear about their ability or about whatever is being measured.
For instance, if one person answers a math question correctly and another
person answers it incorrectly, we would assume - albeit with high
uncertainty - that the former person was more mathematically proficient than
the latter. This method does not work on the vignette instruments. If one
person answers highly on one vignette and another person answers low, we
cannot say anything about their relative stages without the other vignette
questions, for many reasons (suppose the person who answered low answered
high on a higher stage vignette, etc.).

3) Although the vignettes do not rely on a single corpus of text - and in
this regard they are independent - how we analyze them seems in ways to be
interdependent. One response to a question is virtually meaningless without
other answers. Having more questions is important for more than reducing
uncertainty.

We are struggling to decide if Rasch analysis is correct for these data.  We
would really appreciate any feedback you could give.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20110811/b5b8a220/attachment.html 


More information about the Rasch mailing list