[Rasch] Models of unidimensional constructs have their limitations
rodocon at rodoconnorassoc.com
Tue Nov 6 14:18:02 EST 2007
Hello Rasch Community
/Re. "I think the next big advances in psychology will be in how we
combine "variables" to predict outcomes, rather than concerning
ourselves too much with the task of trying to create ever more precise
measurement of the variables' - Paul Barrett 5/11/2007/
As another consultant who is concerned to develop predictive measures -
in my case measures to assess constructs such as 'health-related quality
of life", "medication compliance", and behavioral/demographic bundles
predictive of future ill-health (eg. 'likelihood of developing
diabetes'), I have considerable support for aspects of Paul's views.
In my area and experience failures with complex health-state measures
are not so much due to inelegance in modelling uni-dimensional states,
it has been an unwillingness to spend the detailed investigative time
necessary to understand the variables/factors that impinge on the
complex construct in the first place (such as "medication compliance of
young people with asthma"). Finding/developing/refining a set of items
that are nicely inter-related or ordered tends not to be the primary
issue - it is identifying the variables that are relevant to predicting
self-assessed or observed health states.
Once a likely set of such 'variables' are identified the next most
important job is to work out how they combine. Work by such as
Redelmeier and Kahneman (1996) have demonstrated that perceived health
states or experiences are not simply summed experience but some
unexpected algorithm, such as the 'peak-end' rule where an experienced
health-state is judged based on the /peak/ worst-experience _plus_ how
the experience /ends/ ('alls well that ends well"). I have found such
'discounting' functions myself, in assessing the impact of multiple
disabilities (O'Connor 2004; as did Marilyn Bergner in her work on the
Sickness Impact Profile: Bergner et al, 1976). Fitting a set of items
into a uni-dimensional construct seems not to be the answer, it is
combining such factors "as people do" that is needed to predict the
value of the health construct as assessed by patients/. /Tversky and
Kahneman, Klein, Bargh, and others, have offered evidence that
predicting judgements and preferences may need to violate a simple,
linear, 'conventionally rational' model.
/I admire greatly the elegance of the Rasch approach (it is conceptually
delightful), and advocate its use for modelling uni-dimensional states
in workshops I run (that is, if participants have the resources to go
beyond sum-scales), but I feel it is generally only part of the answer,
and often the lesser part. In reading posts to this very interesting
group I have the impression that contributors sometimes forget that
being able to model a set of items with a unidimensional function may
not be sufficient. Ensuring the right content in terms of the
criterion, and then identifying how to combine the content to predict
what is desired, may be the essentials (although it is an iterative
process). Criterion validity is the most important aspect in my area
too. Or is it just that unidimensional constructs are what is found in
education, and in education people already know the content area, unlike
Bergner M, Bobbitt R, Kressel S, et al The sickness impact profile:
and methodology for the development of a health status measure.
International Journal of
Health Services 1976, Vol. 6, pp.393-415.
O'Connor R. Measuring Quality of Life in Health. Elsevier/ Churchill
Redelmeier DA, Kahneman D. Patients' memories of painful medical
and retrospective evaluations of two minimally invasive procedures.
Pain, 66 (1996) 3-8
Rod O'Connor PhD
Director Rod O'Connor & Associates Pty Ltd,
Consultants in Health Care Measurement and Improvement.
Conjoint Associate Professor, School of Public Health
and Community Medicine, Faculty of Medicine,
University of New South Wales, Sydney, Australia.
ph. +61-2 9555 9916 mob. 0413 60 70 73
Email: rod at RodOConnorAssoc.com
Web site: www.RodOConnorAssoc.com
*From:* Denny Borsboom [mailto:d.borsboom at uva.nl]
*Sent:* Monday, November 05, 2007 9:28 AM
*To:* Paul Barrett
*Subject:* Re: [Rasch] Noise in observations ...
Thanks for your interesting post. I'm wondering why you would use
sumscores and not, say, product scores or any other of the millions
of alternative ways of aggregating items?
Hello Denny - funny you should ask that ..
The answer has to be that intuitively, a simple sumscore is the most
basic way of representing something as crude as "more of this = more or
less of that" - with no more thought than this applied to such an
approach. It really is a kind of "basic" thinking that I'm sure is so
intuitive as it requires nothing more than simple addition of item
The key here is how you treat that "summation". i.e. whether you
continue to rely upon the additivity assumption and so end up with CTT
and and a whole host of "data-model-presumptive" indices - or keep
remembering that really these magnitudes might just as well be crude
orders and so use them as ordered classes and seek evidence for what
these classes might indicate in terms of some expected set of outcomes.
My own "research" work and thinking nowadays is exactly that, seeking
optimal ways of scoring "assessments" (whether questionanire items,
scales, behavioral indicators/observations, or whatever, in terms of
maximizing cross-validated predictive validity - and establishing a set
of clear empirical relations first between "variables" - before
attempting (if at all) to impose a more formal data model upon them.
Indeed, I have said in several presentations now that I think the next
big advances in psychology will be in how we combine "variables" to
predict outcomes, rather than concerning ourselves too much with the
task of trying to create ever more precise measurement of the variables
(because I've also adopted a view that psychology is more likely a
I know, this all looks very sloppy and "informal" against the precision
and elegance of Rasch and Latent Variable modeling in general, but I
really do think these methods require assumptions about the proposed
causality of such variables (i.e the biological/cognitive systems which
are proposed as maintaining the desired response precision in terms of
measured "equal-interval" magnitudes) which are not very plausible given
the current knowledge we have about neuroscience and cognitions (such as
Gigerenzer's work etc.)
Of course, whether such arguments or reasoning apply to the area of
"educational" and "medico-diagnostic" variables to which many would
apply a Rasch model is a moot point.
And, I suffer doubts every day about this ... as to move from a strong
quantitative "imperative" to one that says "it's all a bit fuzzy" but
that's how we humans actually are so deal with it accordingly - is a
huge step back in many onlooker's eyes (althouh I have said this is a
paradox in that one's predictions might well increase in accuracy
given that the criterion itself is reduced to a real-world degree of
precision). But this is another issue; I only mention it here to provide
some little background as to why I'm less convinced than some by "test
theory", "Rasch, and psychometrics in general. I may be quite wrong.
If anybody is interested in my latest presentation papers surrounding
this issue - whizz over http://www.pbarrett.net/NZ_Psych_2007.htm ...
and take a quick look ...
but I'm not sure if these really are relevant to the Rasch community -
as these do not address edumetrics or educational issues per se.
Regards .. Paul
*Paul Barrett, Ph.D.*
*2622 East 21st Street | Tulsa, OK 74114*
*Chief Research Scientist*
*Office | 918.749.0632 Fax | 918.749.0635*
*pbarrett at hoganassessments.com*
-------------- next part --------------
An HTML attachment was scrubbed...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 29886 bytes
Desc: not available
Url : https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071106/60622fc9/attachment.bmp
More information about the Rasch