[Rasch] Models of unidimensional constructs have their limitations

Rod O'Connor rodocon at rodoconnorassoc.com
Tue Nov 6 14:18:02 EST 2007


Hello Rasch Community

/Re. "I think the next big advances in psychology will be in how we 
combine "variables" to predict outcomes, rather than concerning 
ourselves too much with the task of trying to create ever more precise 
measurement of the variables' - Paul Barrett 5/11/2007/

As another consultant who is concerned to develop predictive measures - 
in my case measures to assess constructs such as 'health-related quality 
of life", "medication compliance", and behavioral/demographic bundles 
predictive of future ill-health (eg. 'likelihood of developing 
diabetes'), I have considerable support for aspects of Paul's views. 

In my area and experience failures with complex health-state measures 
are not so much due to inelegance in modelling uni-dimensional states, 
it has been an unwillingness to spend the detailed investigative time 
necessary to understand the variables/factors that impinge on the 
complex construct in the first place (such as "medication compliance of 
young people with asthma").   Finding/developing/refining a set of items 
that are nicely inter-related or ordered tends not to be the primary 
issue - it is identifying the variables that are relevant to predicting 
self-assessed or observed health states. 

Once a likely set of such 'variables' are identified the next most 
important job is to work out how they combine.  Work by such as 
Redelmeier and Kahneman (1996) have demonstrated that perceived health 
states or experiences are not simply summed experience but some 
unexpected algorithm, such as the 'peak-end' rule where an experienced 
health-state is judged based on the /peak/ worst-experience _plus_ how 
the experience /ends/ ('alls well that ends well").  I have found  such 
'discounting' functions myself, in assessing the impact of multiple 
disabilities (O'Connor 2004; as did Marilyn Bergner in her work on the 
Sickness Impact Profile: Bergner et al, 1976).   Fitting a set of items 
into a uni-dimensional construct seems not to be the answer, it is 
combining such factors "as people do" that is needed to predict the 
value of the health construct as assessed by patients/.  /Tversky and 
Kahneman, Klein, Bargh, and others, have offered evidence that 
predicting judgements and preferences may need to violate a simple, 
linear, 'conventionally rational' model.
/
/I admire greatly the elegance of the Rasch approach (it is conceptually 
delightful), and advocate its use for modelling uni-dimensional states 
in workshops I run (that is, if participants have the resources to go 
beyond sum-scales), but I feel it is generally only part of the answer, 
and often the lesser part.  In reading posts to this very interesting 
group I have the impression that contributors sometimes forget that 
being able to model a set of items with a unidimensional function may 
not be sufficient.  Ensuring the right content in terms of the 
criterion, and then identifying how to combine the content to predict 
what is desired, may be the essentials (although it is an iterative 
process).  Criterion validity is the most important aspect in my area 
too.  Or is it just that unidimensional constructs are what is found in 
education, and in education people already know the content area, unlike 
healthcare?  /
/
Warm regards,
Rod

Bergner M, Bobbitt R, Kressel S, et al The sickness impact profile: 
conceptual foundations
and methodology for the development of a health status measure. 
International Journal of
Health Services 1976, Vol. 6, pp.393-415.

O'Connor R.  Measuring Quality of Life in Health. Elsevier/ Churchill 
Livingstone  2004.

Redelmeier DA, Kahneman D. Patients' memories of painful medical 
treatments: real-time
and retrospective evaluations of two minimally invasive procedures. 
Pain, 66 (1996) 3-8

______________________________________

Rod O'Connor PhD
Director Rod O'Connor & Associates Pty Ltd,
Consultants in Health Care Measurement and Improvement.
Conjoint Associate Professor, School of Public Health
and Community Medicine, Faculty of Medicine,
University of New South Wales, Sydney, Australia.
ph. +61-2 9555 9916  mob. 0413 60 70 73
Email: rod at RodOConnorAssoc.com
Web site: www.RodOConnorAssoc.com
_______________________________________

    ------------------------------------------------------------------------
    *From:* Denny Borsboom [mailto:d.borsboom at uva.nl]
    *Sent:* Monday, November 05, 2007 9:28 AM
    *To:* Paul Barrett
    *Subject:* Re: [Rasch] Noise in observations ...

    Hi Paul,
    Thanks for your interesting post. I'm wondering why you would use
    sumscores and not, say, product scores or any other of the millions
    of alternative ways of aggregating items? 
    Best
    Denny

Ha!
 
Hello Denny - funny you should ask that ..
 
The answer has to be that intuitively, a simple sumscore is the most 
basic way of representing something as crude as "more of this = more or 
less of that" - with no more thought than this applied to such an 
approach. It really is a kind of "basic" thinking that I'm sure is so 
intuitive as it requires nothing more than simple addition of item 
responses.
 
The key here is how you treat that "summation". i.e. whether you 
continue to rely upon the additivity assumption and so end up with CTT 
and and a whole host of "data-model-presumptive" indices - or keep 
remembering that really these magnitudes might just as well be crude 
orders and so use them as ordered classes and seek evidence for what 
these classes might indicate in terms of some expected set of outcomes.
 
My own "research" work and thinking nowadays is exactly that, seeking 
optimal ways of scoring "assessments" (whether questionanire items, 
scales, behavioral indicators/observations, or whatever, in terms of 
maximizing cross-validated predictive validity - and establishing a set 
of clear empirical relations first between "variables" - before 
attempting (if at all) to impose a more formal data model upon them.
 
Indeed, I have said in several presentations now that I think the next 
big advances in psychology will be in how we combine "variables" to 
predict outcomes, rather than concerning ourselves too much with the 
task of trying to create ever more precise measurement of the variables 
(because I've also adopted a view that psychology is more likely a 
non-quantitative science).
 
I know, this all looks very sloppy and "informal" against the precision 
and elegance of Rasch and Latent Variable modeling in general, but I 
really do think these methods require assumptions about the proposed 
causality of such variables (i.e the biological/cognitive systems which 
are proposed as maintaining the desired response precision in terms of 
measured "equal-interval" magnitudes) which are not very plausible given 
the current knowledge we have about neuroscience and cognitions (such as 
Gigerenzer's work etc.)
 
Of course, whether such arguments or reasoning apply to the area of 
"educational" and "medico-diagnostic" variables to which many would 
apply a Rasch model is a moot point.
 
And, I suffer doubts every day about this ... as to move from a strong 
quantitative "imperative" to one that says "it's all a bit fuzzy" but 
that's how we humans actually are so deal with it accordingly - is a 
huge step back in many onlooker's eyes (althouh I have said this is a 
paradox in that one's predictions might well increase in accuracy
given that the criterion itself is reduced to a real-world degree of 
precision). But this is another issue; I only mention it here to provide 
some little background as to why I'm less convinced than some by "test 
theory", "Rasch, and psychometrics in general. I may be quite wrong.
 
If anybody is interested in my latest presentation papers surrounding 
this issue - whizz over http://www.pbarrett.net/NZ_Psych_2007.htm ... 
and take a quick look ...
but I'm not sure if these really are relevant to the Rasch community - 
as these do not address edumetrics or educational issues per se.
 
Regards .. Paul
 
 

	

 

*Paul Barrett, Ph.D.*

	

*2622 East 21st Street | Tulsa, OK  74114*

*Chief Research Scientist*

	

*Office | 918.749.0632  Fax | 918.749.0635*

*pbarrett at hoganassessments.com*

	

*      *

*hoganassessments.com <http://www.hoganassessments.com/>*

	

 

 

 

 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071106/60622fc9/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/bmp
Size: 29886 bytes
Desc: not available
Url : https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071106/60622fc9/attachment.bmp 


More information about the Rasch mailing list