[Rasch] Noise in observations ...

Lang, William Steve WSLang at tempest.coedu.usf.edu
Mon Nov 5 15:09:50 EST 2007


Paul:

It is most interesting to me that I have found Rasch provides an excellent model of affective responses (including apperception measures) such as teacher dispositions and complex behaviors, multiple observations, and focus group scoring.  I suspect, since I don't know exactly what your applications are, that the key to my finding Rasch more useful would fall in this paragraph of your post:

"My own "research" work and thinking nowadays is exactly that, seeking
optimal ways of scoring "assessments" (whether questionanire items,
scales, behavioral indicators/observations, or whatever, in terms of
maximizing cross-validated predictive validity - and establishing a set
of clear empirical relations first between "variables" - before
attempting (if at all) to impose a more formal data model upon them."

Establishing VALIDITY has also been my primary concern.  I've found Rasch developed scales worked exceptionally well in circumstances similar to your description.  I'm not sure if email is the way to get into details and we've already been referring to dissertation length documents.  Regardless, I can't imagine any stronger arguments for using Rasch than researching validity. My "optimal ways" acknowledge issues of growth, rater errors, and systematic detection of bias.  Rasch is robust for missing data, works well with "acceptable" connectivity, and is sensitive to finding the issues of interest in exactly the type of assessments you describe.  This would include threshold analysis and disordered categories, estimates of the effects of interventions on measure changes, and all kinds of anchoring/equating options. My Rasch constructed instruments target validity (for example, at assessing affect in alternative assessment teacher candidates) and useful aggregation in a messy world.

I admit this is not John Exner, but I probably would differ from you as I usually spend inordinate time visualizing and mapping (imposing a formal model?) on the construct of interest.  I doubt there is a convergence of current classical analysis with the set of Rasch tools except under specific circumstances such as the ACT described in the earlier dissertation.  

If Rasch "mathematics" are not convincing, perhaps a serious attempt at your most pressing issues in the paragraph cited would empirically demonstrate if Rasch worked as part of a specific context.   If you attempted a "formal model" of the construct, applied the Rasch model, and looked at fit analyses; I would be surprised if the validation research wasn't meaningful and practical for complex assessments and aggregations.  I'll include some links below (not specifically aimed at explaining Rasch, but as examples that incorporate Rasch), because I think they target very similar applications to those you describe and are intended to be user friendly in case the economics of conveying the results to others is of interest.

Steve Lang,
University of South Florida St. Petersburg

(This is not meant as an advertisement; as Trevor points out, none of us are going to be confused with Steven King when the Amazon ranking hovers around one million!)

http://www.corwinpress.com/booksProdDesc.nav?prodId=Book229866
http://www.corwinpress.com/booksProdDesc.nav?prodId=Book230703
 



-----Original Message-----
From: rasch-bounces at acer.edu.au on behalf of Paul Barrett
Sent: Sun 11/4/2007 4:24 PM
To: rasch at acer.edu.au
Subject: RE: [Rasch] Noise in observations ...
 



________________________________

	From: Denny Borsboom [mailto:d.borsboom at uva.nl] 
	Sent: Monday, November 05, 2007 9:28 AM
	To: Paul Barrett
	Subject: Re: [Rasch] Noise in observations ...
	
	
	Hi Paul, 
	Thanks for your interesting post. I'm wondering why you would
use sumscores and not, say, product scores or any other of the millions
of alternative ways of aggregating items? 
	Best
	Denny

Ha! 
 
Hello Denny - funny you should ask that .. 
 
The answer has to be that intuitively, a simple sumscore is the most
basic way of representing something as crude as "more of this = more or
less of that" - with no more thought than this applied to such an
approach. It really is a kind of "basic" thinking that I'm sure is so
intuitive as it requires nothing more than simple addition of item
responses. 
 
The key here is how you treat that "summation". i.e. whether you
continue to rely upon the additivity assumption and so end up with CTT
and and a whole host of "data-model-presumptive" indices - or keep
remembering that really these magnitudes might just as well be crude
orders and so use them as ordered classes and seek evidence for what
these classes might indicate in terms of some expected set of outcomes.
 
My own "research" work and thinking nowadays is exactly that, seeking
optimal ways of scoring "assessments" (whether questionanire items,
scales, behavioral indicators/observations, or whatever, in terms of
maximizing cross-validated predictive validity - and establishing a set
of clear empirical relations first between "variables" - before
attempting (if at all) to impose a more formal data model upon them. 
 
Indeed, I have said in several presentations now that I think the next
big advances in psychology will be in how we combine "variables" to
predict outcomes, rather than concerning ourselves too much with the
task of trying to create ever more precise measurement of the variables
(because I've also adopted a view that psychology is more likely a
non-quantitative science).
 
I know, this all looks very sloppy and "informal" against the precision
and elegance of Rasch and Latent Variable modeling in general, but I
really do think these methods require assumptions about the proposed
causality of such variables (i.e the biological/cognitive systems which
are proposed as maintaining the desired response precision in terms of
measured "equal-interval" magnitudes) which are not very plausible given
the current knowledge we have about neuroscience and cognitions (such as
Gigerenzer's work etc.) 
 
Of course, whether such arguments or reasoning apply to the area of
"educational" and "medico-diagnostic" variables to which many would
apply a Rasch model is a moot point.
 
And, I suffer doubts every day about this ... as to move from a strong
quantitative "imperative" to one that says "it's all a bit fuzzy" but
that's how we humans actually are so deal with it accordingly - is a
huge step back in many onlooker's eyes (althouh I have said this is a
paradox in that one's predictions might well increase in accuracy 
given that the criterion itself is reduced to a real-world degree of
precision). But this is another issue; I only mention it here to provide
some little background as to why I'm less convinced than some by "test
theory", "Rasch, and psychometrics in general. I may be quite wrong.
 
If anybody is interested in my latest presentation papers surrounding
this issue - whizz over http://www.pbarrett.net/NZ_Psych_2007.htm ...
and take a quick look ...
but I'm not sure if these really are relevant to the Rasch community -
as these do not address edumetrics or educational issues per se.
 
Regards .. Paul
 
 
 

 

Paul Barrett, Ph.D.

2622 East 21st Street | Tulsa, OK  74114

Chief Research Scientist

Office | 918.749.0632  Fax | 918.749.0635

pbarrett at hoganassessments.com

      

hoganassessments.com <http://www.hoganassessments.com/> 

 

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071104/9853e506/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: attfa128.bmp
Type: image/bmp
Size: 29886 bytes
Desc: attfa128.bmp
Url : https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071104/9853e506/attachment.bmp 


More information about the Rasch mailing list