[Rasch] Units of measurement in the social sciences

Andrew Kyngdon akyngdon at lexile.com
Wed Oct 24 07:26:32 EST 2007


Paul,

 

I don't consider that you are getting at me or MetaMetrics. Heaven help
us if critical thinking in quantitative psychology gets to the point
where it is universally construed as personal attack.

 

Two points:

 

1.	I should have said that the observed comprehension rate is a
percent correct averaged over individuals, not the percent correct of
each individual. This will probably dampen your enthusiasm for our
results. Of course, individual percentages vary - no doubt due to the
poorly understood workings of the causal systems that you allude to
(plus other sundry extraneous variables).
2.	The Lexile scale measure is not used to predict the forecasted
comprehension rate, but the variable (in logits) of which the Lexile
scale is an affine transform. As you implied, if the Lexile scale
measure was used, then the rounding would introduce significant error.

 

The data collected so far from this ongoing study is as follows (sorry I
could not get the unrounded percentages, but my colleague responsible
for the application has gone home for the day):

 

Sample size: 1145

Total articles read: 16,124

Total items: 222,416

Observed percentage correct: 79%

Expected percentage correct: 79%

 

You lost me a bit in the last couple of paragraphs, but I got the gist
of what you are saying. Luce (1995) touched on it in his "Four tensions"
paper. He remarked that one of these tensions was "Phenomenological
versus process modelling: Unopened and opened black boxes". Luce
discussed this tension at length, with most mathematical models, both
from psychometrics and mathematical psychology, belonging to the former
and biological investigations belonging to the latter. His conclusion, I
think, is quite appropriate:

 

"In physics and applied physics they [phenomenological and process
models] continue to co-exist, even when a detailed model at one level
accounts, at least in principle, for the properties at a higher level.
One does not predict the paths of space probes using particle physics.
My guess is that any successful phenomenological model will always be
seen as an explanatory challenge to process modellers, but that the
latter will rarely supplant the former in all applications" (p.23) 

 

That is, if one has a good phenomenological theory of human behaviour
(e.g. the Lexile theory of prose text comprehension), even if the
neuro-biological systems underpinning reading are fully mapped out and
understood, the latter does not necessarily invalidate the former.
Indeed, if the former makes replicable, accurate predictions of text
comprehension, then it will be a challenge for process modellers to
explain why this is so. As Luce implied, physics seems to live with the
problem well enough.

 

Best regards,

 

Andrew

 

Ref.:

 

Luce, R.D. (1995). Four tensions concerning mathematical modelling in
psychology. Annual Review of Psychology, 46, 1-26.

 

 

________________________________

From: Paul Barrett [mailto:pbarrett at hoganassessments.com] 
Sent: Tuesday, October 23, 2007 3:39 PM
To: Andrew Kyngdon; rasch at acer.edu.au
Subject: RE: [Rasch] Units of measurement in the social sciences

 

 

	
________________________________


	From: Andrew Kyngdon [mailto:akyngdon at lexile.com] 
	Sent: Wednesday, October 24, 2007 3:21 AM
	To: Paul Barrett; rasch at acer.edu.au
	Subject: RE: [Rasch] Units of measurement in the social sciences

	Hi Paul,

	 

	You've raised an important point of which much attention has
been paid at MetaMetrics. There are a few issues here.

	 

	......

	On a related issue, we recently obtained data using an
instructional software application based on the Lexile Framework for
Reading. We found that forecasted text comprehension rates (as
percentages of correct responses) matched observed comprehension rates
to 3 decimal places.

	 

	I hope this helps. Feel free to ask more questions.

	 

Thanks Andrew - seriously impressive data; I have some reading to do!

 

One point that occurs to me though, if the error around the lexile
estimate for an individual is as you related, how is it possible to
report estimated vs actual comprehension rates correct to 3 decimal
places? 

 

Assuming a lexile score is involved somewhere in the calculation of the
estimated correct response rate, wouldn't this introduce greater than
.0005 error in the rate (assuming a rate is reported between 0 and 1 -
which can be multiplied up to a %, but I'm assuming you don't mean 3
decimal places on an already corrected percentage, as this would imply a
standard or accuracy to 5 decimal places)?

 

Presumably this has something to do with how precisely the initial
lexile score was estimated for each individual?

 

How many individuals were used by the way - and did they vary in initial
lexile score (across a reasonable range of lexile scores?).

 

I'm curious because if we think for one moment about how the brain works
(in terms of networks of dynamical system "wetware" biology), the kinds
of precision you seem to be asserting don't make much sense except for
some sensory transducer systems (auditory spatial frequency resolution
etc.). Likewise any proposal that a standard additive-metric unit could
be invoked for any "psychological" variable ("Comprehension" being just
such a variable).

 

But "seem to make sense" is a phrase which can be simply destroyed by
empirical evidence, which is why I find this issue fascinating.

 

I don't mean to be negative here, and I don't want anyone reading this
thinking I'm "getting at" Andrew or Metametrics. It's just my current
thinking and perusal of neuroscience and evolutionary biology has caused
me to doubt strongly that the "behavioral - cognitive" output of a human
brain can ever be "measured" using linear unit-preserving metrics. The
system responsible for these cognitions/behaviors is complex, in the
technical sense of that word. So, we can approximate as a linear
function overlaid on an ordered metric, but quantitative measurement
within such a system would seem impossible as the "causal-biology" of
the system would be unable to sustain a standard unit to the precision
required by the "additivity" property.

 

Hence, I am fascinated by data which seems to demonstrate the
quantitative, near-deterministic (vs stochastic), measurement of a
psychological variable - which is why the validation experiments are so
critical to me personally!

 

Regards ... Paul

 

 

 

 

Paul Barrett, Ph.D.

2622 East 21st Street | Tulsa, OK  74114

Chief Research Scientist

Office | 918.749.0632  Fax | 918.749.0635

pbarrett at hoganassessments.com

      

hoganassessments.com <http://www.hoganassessments.com/> 

 

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071023/c1df3643/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3176 bytes
Desc: image002.jpg
Url : https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071023/c1df3643/attachment.jpe 


More information about the Rasch mailing list