[Rasch] Rasch analysis of interval data

Andrew Kyngdon akyngdon at lexile.com
Wed Jul 23 01:13:29 EST 2008


 

This is still not a guarantee that the lexile unit captures the "true"
spacing, but fortunately it does not seem to matter.

 

The Lexile unit is best defined unit I have come across in the
behavioural sciences, so by what criteria one judges to be a "true
spacing" (whatever that is) seems to be a mystery...

 

Andrew Kyngdon, PhD

Senior Research Scientist

MetaMetrics, Inc.

1000 Park Forty Plaza Drive

Durham NC 27713 USA

Tel. 1 919 354 3473

Fax. 1 919 547 3401

<http://www.lexile.com/conference2008>  

________________________________

From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of Mark Moulton
Sent: Monday, July 21, 2008 6:09 PM
To: Paul Barrett
Cc: rasch at acer.edu.au
Subject: Re: [Rasch] Rasch analysis of interval data

 

Paul,

 

Thank you for your explanations and for your presentation 10 years ago,
which are very helpful to me.  You raise a fundamental issue, still
controversial, still worth visiting in my opinion. You make the case
that Rasch measures are equal-interval representations of counts (ratios
of counts), and that is all, and that they do not necessarily capture a
fundamental unit of measurement in the underlying construct.  

 

I think your point is amplified by a simple desktop experiment.  Lay a
ruler on a sheet of paper.  Draw dots (representing persons, say) in
various distributions on the paper.  For each centimeter increment on
the ruler (representing items), count the dots above and below that
increment and calculate their log ratio.  One finds that the logit
spacings of the ruler increments may be highly unequal (unlike the
centimeters), depending on how one distributes the dots.  If one
distributes the dots equally up and down the ruler, the logit lengths
between increments appear to get fatter at the extremes.  If one clumps
the dots in multiple modes, the logit lengths can be distorted in all
sorts of cool ways.  Interestingly, as the distribution approaches
normal, the logit lengths between increments seem to approach a linear
relationship with the centimeters, (I don't have a proof for why this
would be true, but presumably it has something to do with the
relationship between the logistic and normal distributions and may
account for the similarity between independently calibrated scales).

 

So, I agree that Rasch logits do not capture fundamental units of
measurement, and are sample-dependent in this sense (and in several
other senses, too).  My question is:  What does this do to Rasch claims
of "invariance," aka "special objectivity," the notion that the relative
logit spacings of persons will remain the same regardless of how the
items are spaced?  Strangely, I don't think it has any effect at all.
The disappearance of the item parameter when calculating the person
parameter, and vice versa, has the same force and implication that it
always did.  And due to how Rasch conjointly calculates persons and
items, whatever distortions may occur affect the persons and items
equally.

 

I am left with a relativistic notion of psychometric spaces.  Each Rasch
analysis erects a unique space.  That space bears no necessary
relationship to any other Rasch space (except perhaps in some
topographic one-to-one homeomorphic kind of way).  However, objects
within that space are distributed in a way that is reproducible, hence
objective, with respect to other objects in that space.  Two Rasch
spaces can be reconciled only by "anchoring" one space to the other via
common persons or items.  This forces the two spaces to share the same
"distortions," and thus to become one space, and to preserve invariance
for all objects residing in that space.

 

Your point about the MetaMetrics lexile scale is well-taken.  All texts
and readers are forced into a common space anchored on the physical
properties represented by word frequency and sentence length (or log
transformations thereof).  This was facilitated by the fact that
MetaMetrics discovered and exploited a linear relationship between
textual empirical variables and item difficulties.  But even without
that relationship, the two types of variables could have been forced (by
a method MetaMetrics did not use, or need to use) into a common space
through an anchoring procedure.  This is still not a guarantee that the
lexile unit captures the "true" spacing, but fortunately it does not
seem to matter.  

 

One space is as good as another, so long as they are internally
consistent.  Am I reading this right?

 

Mark H. Moulton

Educational Data Systems

 

 

 

 

 

 

2008/7/21 Paul Barrett <pbarrett at hoganassessments.com>:

From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of Anthony James
Sent: Wednesday, July 16, 2008 7:19 AM
To: rasch at acer.edu.au
Subject: [Rasch] Rasch analysis of interval data

 

Hi all,

Has anyone ever tried to Rasch analyse a variable for which there's
concatenation-based objective measurement? Suppose we make a height
scale with 6 points:

 

________________________________

From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of Andrew Kyngdon
Sent: Wednesday, July 16, 2008 8:13 AM


To: rasch at acer.edu.au

Subject: RE: [Rasch] Rasch analysis of interval data

I think Paul Barrett did something like this once...

 

________________________________

 

Yep - 10 years ago to be exact!

 

Sorry I haven't replied until now ... 

 

The presentation about the simulation can be downloaded at:

 

http://www.pbarrett.net/presentations/BPS-rasch_98.pdf

 

>From my web-page abstract ...

Beyond Psychometrics: the recovery of a standard unit of length: This
50-slide presentation was given at the British Psychological Society's
Division of Occupational Psychology conference: Assessment in the
Millennium: Beyond Psychometrics, November 1998,  at Birkbeck
(University of London). The theme of this presentation was about Rasch
scaling, and its capacity to construct a standard unit from
observational data. This presentation contained a data simulation that
attempted to hide a true quantitatively structured latent variable of
length behind some poor ordinal observations. All the Rasch scaling did
was to construct an equal-interval latent variable of ordinal lengths!
This simulation was heavily criticised Ben Wright and others, and I have
included these criticisms as an addendum to the presentation - along
with my reply. However, recent papers seem to have vindicated my
conclusions in some respects.....The reality is that these methods
simply construct linear latent variables in complete isolation of any
empirical evidence that such variables might indeed be quantitatively
structured.. In my opinion, from a scientific perspective, these scaling
methods are frankly of little utility, but they are ingenious from a
psychometric perspective and do have great utility in a more pragmatic
sense. It all comes down to what the purpose is for using such scaling,
science or number scaling. 

 

10 years on - with some better understanding of things (!) - the goal
and conclusions of the presentation still make sense - but now I fully
understand why. Rasch scaling cannot "uncover" a linear latent variable
from ordinal measures. It simply scales counts and in effect, the
numbers applied to its algorithms, without regard to whether those
counts or numbers are drawn from an ordinal or linear scale. 

 

The mistake made by many psychologists is to forget that latent variable
theory implies nothing about the measurement properties of the variable
of interest - latent variables are simply constructed ad-hoc to possess
linear properties of measurement. That is not how normal science
proceeds, it is as Michell states a "pathology of science" (2000).

 

I propose that a key exemplar which shows how to properly model data
while invoking a latent variable, is the work done by Metametrics. It is
no accident that the initial exploratory work was empirical and based
upon much cognitive psychological experimentation, PRIOR to the
scaling/modeling exercises. Andrew has already provided excellent
explanations of the history of this work, along with another exposition
recently in his peer response to Michell's target article in the journal
Measurement (references below).

 

However, if we view edumetrics-psychometrics as largely
pragmatic/technical work, which is concerned with the efficiencies to be
gained in standards-based testing/examination/cumulative risk-scale
environments, then IRT models in general, and the Rasch model make a
great deal of sense. I think it is an illusion that the Rasch or any
IRT/latent variable model magically produces  "fundamental measurement"
in any sense of the word. Michell (2004, and now 2008) has put paid to
this notion.

 

I don't think this is a controversial point anymore - from the
standpoint of simple logic, the work by Robert Wood, and from my own
small and almost stupid simulation, the Rasch model cannot possibly
"uncover/discover" the true metric for a "statistically constructed
latent variable". It just does what it does given the data with which it
is presented. Whether or not that data is an accurate representation/set
of observations of the phenomenon of interest (my "bad ruler"), the
Rasch scaling will simple create a latent variable anyway - given
sufficient stochastic error in the observations (as with Wood's
coin-tosses). Which is why I think the Metametrics exemplar is so very
important, the scaling is constructed around a wealth of empirical
phenomena and magnitude relationships - and not just banks of "item
responses".

 

Regards ... Paul

__________________________________________________
Paul Barrett                                                918.749-0632
x 326
Chief Research Scientist                                   Skype:
pbar088
Hogan Assessment Systems Inc.
2622 East 21st St., Tulsa, OK 74114          
 

 

References

 

Kyngdon, A. (2008) Treating the Pathology of Psychometrics: An Example
from the Comprehension of Continuous Prose Text. Measurement:
Interdisciplinary Research & Perspective, 6, 1 & 2, 108-113.

 

Michell. J. (2000) Normal science, pathological Science, and
psychometrics. Theory and Psychology, 10, 5, 639-667.

 

Michell, J. (2004) Item Response Models, pathological science, and the
shape of error. Theory and Psychology, 14, 1, 121-129.

 

Michell, J. (2008) Is psychometrics pathological science? Measurement:
Interdisciplinary Research & Perspective, 6, 1, 7-24

 

Wood, R. (1978) Fitting the rasch model - a heady tale. British Journal
of Mathematical and Statistical Psychology, 31, , 27-32.

 

An aside  

The journal "Measurement: Interdisciplinary Research and
Perspective"published issues two issues simultaneously - three target
articles and commenatries on the issue:

The Conceptual Foundations of Psychological Measurement

 

The target papers by Denny Borsboom and Keith Markus are also excellent
expositions of their respective positions. Very nice position pieces.

 

I've attached the journal link here so you can look at the paper titles
etc. 

http://www.informaworld.com/smpp/title~content=g794512699~db=all
<http://www.informaworld.com/smpp/title~content=g794512699~db=all> 


_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
http://mailinglist.acer.edu.au/mailman/listinfo/rasch

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20080722/04a79af8/attachment.html 


More information about the Rasch mailing list