[Rasch] Models of unidimensional constructs have theirlimitations: what is a "good thing"?

Agustin Tristan ici_kalt at yahoo.com
Thu Nov 8 14:36:02 EST 2007


Hello!
  In addition to this problem of one or several dimensions, the discussion with teachers is that "we" (loving measure people) see things is a very simplified way, while "they" say that education and learning is frankly multidimensional; so we are reducing things to a single dimension. I agree with this (I'm teacher too, as many of us), but evidence is very clear that we are trying tp measuring something like blood pressure or glucose because it is a "simple" evidence  of a very complicated situation and functioning of the body. 
  Now, let's think on a 100 meter male runner in Olympics. Does he only train running several hours a day? Of course not! Besides running, he has to go to the gym, strength his arms, torax and other parts of his body, he also has to follow instructions of the trainer regarding breath, mental attitude, motivation and other psychological aspects; he has to check his heart rate, he has to control his food, to eat this or that according to a plan, to be at the correct weight, and N other things. BUT the ONLY evidence we finally have is one 100 meter race! It is probably not fair: many other runners eat correctly what they need...others are very motivated...others have a very good attitude...but the race lasts less than 10 seconds and we cannot see all the effort he has to do to be the fastest man on earth, how many months he has to prepare this 9.8 seconds race. Olympics are not fair then, as a race doesn't consider the "process", only the "result" or the "product".
  Football, artistic skate, even bull-fight are not fair, we judge a single dimension during the match (the test). Health checking is not fair either: a  high level of glucose in my blood is the indicator that I'm diabetic? 
  Is it a limited way of thinking? Certainly it is, but it is very complicated to define the purpose of our measurement, to arrive to measure one dimension, so how can we try to measure something that is many times more complicated? The only solution is to avoid measurement: let's ask all the runners to come for an interview, they will tell how well they train, how well motivated they are, the food they eat, etc, except to participate in a 100 meter race, and  I invite judges to use this information to define which runner will have the gold medal in the next Olympics, for a 100 meter multidimensional race. Am I simplifying too much?
  Hope this helps.
  Agustin Tristan 

"Lang, William Steve" <WSLang at tempest.coedu.usf.edu> wrote:
      "It seems to me that Rasch modelling is working towards being able to use this kind of multi-dimensional thinking, but isn’t quite ready for it yet. We have to be able to reliably build a precision, high-resolution measure (ruler) in one dimension (length), and be able to independently confirm the scale, and even perhaps the units, against a commonly agreed “standard” (like the standard Metre) before we can start constructing multi-dimensional construct spaces using orthogonal single dimensions. Do we have a “standard ability unit” locked away in a vault somewhere, against which our measures of ability can be compared?" (quote from Rod)

This part of your comment provides me with some possible insight, thanks:

Isn't the locked away "meter" or "kilogram" an arbitrary but constant unit?  Someone else could use a "yard" or "pound" as long as they used defined units consistently.  That is pretty much what Lexiles are to reading...you are correct that all disciplines haven't agree on the units, but that is a philosophical decision as much as a scaling problem.  If the construct is defined (scaled), the units are a simple transformation.

I'd be at a loss to discuss string theory, but we measure blood pressure as one dimension even though we know that many variables contribute to the results.  We then use blood pressure as one measure of "health" when added to other variables such as blood sugar, weight, etc.  I think that if we tackle development of the scales that are definable and useful one at a time, we can search out the underlying variables and additive constructs next.  Our "well-constructed" measure of blood pressure is an intermediate starting point.  Any of our measures may be composites of more than one variable and still be useful as long as they function along a linear dimension.  The future determination of the separate underlying variables may remain unknown for now.

If we try to understand all the variables to create a "super measure" that accounts for everything to start with, we will never visualize a useful instrument.  That is like a "Healthometer" that gives one value for all variables.  It is difficult to visualize or diagnose all the elements at once, yet we often hear social scientists attempt to do that.

It seems most important to me that we avoid creating instruments based on tools that are not interval level (2PL and 3PL for example), or we will never define the "locked away units" accurately so that we can create the scales!  Mixing questionable instrument development (2PL and 3PL) with attempts to explain "all the variables" at all dimensions really confounds the problem and we are stuck in quicksand.  This seems to be a common occurrence.

It seems to me that Rasch provides a tool to scale defined units.  The difficult restriction of Rasch is actually necessary for each of the measures to ultimately be useful as part of the "system".  It also seems that there are some efforts to develop multi-dimensional extensions (see Multivariate and Mixture Distribution Rasch Models, Davier & Carstensen), but the tendency to define everything at once confuses me.  I'm happy with one well-defined construct at a time that is useful for a given purpose.  Rod is right though, eventually some complex systems will have to be explained. 

Simplistically for me, fit analysis leads to explaining the underlying variables.  FACETS might be one way to "add" dimensions across scales.  Scaling a set of assessments that are linked by a taxonomy or framework is another non-mathematical way to visualize "dimensions" in social science.

Steve Lang
University of South Florida St. Petersburg


-----Original Message-----
From: Rodney Staples [mailto:rodstaples at ozemail.com.au]
Sent: Tue 11/6/2007 7:46 PM
To: Lang, William Steve; Rod O'Connor; Trevor Bond; rasch at acer.edu.au
Cc: jwilkers at fgcu.edu
Subject: RE: [Rasch] Models of unidimensional constructs have theirlimitations: what is a "good thing"?

Hi Folks,
Thank you all for you most interesting and informative discussions over the
past few weeks. Please keep up the kinds of discussion, even if the initial
question seems "dumb", because to relative "newbies" like me it is really
useful.

Now its time for me to add my "three-penneth" to the discussions!

In the physical sciences we have no problem talking about the three
dimensions of physical space, or about the four dimensions of space-time, or
the 11? Dimensions currently associated with string theory. I have no
problem imagining a circumstance in complex human behaviour that needs more
than a single dimension to describe. But in the physical sciences we use
measurements in the physical space, one dimension at a time, to create areas
and volumes. or tesserects!

It seems to me that Rasch modelling is working towards being able to use
this kind of multi-dimensional thinking, but isn't quite ready for it yet.
We have to be able to reliably build a precision, high-resolution measure
(ruler) in one dimension (length), and be able to independently confirm the
scale, and even perhaps the units, against a commonly agreed "standard"
(like the standard Metre) before we can start constructing multi-dimensional
construct spaces using orthogonal single dimensions. Do we have a "standard
ability unit" locked away in a vault somewhere, against which our measures
of ability can be compared?

I'm happily working on trying to build a uni-dimensional construct around
ability in a context, and that is more than enough challenge for me at the
moment. it is the reason why I find your recent discussion so relevant and
timely. But I can see a time, not so far into the future, when I will need
to consider more than one dimension.

For example, we can build a single construct around ability as a
technologist. But when a technologist also must perform as a manager, then
the construct defining the role as a manager will be quite different from
that describing the underlying technology latent trait (although there may
be some generalised abilities that are common. beyond my capacity to deal
with at this stage). Thus to define an engineering manager, we need to have
a two dimensional construct: an engineering dimension; and a management
dimension. In another case more closely related to education, we already
have multi-dimensional assessment when we have a theory component to
assessment (an exam), and a practical component (laboratory work, etc).

Thanks again for the discussion
Regards,
Rod


___________________________________________________________________________
Dr. Rodney Staples.
e-mail: rodstaples at ozemail.com.au
Telephone: +61 3 9770 2484
Mobile: +61 4 1935 9082
Web: http://members.ozemail.com.au/~rodstaples/


-----Original Message-----
From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au]On Behalf
Of Lang, William Steve
Sent: Tuesday, 6 November 2007 11:42 PM
To: Rod O'Connor; Trevor Bond; rasch at acer.edu.au
Cc: jwilkers at fgcu.edu
Subject: RE: [Rasch] Models of unidimensional constructs have
theirlimitations: what is a "good thing"?

I believe this conversation is on track!  Just as "reading comprehension"
was bogged down in a complex search for "variables" when Lou Bashaw and Jack
Stenner envisioned a usable measure that spurred a new wave of research,
Rasch "forces" the scale to understand and map the relevant construct at a
targeted level.

Judy Wilkerson and I also witnessed an endless and unresolvable debate about
the complexities of "teacher dispositions" based on morals, ethics, and
personality that resulted in useless attempts to measure the construct.
(See "Teacher Dispositions: Building a Teacher Education Framework of Moral
Standards", "Teacher Dispositions: Envisioning Their Role in Education", and
"Dispostions in Teacher Education" - all recent books.)  Decades of debate
could not deconstruct the variables.  The solution was to validate
job-related and predictive measures based on a defined and mapped construct
that doesn't attempt to sub-divide all the possible variables. (I hope that
Lou realizes that I was listening in that class!)

Now that the disposition scales are becoming available, appropriate research
and discussion make sense.  Without the measures, you have an endless
"factor analysis nightmare" based on any individual's idea du jour.  Until a
useful measure at some useful dimensional level starts the process, it is
unlikely that a synthesis of "stuff" will magically become useful.  I think
this appears to happen in personality, theories of creativity, theories of
intelligence, and likely some of the health care measures. (Does anyone
remember the Functional Independence Measure discussion on thresholds in
rmt?)

When the "model", such as 3PL, allows the measures to "play around" with the
parameters so that there is a false fit, you are off on the invalidity
slippery slope.  Rasch seems (to me) to force the measures to explain the
construct AT THE LEVEL OF INTEREST.  Once it is working, there are plenty of
opportunities (FACETS, etc.) to break down and aggregate the real
constituent parts!

More importantly to those of us who are looking at "complex human
variables", it is the DPF analysis that seems to inform the construct
interpretation in ways that simply are not envisioned in the original scale
development!  If you look at "Item Response Theory" (Baker & Kim) or "Item
Response Theory for Psychologists" (Embretson & Reise), you hardly see any
mention of keyforms, kidmaps, or construct maps that are familiar to us
Rasch folks.  I don't think you can EVER VALIDATE (or interpret) the
construct or sub-constructs of these complex variables without the person
profiles which are completely confounded in 2PL and 3PL models!

Judy and I will repeat, "Rasch puts the people in assessment".  If you can't
develop the valid unidimensional constructs at some useful level (especially
complex human ones), then you'll likely never have the research tools that
encourage one to break out of the tail-chasing debates and identify the
"real variables" that actually exist in the data.

This is entirely applied and practical, regardless of the mathematical logic
of sufficiency, specific objectivity, and independence even though I'm glad
that others more talented than me have provided that logic.

Steve Lang
University of South Florida St. Petersburg


-----Original Message-----
From: rasch-bounces at acer.edu.au on behalf of Rod O'Connor
Sent: Tue 11/6/2007 12:25 AM
To: Trevor Bond; rasch at acer.edu.au
Subject: Re: [Rasch] Models of unidimensional constructs have their
limitations

Hello Trevor, very nice to hear from you (I recall your earlier
assistance...).

I have no doubt one could fit global HQoL data to a single Rasch scale
(any notion can be/is one construct, and multiple items can generally be
identified), however I fear that people might think this is
automatically a good thing and that having done so the healthcare
researcher has 'done their job'.  Unfortunately in principle such a
scale might not be particularly helpful in 'improving healthcare',
potentially offering little more than a 'do you feel better' question
(except the scale would have more than one item).  My view is that a
HQoL measure useful for healthcare improvement needs to not only provide
an indication of extent but also play a diagnostic role (allow
'formative' as well as 'summative' evaluation), identifying where and
how a change has occurred, and hence where treatment could usefully be
targeted.

Of course it all depends on the criteria by which one would assess
success for the measure, which should always be set-up first.
Personally I think an overtly multi-dimensional assessment is likely to
be needed, with an accompanying complex scoring/combination rule to
predict values consistent with patient judgement.

Warm regards,
Rod


_Trevor Bond said the following on 6/11/2007 3:34 PM:_
> mea culpa...denied not denied (sorry Dr Freud...it was a slip)
>> Dear Rod
>> HRQoL is one of the areas where Rasch is having a major impact, if I
>> can judge by the articles I am asked to review.
>> Are multidimensional states a composite of unidimensional states?
>> Why would we ever expect that any human condition could be DEFINED by
>> one dimension?
>> thanks for your prompt!
>> TGB
>
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
http://mailinglist.acer.edu.au/mailman/listinfo/rasch

No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.5.503 / Virus Database: 269.15.22/1112 - Release Date: 5/11/2007
7:11 PM


_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
http://mailinglist.acer.edu.au/mailman/listinfo/rasch



FAMILIA DE PROGRAMAS KALT. 
Mariano Jiménez 1830 A 
Col. Balcones del Valle 
78280, San Luis Potosí, S.L.P. México 
TEL (52) 44-4820 37 88, 44-4820 04 31 
FAX (52) 44-4815 48 48 
web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com

 __________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20071107/6f4ca3a0/attachment.html 


More information about the Rasch mailing list