[Rasch] Rasch applied rashly to quash innovation

Looveer, Juho Juho.Looveer at det.nsw.edu.au
Fri Dec 14 10:26:10 EST 2007

Fellow Raschers (and interested lookers-on)
Part of the problem could be due to how the original scale was constructed.
We undertook some work in constructing a Kindergarten-Year 6 numeracy scale in recent years.  We found that the original scale did not work well.  The problem was due to the performance of link items (items linking testlets across different year groups).  Items that had been purported to be suitable as link items actually did not work as expected; this was due to various factors such as recency of revision of that topic, i.e. Year 2 may have recently learned a topic that the Year 3 students had not revised in the previous 12 months; hence the Year 2 students appeared to perform better on an item than did the Year 3 students.
After revising each item for year-DIF and using different items as links, the scale worked well.
A paper on this has been submitted to the forth-coming ARM, and will also be presented at the Rasch conference in Perth (Down-Under = Australia) in January 2008.  
[I understand that the basic issues have also been presented to some groups in N America]
Happy to discuss further with anyone interested.
Dr Juho Looveer
Planning and Innovation
NSW Department of Education and Training
Level 5, 35 Bridge St Sydney
GPO Box 33
Sydney NSW 2001
work phone: 956 18192
work fax: 956 18055
Juho.Looveer at det.nsw.edu.au 


From: rasch-bounces at acer.edu.au on behalf of Steven L. Kramer
Sent: Fri 14/12/2007 2:25 AM
To: rasch at acer.edu.au
Cc: Robin Marcus
Subject: [Rasch] Rasch applied rashly to quash innovation

Dear Rasch community:
I am going to make a controversial and important claim, and provide evidence for it.  I'd like your response, indicating 
a. Am I wrong? and 
b. If I'm correct, what can be done about it?
Adaptive testing in mathematics diagnosis and placement is punishing mathematics innovators and harming their students.  Inappropriate application of Rasch modeling is largely to blame.
The specific example I have in mind is the Core Plus mathematics curriculum and its interaction with the AccuPlacer college placement test--but the principle applies to other innovations and other tests.
Core Plus was one of the "integrated" high school mathematics curricula developed in the 1990s with U.S. National Science Foundation funding.  It teaches concepts in what the developers believed was a superior and easier to learn order than is traditional.  Compared to Traditional students, pilot testing indicated that first edition Core Plus students on average scored much higher on "pre-Calculus" and other advanced topics--but somewhat lower on fluent application of Algebra 1 procedures.  
Curriculum developers also consciously set out to reduce "barriers" denying students access to interesting mathematics they might learn.  
So, a student unable to consistently multiply 35 x 27 by hand might be given a pocket calculator, and might learn Algebra 1, Geometry, and Trigonometry quite well.
The harm from Accuplacer and other adaptive tests resides in inappropriately assuming that mathematics skill is unidimensional.  Mathematics items are ordered by difficulty on this uni-dimensional scale, and it is assumed that a student who is 75% likely to be able to solve a word problem that requires setting up simultaneous equations, or with a solid conceptual understanding of how multiplication by "i" can be conceived as a 90 degree rotation in the coordinate plain,  is almost certain to be able to multiply 35 x 27.  But for Core Plus students this assumption is not true.
At the extreme end, the result is this:  if students are started at the "difficult" problems in an adaptive test like Accuplacer, they place into Calculus.  If started on the "easy" problems, they fail enough so that they place into arithmetic.  Most testers start with the easiest end of the test and work up--so the Core Plus students and teachers are punished.
Similar applications of adaptive testing, e.g. by the Northwest Education Association, have been used for formative assessment--but they diagnose Core Plus students incorrectly.
Bottom line:  nothing in real life ever fits a model perfectly, and no test or learning construct is ever truly unidimensional.  But often, tests and constructs are close enough to "unidimensional" so that a Rasch model can be applied fruitfully.  Often, this uni-dimensionality is socially constructed, as when everyone has the same math concepts instructed in the same order, so that in typical situations with a typically instructed population something like "math skill" can be usefully treated as though it were uni-dimensional.
But in context of an innovation, this uni-dimensionality assumption is shattered.  And the "rebel" population is punished, because people are blindy over-applying the Rasch model--which in an adaptive testing environment that starts with typically 'easy' problems punishes non-conformists with low scores.
Am I correct?  And what should be done?
Steve Kramer
Arcadia  University
This message is intended for the addressee named and may contain
privileged information or confidential information or both. If you
are not the intended recipient please delete it and notify the sender.

More information about the Rasch mailing list