[Rasch] PISA critique in TES - a competition?
Brady Michael Jack
bradyjack at gmail.com
Mon Sep 29 11:23:54 EST 2014
In the article "Linking PISA competencies over three cycles - results from
Germany", Claus H. Carstensen (p. 204) explains how the uni-dimensionality
of PISA items are assessed. His description may be of value to the
Carstensen, C. H. (2013). Linking PISA competencies over three cycles -
results from Germany. In M. Prenzel, M. Kobarg, K. Schöps & S. Rönnebeck
(Eds.), *Research on PISA: research outcomes of the PISA research
conference 2009* (pp. 199-214): Springer.
On Mon, Sep 29, 2014 at 8:51 AM, Steve Kramer <skramer1958 at verizon.net>
> There is a lot of noise in the TES article, but one potentially legitimate
> problem: Kreiner claims that the PISA questions violate the
> Rasch uni-dimensionality assumption to such a degree that a Rasch model
> can't be used, or at least can't be used with sufficient precision to rank
> countries meaningfully. He says he tested this by trying out legitimate
> subsets of questions, and investigating whether the differing subsets
> predicted rankings that were similar to one another-and they didn't.
> Specifically, "Canada could have finished anywhere between second and 25th
> and Japan between eighth and 40th.".
> Ray Adams responded by saying that PISA accounts for this problem, noting,
> "We have always shown things like range of possible ranks, standard errors
> and so on. We've also reported the effects of item selection ."
> In fact the 2012 country report on Japan looks nothing like "between 8th
> 40th". In 2012 Japan was ranked between 1st and 3rd in all subjects.
> Kreiner may have been using data from a different year, but I can't imagine
> that the RANGE of possible ranks shrank from 32 (40-8) down to 2 (3-1). I
> see only two possibilities: either Kreiner's methodology was absolutely
> lousy, or else the PISA "range of ranks" was computed ASSUMING
> uni-dimensionality and did not adequately CHECK for uni-dimensionality.
> Ray, do you have any technical articles you can reference explaining how
> PISA either designed test items for uni-dimensionality or else checked that
> uni-dimensionality was an adequate model for creating country ranks? Also,
> are there any tech reports on how PISA determined each country's potential
> range of ranks? Finally, I'd appreciate any tech reports in which PISA
> investigated how choosing differing subsets of items affected country
> ranking, or else articles (not necessarily PISA) explaining why a procedure
> like Kreiner's sub-setting is not a legitimate test. I suspect that
> Kreiner's claims are simply based on invalid methodology, but I'd like to
> able to verify that suspicion.
> Steve Kramer
> The 21st Century Partnership for STEM Education
> -----Original Message-----
> From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
> Of Mike Linacre
> Sent: Sunday, September 28, 2014 6:13 PM
> To: rasch at acer.edu.au
> Subject: Re: [Rasch] PISA critique in TES - a competition?
> Yes, Jason.
> Are PISA, TIMSS and similar studies really intended to advance education
> worldwide or to advance political agendas? If the answer is "political
> agendas" then the most politically-acceptable statistical methodologies are
> the ones to choose. If the answer is "advance education" then everyone,
> including the politicians, should be working towards discovering and using
> the most effective statistical methodologies.
> According to www.rasch.org/software.htm there are now seven Rasch-related
> modules. They are free. Wonderful! But there are 5,889 R modules. We will
> need more Rasch R modules before we make a noticeable impact.
> Mike L.
> On 9/28/2014 21:21 PM, liasonas wrote:
> > Mike, this is a great idea. But can the policy makers and the
> > politicians allow us (the academics) to spoil their new toys ( the
> > international studies)? The politicians use us (the academics) to
> > produce data and reports which then the politicians use to carry out
> > their little in-fightings and political debates.
> > We cannot afford to angry them, because we need their money and
> > support. Maybe we need to train them on how to use our data most
> > appropriately and sensibly. Pisa and Timms tables, for example, can be
> > useful, but they are not the equivalent of The Bible.
> > Having said all that, we need to thank Margaret and the other
> > researchers for providing the methodological tools and packages (have
> > you all had a glance of the TAM package on the R platform?). But we
> > also need to thak Paul for seeding the seeds of doubt, because this is
> > the only way for science to prosper.
> > Jason
> Rasch mailing list
> email: Rasch at acer.edu.au
> Rasch mailing list
> email: Rasch at acer.edu.au
Brady Michael Jack, Ph.D.
Postdoctoral Research Fellow
Center for General Education
National Sun Yat-sen University
70 Lien Hai Road,
Kaohsiung 804 Taiwan
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Rasch