[Rasch] PISA critique in TES - a competition?

Brady Michael Jack bradyjack at gmail.com
Mon Sep 29 11:23:54 EST 2014

Hi Steve,

In the article "Linking PISA competencies over three cycles - results from
Germany", Claus H. Carstensen (p. 204) explains how the uni-dimensionality
of PISA items are assessed. His description may be of value to the

Carstensen, C. H. (2013). Linking PISA competencies over three cycles -
results from Germany. In M. Prenzel, M. Kobarg, K. Schöps & S. Rönnebeck
(Eds.), *Research on PISA: research outcomes of the PISA research
conference 2009* (pp. 199-214): Springer.


On Mon, Sep 29, 2014 at 8:51 AM, Steve Kramer <skramer1958 at verizon.net>

> There  is a lot of noise in the TES article, but one potentially legitimate
> problem: Kreiner claims that the PISA questions violate the
> Rasch uni-dimensionality assumption to such a degree that a Rasch model
> can't be used, or at least can't be used with sufficient precision to rank
> countries meaningfully.  He says he tested this by trying out legitimate
> subsets of questions, and investigating whether the differing subsets
> predicted rankings that were similar to one another-and they didn't.
> Specifically, "Canada could have finished anywhere between second and 25th
> and Japan between eighth and 40th.".
> Ray Adams responded by saying that PISA accounts for this problem, noting,
> "We have always shown things like range of possible ranks, standard errors
> and so on. We've also reported the effects of item selection ."
> In fact the 2012 country report on Japan looks nothing like "between 8th
> and
> 40th".  In 2012 Japan was ranked between 1st and 3rd in all subjects.
> Kreiner may have been using data from a different year, but I can't imagine
> that the RANGE of possible ranks shrank from 32 (40-8) down to 2 (3-1).  I
> see only two possibilities:  either Kreiner's methodology was absolutely
> lousy, or else the PISA "range of ranks" was computed ASSUMING
> uni-dimensionality and did not adequately CHECK for uni-dimensionality.
> Ray, do you have any technical articles you can reference explaining how
> PISA either designed test items for uni-dimensionality or else checked that
> uni-dimensionality was an adequate model for creating country ranks? Also,
> are there any tech reports on how PISA determined each country's potential
> range of ranks?   Finally, I'd appreciate any tech reports in which PISA
> investigated how choosing differing subsets of items affected country
> ranking, or else articles (not necessarily PISA) explaining why a procedure
> like Kreiner's sub-setting is not a legitimate test.    I suspect that
> Kreiner's claims are simply based on invalid methodology, but I'd like to
> be
> able to verify that suspicion.
> Steve Kramer
> The 21st Century Partnership for STEM Education
> -----Original Message-----
> From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
> Behalf
> Of Mike Linacre
> Sent: Sunday, September 28, 2014 6:13 PM
> To: rasch at acer.edu.au
> Subject: Re: [Rasch] PISA critique in TES - a competition?
> Yes, Jason.
> Are PISA, TIMSS and similar studies really intended to advance education
> worldwide or to advance political agendas? If the answer is "political
> agendas" then the most politically-acceptable statistical methodologies are
> the ones to choose. If the answer is "advance education" then everyone,
> including the politicians, should be working towards discovering and using
> the most effective statistical methodologies.
> According to www.rasch.org/software.htm there are now seven Rasch-related
> R
> modules. They are free. Wonderful! But there are 5,889 R modules. We will
> need more Rasch R modules before we make a noticeable impact.
> Mike L.
> On 9/28/2014 21:21 PM, liasonas wrote:
> > Mike, this is a great idea. But can the policy makers and the
> > politicians allow us (the academics) to spoil their new toys ( the
> > international studies)? The politicians use us (the academics) to
> > produce data and reports which then the politicians use to carry out
> > their little in-fightings and political debates.
> >  We cannot afford to angry them, because we need their money and
> > support. Maybe we need to train them on how to use our data most
> > appropriately and sensibly. Pisa and Timms tables, for example, can be
> > useful, but they are not the equivalent of The Bible.
> > Having said all that, we need to thank Margaret and the other
> > researchers for providing the methodological tools and packages (have
> > you all had a glance of the TAM package on the R platform?). But we
> > also need to thak Paul for seeding the seeds of doubt, because this is
> > the only way for science to prosper.
> > Jason
> ________________________________________
> Rasch mailing list
> email: Rasch at acer.edu.au
> web:
> https://mailinglist.acer.edu.au/mailman/options/rasch/skramer1958%40verizon.
> net
> ________________________________________
> Rasch mailing list
> email: Rasch at acer.edu.au
> web:
> https://mailinglist.acer.edu.au/mailman/options/rasch/bradyjack%40gmail.com

Brady Michael Jack, Ph.D.
Postdoctoral Research Fellow
Center for General Education
National Sun Yat-sen University
70 Lien Hai Road,
Kaohsiung 804 Taiwan
Phone: +886-7-525-2000
Website: http://www.researchgate.net/profile/Brady_Jack/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20140929/5322c1d0/attachment-0001.html 

More information about the Rasch mailing list