[Rasch] PISA critique in TES - a competition? - from a distance

John Barnard JohnBarnard at bigpond.com
Thu Oct 2 12:35:57 EST 2014

This is certainly not an easy task - trying to "satisfy" the measurement
world as well as the "world of politics" - and then we have reality! We
should not forget that the Rasch model is a mathematical model. It most
certainly advanced our understanding and analyses from the classical stages,
but it cannot perfectly solve all the measurement problems and issues we

Let's start at the beginning - what are we trying to measure? A "simple"
concept such as mathematical ability can be defined in so many ways, e.g. a
crystallized definition related to a certain curriculum/syllabus or a fluid
definition based on lateral thinking and solution of "new" problems. It is
obvious that measures of these two constructs will differ. Is mathematical
ability defined the same in the different countries? What about the teaching
thereof - are all based on a formalist approach or are some intuitionist,
logicist or outcomes-based? Would this not make a difference? Are all at the
same level, e.g. a question to differentiate 2x + 3 will be a
straightforward simple question after a week of introductory calculus but a
mind boggler to a student who hasn't heard of calculus yet, although they
may be the same age and even the same year level but the teachers taught
topics in different order. (I know that they are trying to compare apples
with apples, but you get what I mean). I don't even want to get into issues
of translations of questions into different languages. Is the emphasis on
norms or what students can actually do? Over so many different countries we
may be pushing it when claiming apples are compared to apples - irrespective
of all the sophisticated DIFF analysis, MH, etc. But, even if we succeed,
politicians use the info to serve their purposes and promises. Some years
ago I heard one say that if he is elected, he will make sure that all
students perform above average ...


-----Original Message-----
From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf
Of Stefan Cano
Sent: Wednesday, 1 October 2014 8:58 PM
To: rasch at acer.edu.au
Subject: Re: [Rasch] PISA critique in TES - a competition? - from a distance

My own view:

There is critique and then there is criticism. The former is always
important; the latter more often "trivially true", but less helpful (and
more divisive) unless something better is proposed.

Like Thomas I am outside of education. And I share his concerns. 

And so taking an "outsider's" viewpoint, it could be argued that the two
take home messages from this thread are: 1) the Rasch model isn't very
useful; and 2) those who use the Rasch model not only do not agree with each
other, they actually think each other is "wrong".

I would suggest that's probably not very helpful.

-----Original Message-----
From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf
Of Bond, Trevor
Sent: 01 October 2014 11:07
To: <rasch at acer.edu.au>
Subject: Re: [Rasch] PISA critique in TES - a competition? - from a distance
Importance: High

Are we suggesting that critique should be hidden from view?

Sent from 007's iPad

> On 1 Oct 2014, at 5:50 pm, "Stefan Cano" <stefanjcano at gmail.com> wrote:
> Thank you for posting this Thomas. Very nicely articulated.
> -----Original Message-----
> From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On 
> Behalf Of David Andrich
> Sent: 30 September 2014 12:45
> To: rasch at acer.edu.au
> Subject: Re: [Rasch] PISA critique in TES - a competition? - from a 
> distance
> Importance: High
> Totally agree with Steve. 
> David
> David Andrich, BSc MEd W.Aust., PhD Chic, FASSA Chapple Professor 
> david.andrich at uwa.edu.au  Graduate School of Education The University 
> of Western Australia M428, 35 Stirling Highway, Crawley, WA, 6009
> Telephone: +61 8 6488 1085;   Fax: +61 8 6488 1052
> www.matildabayclub.net
> http://www.education.uwa.edu.au/ppl/courses<SPAN style="COLOR: nav 
> ________________________________________
> From: rasch-bounces at acer.edu.au [rasch-bounces at acer.edu.au] On Behalf 
> Of Stephen Humphry [stephen.humphry at uwa.edu.au]
> Sent: Tuesday, 30 September 2014 7:19 PM
> To: rasch at acer.edu.au
> Subject: Re: [Rasch] PISA critique in TES - a competition? - from a 
> distance
> Nicely said Thomas.
> Steve
> From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On 
> Behalf Of Thomas Salzberger
> Sent: Tuesday, 30 September 2014 6:55 PM
> To: rasch at acer.edu.au
> Subject: Re: [Rasch] PISA critique in TES - a competition? - from a 
> distance
> Dear all,
> I have been watching the discussion from a distance since education is 
> not my field of expertise.
> Consequently, I refrain from making specific comments about the content.
> Anyhow, in my mind, it is unfortunate that the controversy divides 
> people who are all, by and large, building upon the same premises and 
> sharing more or less the same understanding of what measurement is or 
> what
it should be.
> It is an all to frequent phenomenon that the Rasch-community/ies
> get(s) roped into discussions on issues we actually should not assume 
> responsibility for.
> I, personally, do believe that PISA is first and foremost a political
> Isn't there an obsession with comparing everything, everybody, and 
> every country?
> Admittedly, I do see the point. It is interesting to compare the 
> outcome of educational systems of all sorts across all possible 
> countries and derive conclusions. The ones lagging behind can learn 
> from
the vanguard. All fine.
> The objectives are commendable.
> On the other hand, when does it come done to comparing apples and oranges?
> Aren't there different curricula? Different emphasis from one country 
> to another?
> Different people? Culture?
> Again, I am not an expert in the field, but I would think all of the 
> above matters.
> Or should all be the same all over the world? Globalization, same 
> goals, one system?
> Should we all teach to the PISA-test?
> Rasch is a wonderful tool. Providing us with very useful properties if 
> requirements are met.
> But isn't the frame of reference key? How far can we stretch the frame 
> of reference when it comes to PISA? Apples and oranges, important? All 
> fruits, after all.
> However, what if one country tops the list in terms of apples but 
> performs poorly in terms of oranges?
> Whisk the thing and it is all the same?
> I assume that multiple rankings (or a range of rankings) provide a 
> much more valid representation, if one wants to apply Rasch. And I 
> further assume everybody familiar with Rasch and measurement openly or
secretly agrees.
> (Maybe I am wrong.) The problem is, there is a political agenda. No 
> politician (and the public) would have a clue what these fuzzy 
> rankings would mean. They want an unambiguous answer. Simplicity 
> rather than complexity. Damn thing that we live in a complex world.
> In fact, I do understand politicians. If they want to derive clear 
> conclusions, one equivocal ranking is much more useful. So, I do 
> understand all who are behind PISA when they desperately try to 
> achieve
> The question remains what conclusions politicians actually draw anyway?
> This is not a rhetorical question, I really do not know the answer.
> In my own country (Austria), I observed that
> - politicians refer to the gap between Austria and the top (mostly the
> Finns) whenever they want to emphasise that something needs to be done 
> and the thing that needs to be done is what they have always said 
> needs to be done (Pisa as the justification that action is required, 
> but nobody really thinks about how the apparently successful 
> procedures followed in Finland could be implemented in Austria; when 
> asked about that politicians say "Ah that wouldn't work here" or 
> "That's
too expensive, we cannot afford it").
> - ruling parties acclaim any change to the better (even if it may only 
> be a result of random fluctuation, it's different cohorts, after all), 
> and claim that this is a sign that their policy is the right one
> - whenever there is decline, ruling parties take this is evidence that 
> there policy is still the right one but that we need more of that 
> policy (we are on the right track, but it takes time, etc).
> - the opposition claims the reverse.
> No wonder that, as a citizen, I have doubts as to the usefulness of 
> PISA once the results are in the hand of politicians. I guess experts 
> in education might learn more from PISA results and they would know 
> how to interpret ambiguity. Alas, it is not experts who rule this 
> world (perhabs this is a good thing).
> My fear is that the controversy about PISA will not be understood as a 
> controversy primarily about PISA but more as one about the Rasch model.
> People will say "PISA? Isn't that this test where this obscure Rasch 
> model is applied?"
> Or "No wonder that they are in a mess. Rasch just doesn't work."
> This is not to say that we should not discuss the issue. It is one of 
> the great scientific assets of Rasch measurement that we do not shy 
> away from such debates. But we need to emphasise that problems, if 
> any, are not due to the Rasch model but due to the immense complexity 
> of
the task.
> How this can be done? I don't know.
> In any case, the purpose of Rasch is exposing problems, not 
> necessarily solving them.
> This brings us back to politics. Politicians don't fancy problems, 
> they want solutions ...
> Best
> Thomas
> --
> ___________________________________
> Thomas.Salzberger at gmail.com<mailto:Thomas.Salzberger at gmail.com>
> Thomas.Salzberger at wu.ac.at<mailto:Thomas.Salzberger at wu.ac.at>
> ________________________________________
> Rasch mailing list
> email: Rasch at acer.edu.au
> web:
> https://mailinglist.acer.edu.au/mailman/options/rasch/stefanjcano%40gm
> ail.co
> m
> ________________________________________
> Rasch mailing list
> email: Rasch at acer.edu.au
> web: 
> https://mailinglist.acer.edu.au/mailman/options/rasch/trevor.bond%40jc
> u.edu.au
Rasch mailing list
email: Rasch at acer.edu.au

Rasch mailing list
email: Rasch at acer.edu.au

More information about the Rasch mailing list