[Rasch] complex sampling Rasch -- not sample free?

Adams, Ray adams at acer.edu.au
Mon Mar 1 18:39:45 EST 2010


Svend,

All PISA data is available for you to check yourself.  For example at: http://pisa2006.acer.edu.au/downloads.php

You will find everything you need.  raw data files, scored data files, spss and sas syntax to read and fully document then files.  There are other matching sites for 2000 and 2003.

As far as I can see Alla's note to you is full and comprehensive.   We prepared the DIF statistics etc and return them to national centres who own them. ACER does not own the data and cannot send nationally owned materials to all of the many researchers who contact us.  She has referred you to the Danish team and I can assure you they have DIF information -- lot's of it.  If they are not sure of what you are asking for then I suggest you ask them to speak with us so that we can help them.

But as I said, all data is the public domain, easily accessible and fully documented so you can scale it all yourself if you wish.

Ray


-----Original Message-----
From: Svend Kreiner [mailto:S.Kreiner at biostat.ku.dk]
Sent: Mon 2/22/2010 11:02 PM
To: Adams, Ray
Cc: Agustin Tristan; rasch
Subject: Re: [Rasch] complex sampling Rasch -- not sample free?
 
Ray,

Thank you for responding to my comments. My comments were caused by my 
failed attempts to get information on the issue of country DIF from the 
Danish PISA group and from Acer. The Danish manager claimed that he knew 
nothing about any reports on these issues (this is in print in the 
monthly journal of the Danish Teacher's organization in 2005). Instead 
he suggested that I should contact Acer for such information. I have 
added a pdf file with my e-mail to Acer and the response I got back. The 
response consisted of some remarks that are basically a quote from the 
technical report plus the information that I should contact the PISA 
center in Denmark if I wanted the reports. Since I got no response on my 
second e-mail to ACER and since the Danish PISA group had said that they 
knew of no such reports I gave up on this line of inquiry. I have also 
downloaded hundreds of pages on PISA looking for concrete information on 
country DIF in the educational tests. The only things that I could find 
on DIF in the educational tests in the 2006 technical report were the 
chapter on "Scaling PISA Cognitive Data" where the DIF procedure was 
described and Table 12.5 on "Items deleted at the national level". This 
is hardly evidence supporting the claims that PISA tests (after 
purification) has no country DIF, and by the way not evidence that the 
PISA items fit a Rasch model at all, buf if you say so I am prepared to 
believe that the evidence must exist. I would like to see it, however, 
so please point me in the right direction.

I have added some additional comments below.

Svend


Adams, Ray skrev:
>
> Svend & others,
>
>  
>
> (1)  Your comment "The country DIF problems could be easily solved by 
> item splitting across countries if they cared to do so" suggests 
> matters such as this are ones that we don't think long and hard about 
> -- I can assure you we do.
>
>  
>
> I remind you of Steve Humphry's post were he makes it clear that you 
> don't magically create sample independence by splitting items, his 
> point are subtle but important. That being said I could certainly be 
> convinced that splitting items may well result in more valid 
> measurement and it has routinely been the  recommendation of the 
> technical panel of PISA and before that TIMSS. But, now let's 
> introduce a touch of reality.  Suppose I have an item that exhibits 
> DIF and is relatively harder in Japan than Denmark.  The technical 
> advice is therefore to treat it as two different items, so that in 
> Japan the parameter estimate for the item will be a higher number than 
> in Denmark.  Now we have a sample of Japanese students who respond to 
> a PISA booklet which contains this item and their mean raw score is 
> 50%, similarly we have a sample of Swedish students who responded to 
> the same booklet and they also have a mean raw score of 50%.  In the 
> PISA table that shows mean raw scores the two countries will have the 
> same results for this booklet, but for (Rasch) scale scores Japan will 
> have a higher mean than Denmark.  I invite you to explain this to your 
> local Minister or talk-back radio host. Once you've successfully done 
> that you can go to the US and explain the same phenomenon when it 
> occurs in the comparison of African Americans and non-African Americans.
>
That is not how I read Steve's comments, but it makes it clear that we 
should be more precise about what we mean when we talk about sample 
independence. I can add, that item parameter estimates are never sample 
free in the sense that the error of the estimates (and therefore also 
the error of the person estimates) depend on the size of the sample and 
how well the items are targeted to the sample.

Concerning the minister and - in particular - the talk-back radio host 
we do have a problem making them understand what we are doing. Do they 
understand what a logit is? Why it is better with plausible values than 
score and person estimates. I do not think the item splitting is more 
difficult that the rest of the technicalities and I have been able to 
make my contacts in the Ministry of Education understand the idea about 
item splitting.
>
>  
>
> (2) Your comment: "the managers try to hide the fact" is a 
> complimentary one indeed, thanks. Can I request your evidence for 
> this?  All data from PISA concerning the technical quality of the test 
> is available to the public. The only secure materials are the actual 
> items that are to be used for equating purposes.  I invite you to 
> peruse the technical reports, the websites (mypisa.acer.edu.au and 
> www.pisa.oecd.org <http://www.pisa.oecd.org>) and contact 
> pisa at acer.edu.au <mailto:pisa at acer.edu.au>.  You will find that DIF by 
> country (and other variables) is fully reported and discussed.
>

I have checked the websites, but have not been able to find the country 
DIF analyses and the other results supporting the Rasch models. Is data 
on item responses available for the public (e.g. the calibration sample 
containing responses from 500 students from each country)? If the data 
is available we can check for ourselves that things are ok.
>
>  
>
> One of your other comments is relevant here -- a comment that I 
> completely agree with.  You say: "Personally, I do not embrace the 
> idea that samples and the power of test statistics can be too large 
> myself."  With this I agree completely and I find discussions of about 
> the effect of sample size on fit tests particularly aggravating. Of 
> course fit tests should be influenced by sample size, the more data I 
> have the more information I have and the better off I am in terms of 
> understanding lack of fit and the magnitude of that lack of fit.  The 
> hard question is what to do when you observe misfit (eg DIF) and here 
> unlike you I don't believe there are easy answers.  If your study is 
> as big and as well designed and implemented as PISA you'll find DIF, 
> plenty of it, and the power of the study will be such that even small 
> amounts of DIF can matter.  PISA has adopted an approach where items 
> are omitted if they have "substantial" DIF, the omissions can occur 
> international, nationally or at sub-national levels. We have not had 
> recommendations to split items accepted due to the reasons I mention 
> above.  Inevitably our decisions involve a subjective element.
>
>  
>
> The fundamental problem is that the world does not always behave like 
> the Rasch model and a whole host of issues need to be weighed up in 
> the context of a study such as PISA when deciding how to respond to 
> DIF.  Oh, and none of what I say is specific to the Rasch model, it 
> also holds for the generalised partial credit/2PL, all of PISA 2006 
> has been alternatively scaled using the generalised partial credit 
> model and the DIF results are no different.
>

I agree completely. And I do not agree with the sentiment that such 
models do not provide valid measurement, and I think that you should 
estimate persons based on these models if the Rasch model does not fit 
and one of the other models do fit. (I would prefer a Rasch model if it 
fits, of course)

>  
>
> (3)  I also agree with our comment 'The fit tests for the Rasch model 
> are not "weak in detecting model violations"'.  I can only assume 
> Margaret was referring to the standard infit/outfit measures.  
> Margaret has an analytic proof that these are indicators of the 
> consistency of the ICC slopes (ie they are discrimination indices) and 
> in general terms they aren't great as omnibus tests of item fit.  
> There are however many focussed fit tests that are very good at 
> exposing model violations.
>
>  
>
For this reason it is unfortunate that the test of fit of the PISA items 
to the Rasch model appears to be based primarily on infits (and 
discrimnination coefficients).

> Sorry for the long post.
>
>  
>
> Ray
>
>  
>
>  
>
>  
>
>  
>
> *From:* Svend Kreiner [mailto:S.Kreiner at biostat.ku.dk]
> *Sent:* Thursday, 18 February 2010 2:21 AM
> *To:* Margaret Wu
> *Cc:* 'Agustin Tristan'; rasch; Adams, Ray
> *Subject:* Re: [Rasch] complex sampling Rasch -- not sample free?
>
>  
>
> Dear Raschers.
>
> Just for the record: The fit tests for the Rasch model are not "weak 
> in detecting model violations". Quite the contrary, in fact. Those of 
> you who have taken part in the MBC workshops will remember our 
> discussions of this 'problem' and the suggestions by some that sample 
> sizes for Rasch analyses should never be too large (and sometimes even 
> be reduced to hide the fact that items do not fit the model).
>
> With sample sizes like they have in the PISA project, there are no 
> problems at all in detecting that PISA items do not fit the Rasch 
> model.  The country DIF problems could be easily solved by item 
> splitting across countries if they cared to do so. Since they have 
> decided not to do so, they are not estimating person and item 
> parameters of Rasch models at all and their 'measurements' are neither 
> valid, objective or sample free in the sense that we usually 
> understand these terms.
>
> Svend
>
>
>
>
> Margaret Wu skrev:
>
> Dear Agustin,
>
> When the data fit the Rasch model (e.g., you simulate some data sets 
> that fit the Rasch model), the item parameters will be "sample free", 
> in the sense that the item parameters will be the "same" (up to 
> measurement error) irrespective of which people (sub sample) you select.
>
>  
>
> If the data set does not fit the Rasch model, then the item parameters 
> may not be sample free. For example, we have found that girls 
> outperform boys by a great deal on reading texts that involve human 
> relationships, but not on reading texts that are scientific. If a 
> sample chosen consists of more girls than boys, the item parameters 
> may differ from a calibration where there are an equal number of girls 
> and boys. In this case, the calibrations are NOT sample free. For this 
> reason, in surveys like PISA, the calibration sample is carefully 
> selected (with sampling weights, etc) to represent the population, so 
> that the calibrated item parameters should represent those if the 
> whole population is tested.
>
>  
>
> As the Rasch model is a mathematical model, there is no guarantee that 
> any data set you collect will fit the model. While you can check for 
> model violations, typically the fit tests are weak in detecting model 
> violations. For these reasons, it will always be better to select your 
> calibration sample carefully, and not just pick any sample. This is a 
> precaution and it is a good practice.
>
>  
>
> As for whether "Rasch parameters are sample free or not", the answer 
> is YES, if the data fit the model. If the data do not fit the model, 
> by running the data through a Rasch model software program, you will 
> not have the sample free property. I hope this clears the confusion.
>
>  
>
> Margaret
>
>  
>
>  
>
> ------------------------------------------------------------------------
>
> *From:* Agustin Tristan [mailto:ici_kalt at yahoo.com]
> *Sent:* Wednesday, 17 February 2010 1:01 AM
> *To:* Rasch
> *Cc:* RayAdams; wu at edmeasurement.com.au <mailto:wu at edmeasurement.com.au>
> *Subject:* RE: [Rasch] complex sampling Rasch -- not sample free?
>
>  
>
> Hello!
>
> Margaret points out: "Rasch parameters are NOT sample free". I cannot 
> understand very well how
> "properly" drawn samples (i.e., with weights to represent countries) 
> SHOWS" this fact, certainly I need more explanattions of how properly 
> drawn samples can be a demonstration of that, but this point can be 
> regarded later.
>
> At this moment the most important point concerns what we have to say 
> when we talk about the Rasch model and sampling. In a previous message 
> I just used the words by Ray when he wrote:
>
> "if you useconditional maximum likelihood or a pairwise routine then 
> the item parameter estimates are sample free and so the sample design 
> does not influence the estimates.  If you use unconditional or 
> marginal maximum likelihood then "in theory" there may be some minor 
> effects, but I've never seen any evidence that it has any practical 
> consequence" , similar form of expression was used in several papers 
> by Wright using the common "sample free calibration".
>
> But probably it is not the correct way to say and I assume that Ray or 
> Ben were trying to say something in English an I'm incorrectly 
> translating or giving an undesirable interpretation into Spanish, or 
> probably I'm missing something important all those years.
>
> If you can help me, I think it could be an important point to define 
> from now on  the correct form to define or express this.
>
> Please let me know if it is correct to say:
>
> "Rasch model parameters are INDEED NOT sample free"?
>
> or just
>
> "Rasch model parameters MAY NOT BE sample free"
>
> How do I have to say that in correct English? I don't want to commit 
> more mistakes in the future, please.
>
> By the way, this will finish an eternal discussion in this listserve 
> and other forum where other people reject the Rasch model and we try 
> (incorrectly, as I did learn now) to defend that one of its beauties 
> is that it is sample free.
>
> Or probably we need a new definition of "sample free calibration"?
>
> Regards and thank you for your answer.
>
> Agustin Tristan
>
>  
>
>
>
> --- On *Mon, 2/15/10, wu at edmeasurement.com.au 
> <mailto:wu at edmeasurement.com.au> /<wu at edmeasurement.com.au> 
> <mailto:wu at edmeasurement.com.au>/* wrote:
>
>
> From: wu at edmeasurement.com.au <mailto:wu at edmeasurement.com.au> 
> <wu at edmeasurement.com.au> <mailto:wu at edmeasurement.com.au>
> Subject: RE: [Rasch] complex sampling Rasch
> To: "Adams, Ray" <adams at acer.edu.au> <mailto:adams at acer.edu.au>
> Cc: "Agustin Tristan" <ici_kalt at yahoo.com> 
> <mailto:ici_kalt at yahoo.com>, "rasch" <rasch at mailinglist.acer.edu.au> 
> <mailto:rasch at mailinglist.acer.edu.au>
> Date: Monday, February 15, 2010, 5:37 PM
>
> We need to be careful in saying that "As the Rasch model is sample
> free,...", because real data sets never fit the Rasch model. The PISA data
> set is no exception. (not the fault of the Rasch model, but the "fault" of
> the real data set).
>
> The fact that in PISA, the calibration had to use "properly" drawn samples
> (i.e., with weights to represent countries) shows that the item responses
> are NOT sample free (otherwise we can just take any sample). In fact,
> there are large DIF across countries, particularly between different
> language groups.
>
> One should always take the precaution and assume that the (real) data set
> is not sample free.
>
> Margaret
>
> > (a)  This is effectively what happens.  Item parameters in PISA are
> > estimate using a "calibration sample".  The calibration sample is a
> > subsample of 500 students from each OECD country.  The subsample is
> > drawn using weights so that schools and students are appropriately
> > represented, and then with 500 per country each country has the same
> > influence on the item parameter estimates.
> >
> >
> >
> > (b) PISA illustrates a potential for quite strong position effects.  To
> > control for this our test design, which has multiple test booklets,
> > makes sure that each item is located once in each of four locations:
> > first 30-minute cluster, second 30-minute cluster, third 30-minute
> > cluster and fourth 30-minute cluster.
> >
> >
> >
> > Ray
> >
> >
> >
> >
> >
> > From: Agustin Tristan [mailto:ici_kalt at yahoo.com 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=ici_kalt@yahoo.com>]
> > Sent: Tuesday, 16 February 2010 1:58 AM
> > To: rasch
> > Cc: Adams, Ray; Randy & Shelley MacIntosh; rasch
> > Subject: RE: [Rasch] complex sampling Rasch
> >
> >
> >
> > Hello! Thank your for the explanation concerng the sample design and the
> > Rasch item's parameters, it is useful because we are also interested to
> > use some ideas from the PISA project . In addition to Randy's question,
> > I wonder if some of the following aspects are considered as important
> > for the item and test calibration for the PISA project:
> >
> > a) Could it be convenient to get the item calibration for each
> > sub-sample  (schools) and after that to combine the calibrations using
> > the weight for each sample unit to obtain a "general" item calibration?
> >
> > b) As the Rasch model is sample free (or without a practical implication
> > as Ray did explain), can we use the item calibration of the anchor items
> > independently of the position of the items  in the test?
> >
> > Regards
> >
> > Agustin Tristan
> >
> >
> >
> >
> >
> >
> > --- On Sun, 2/14/10, Adams, Ray <adams at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=adams@acer.edu.au>> wrote:
> >
> >
> >     From: Adams, Ray <adams at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=adams@acer.edu.au>>
> >     Subject: RE: [Rasch] complex sampling Rasch
> >     To: "Randy & Shelley MacIntosh" <srmac at bluebottle.com 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=srmac@bluebottle.com>>, 
> "rasch"
> > <rasch at mailinglist.acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch@mailinglist.acer.edu.au>>
> >     Date: Sunday, February 14, 2010, 7:31 PM
> >
> >     Randy,
> >
> >     The answer is a bit complicated. Let's put fit aside and assume
> > the
> >     Rasch model and your data are compatible.  Then, it depends upon
> > the
> >     modelling assumptions you make and your estimation method.  If
> > you use
> >     conditional maximum likelihood or a pairwise routine then the
> > item
> >     parameter estimates are sample free and so the sample design
> > does not
> >     influence the estimates.  If you use unconditional or marginal
> > maximum
> >     likelihood then "in theory" there may be some minor effects, but
> > I've
> >     never seen any evidence that it has any practical consequence.
> >
> >     In PISA we do not take the complex sample design into account
> > when
> >     estimating item parameters.
> >
> >     Ray
> >
> >
> >     -----Original Message-----
> >     From: rasch-bounces at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au>
> > <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au
> >>  [mailto:rasch-bounces at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au>
> > <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au
> >> ] On
> >     Behalf Of Randy & Shelley MacIntosh
> >     Sent: Monday, 15 February 2010 7:42 AM
> >     To: rasch
> >     Subject: [Rasch] complex sampling Rasch
> >
> >     I am interested in applying the Rasch model to data from a
> > complex
> >     sample design.
> >     I was wondering if this has been done?
> >     For example, is the nature of the PISA sample design explicitly
> > taken
> >     into account when the Rasch estimates are produced?
> >
> >
> >     Thanks,
> >     Randy MacIntosh
> >
> >
> >
> >     -------------------------------------------------
> >     Please consider the environment before you print
> >     _______________________________________________
> >     Rasch mailing list
> >     Rasch at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> > <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> >     https://mailinglist.acer.edu.au/mailman/listinfo/rasch
> >
> >     -------------------------------------------------
> >     Please consider the environment before you print
> >     _______________________________________________
> >     Rasch mailing list
> >     Rasch at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> > <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> >     https://mailinglist.acer.edu.au/mailman/listinfo/rasch
> >
> >
> >
> >
> > -------------------------------------------------
> > Please consider the environment before you print
> > _______________________________________________
> > Rasch mailing list
> > Rasch at acer.edu.au 
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> > https://mailinglist.acer.edu.au/mailman/listinfo/rasch
> >
>
>
>  
>
> ------------------------------------------------- Please consider the 
> environment before you print
>
>  
>
>
> ------------------------------------------------------------------------
>
>
>
>
>   
>  
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au <mailto:Rasch at acer.edu.au>
> https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>   
>
>
>
> -- 
> Svend Kreiner
> Professor
> Department of Biostatistics  
> University of Copenhagen 
>  
> Øster farimagsgade 5, entr. B
> P.O. Box 2099
> DK-1014 Copenhagen K, Denmark
>  
> Email: S.Kreiner at biostat.ku.dk <mailto:S.Kreiner at biostat.ku.dk>
> Phone: (+45) 35 32 75 97     
>  
> Fax: (+45) 35 32 79 07
> ------------------------------------------------- Please consider the 
> environment before you print 

-- 
Svend Kreiner
Professor
Department of Biostatistics  
University of Copenhagen 

Øster farimagsgade 5, entr. B
P.O. Box 2099
DK-1014 Copenhagen K, Denmark

Email: S.Kreiner at biostat.ku.dk
Phone: (+45) 35 32 75 97     

Fax: (+45) 35 32 79 07



-------------------------------------------------
Please consider the environment before you print
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20100301/5521b24b/attachment.html 


More information about the Rasch mailing list