[Rasch] complex sampling Rasch -- not sample free?
Agustin Tristan
ici_kalt at yahoo.com
Thu Feb 18 04:01:45 EST 2010
Thank you Margaret, Mike, Steve and Svend for the explanations.
It seems that the concept "sample free" is clear and not different to what many of us used before. It is clear the meaning also for DIF applications.
Svend points two important things:
1) "...MBC workshops will remember our discussions of this 'problem' and the suggestions by some that sample sizes for Rasch analyses should never be too large (and sometimes even be reduced to hide the fact that items do not fit the model)."
It is again the problem on definition of "too large", or "reduced to what amount"?
Rasch model pretends to provide a scale and a measure unit, but we cannot have an acceptable value for large and reduced?
2) Svend wrote:
"Since they have decided not to do so, they are not estimating person and item parameters of Rasch models at all and their 'measurements' are neither valid, objective or sample free in the sense that we usually understand these terms."
When you way "they", do you mean the countries interested on DIF or the managers of the PISA project?
Regards
Agustin
--- On Wed, 2/17/10, Svend Kreiner <S.Kreiner at biostat.ku.dk> wrote:
From: Svend Kreiner <S.Kreiner at biostat.ku.dk>
Subject: Re: [Rasch] complex sampling Rasch -- not sample free?
To: "Margaret Wu" <wu at edmeasurement.com.au>
Cc: "'Agustin Tristan'" <ici_kalt at yahoo.com>, "'Rasch'" <rasch at acer.edu.au>, "'RayAdams'" <adams at acer.edu.au>
Date: Wednesday, February 17, 2010, 9:21 AM
Dear Raschers.
Just for the record: The fit tests for the Rasch model are not "weak in detecting model violations". Quite the contrary, in fact. Those of you who have taken part in the MBC workshops will remember our discussions of this 'problem' and the suggestions by some that sample sizes for Rasch analyses should never be too large (and sometimes even be reduced to hide the fact that items do not fit the model).
With sample sizes like they have in the PISA project, there are no problems at all in detecting that PISA items do not fit the Rasch model. The country DIF problems could be easily solved by item splitting across countries if they cared to do so. Since they have decided not to do so, they are not estimating person and item parameters of Rasch models at all and their 'measurements' are neither valid, objective or sample free in the sense that we usually understand these terms.
Svend
Margaret Wu skrev:
Dear Agustin,
When the data fit the Rasch model (e.g., you simulate some data sets that fit the Rasch model), the item parameters will be “sample free”, in the sense that the item parameters will be the “same” (up to measurement error) irrespective of which people (sub sample) you select.
If the data set does not fit the Rasch model, then the item parameters may not be sample free. For example, we have found that girls outperform boys by a great deal on reading texts that involve human relationships, but not on reading texts that are scientific. If a sample chosen consists of more girls than boys, the item parameters may differ from a calibration where there are an equal number of girls and boys. In this case, the calibrations are NOT sample free. For this reason, in surveys like PISA , the calibration sample is carefully selected (with sampling weights, etc) to represent the population, so that the calibrated item parameters should represent those if the whole population is tested.
As the Rasch model is a mathematical model, there is no guarantee that any data set you collect will fit the model. While you can check for model violations, typically the fit tests are weak in detecting model violations. For these reasons, it will always be better to select your calibration sample carefully, and not just pick any sample. This is a precaution and it is a good practice.
As for whether "Rasch parameters are sample free or not", the answer is YES, if the data fit the model. If the data do not fit the model, by running the data through a Rasch model software program, you will not have the sample free property. I hope this clears the confusion.
Margaret
From: Agustin Tristan [mailto:ici_kalt at yahoo.com]
Sent: Wednesday, 17 February 2010 1:01 AM
To: Rasch
Cc: RayAdams; wu at edmeasurement.com.au
Subject: RE: [Rasch] complex sampling Rasch -- not sample free?
Hello!
Margaret points out: "Rasch parameters are NOT sample free". I cannot understand very well how
"properly" drawn samples (i.e., with weights to represent countries) SHOWS" this fact, certainly I need more explanattions of how properly drawn samples can be a demonstration of that, but this point can be regarded later.
At this moment the most important point concerns what we have to say when we talk about the Rasch model and sampling. In a previous message I just used the words by Ray when he wrote:
"if you useconditional maximum likelihood or a pairwise routine then the item parameter estimates are sample free and so the sample design does not influence the estimates. If you use unconditional or marginal maximum likelihood then "in theory" there may be some minor effects, but I've never seen any evidence that it has any practical consequence" , similar form of expression was used in several papers by Wright using the common "sample free calibration".
But probably it is not the correct way to say and I assume that Ray or Ben were trying to say something in English an I'm incorrectly translating or giving an undesirable interpretation into Spanish, or probably I'm missing something important all those years.
If you can help me, I think it could be an important point to define from now on the correct form to define or express this.
Please let me know if it is correct to say:
"Rasch model parameters are INDEED NOT sample free"?
or just
"Rasch model parameters MAY NOT BE sample free"
How do I have to say that in correct English? I don't want to commit more mistakes in the future, please.
By the way, this will finish an eternal discussion in this listserve and other forum where other people reject the Rasch model and we try (incorrectly, as I did learn now) to defend that one of its beauties is that it is sample free.
Or probably we need a new definition of "sample free calibration"?
Regards and thank you for your answer.
Agustin Tristan
--- On Mon, 2/15/10, wu at edmeasurement.com.au <wu at edmeasurement.com.au> wrote:
From: wu at edmeasurement.com.au <wu at edmeasurement.com.au>
Subject: RE: [Rasch] complex sampling Rasch
To: "Adams, Ray" <adams at acer.edu.au>
Cc: "Agustin Tristan" <ici_kalt at yahoo.com>, "rasch" <rasch at mailinglist.acer.edu.au>
Date: Monday, February 15, 2010, 5:37 PM
We need to be careful in saying that "As the Rasch model is sample
free,...", because real data sets never fit the Rasch model. The PISA data
set is no exception. (not the fault of the Rasch model, but the "fault" of
the real data set).
The fact that in PISA , the calibration had to use "properly" drawn samples
(i.e., with weights to represent countries) shows that the item responses
are NOT sample free (otherwise we can just take any sample). In fact,
there are large DIF across countries, particularly between different
language groups.
One should always take the precaution and assume that the (real) data set
is not sample free.
Margaret
> (a) This is effectively what happens. Item parameters in PISA are
> estimate using a "calibration sample". The calibration sample is a
> subsample of 500 students from each OECD country. The subsample is
> drawn using weights so that schools and students are appropriately
> represented, and then with 500 per country each country has the same
> influence on the item parameter estimates.
>
>
>
> (b) PISA illustrates a potential for quite strong position effects. To
> control for this our test design, which has multiple test booklets,
> makes sure that each item is located once in each of four locations:
> first 30-minute cluster, second 30-minute cluster, third 30-minute
> cluster and fourth 30-minute cluster.
>
>
>
> Ray
>
>
>
>
>
> From: Agustin Tristan [mailto:ici_kalt at yahoo.com]
> Sent: Tuesday, 16 February 2010 1:58 AM
> To: rasch
> Cc: Adams, Ray; Randy & Shelley MacIntosh; rasch
> Subject: RE: [Rasch] complex sampling Rasch
>
>
>
> Hello! Thank your for the explanation concerng the sample design and the
> Rasch item's parameters, it is useful because we are also interested to
> use some ideas from the PISA project . In addition to Randy's question,
> I wonder if some of the following aspects are considered as important
> for the item and test calibration for the PISA project:
>
> a) Could it be convenient to get the item calibration for each
> sub-sample (schools) and after that to combine the calibrations using
> the weight for each sample unit to obtain a "general" item calibration?
>
> b) As the Rasch model is sample free (or without a practical implication
> as Ray did explain), can we use the item calibration of the anchor items
> independently of the position of the items in the test?
>
> Regards
>
> Agustin Tristan
>
>
>
>
>
>
> --- On Sun, 2/14/10, Adams , Ray <adams at acer.edu.au> wrote:
>
>
> From: Adams, Ray <adams at acer.edu.au>
> Subject: RE: [Rasch] complex sampling Rasch
> To: "Randy & Shelley MacIntosh" <srmac at bluebottle.com>, "rasch"
> <rasch at mailinglist.acer.edu.au>
> Date: Sunday, February 14, 2010, 7:31 PM
>
> Randy,
>
> The answer is a bit complicated. Let's put fit aside and assume
> the
> Rasch model and your data are compatible. Then, it depends upon
> the
> modelling assumptions you make and your estimation method. If
> you use
> conditional maximum likelihood or a pairwise routine then the
> item
> parameter estimates are sample free and so the sample design
> does not
> influence the estimates. If you use unconditional or marginal
> maximum
> likelihood then "in theory" there may be some minor effects, but
> I've
> never seen any evidence that it has any practical consequence.
>
> In PISA we do not take the complex sample design into account
> when
> estimating item parameters.
>
> Ray
>
>
> -----Original Message-----
> From: rasch-bounces at acer.edu.au
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au
>> [mailto:rasch-bounces at acer.edu.au
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au
>> ] On
> Behalf Of Randy & Shelley MacIntosh
> Sent: Monday, 15 February 2010 7:42 AM
> To: rasch
> Subject: [Rasch] complex sampling Rasch
>
> I am interested in applying the Rasch model to data from a
> complex
> sample design.
> I was wondering if this has been done?
> For example, is the nature of the PISA sample design explicitly
> taken
> into account when the Rasch estimates are produced?
>
>
> Thanks,
> Randy MacIntosh
>
>
>
> -------------------------------------------------
> Please consider the environment before you print
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
> -------------------------------------------------
> Please consider the environment before you print
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
>
>
>
> -------------------------------------------------
> Please consider the environment before you print
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
------------------------------------------------- Please consider the environment before you print
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
https://mailinglist.acer.edu.au/mailman/listinfo/rasch
--
Svend Kreiner
Professor
Department of Biostatistics
University of Copenhagen
Øster farimagsgade 5, entr. B
P.O. Box 2099
DK-1014 Copenhagen K, Denmark
Email: S.Kreiner at biostat.ku.dk
Phone: (+45) 35 32 75 97
Fax: (+45) 35 32 79 07
-------------------------------------------------
Please consider the environment before you print
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20100217/9da26e56/attachment.html
More information about the Rasch
mailing list