[Rasch] complex sampling Rasch -- not sample free?

Stephen Humphry stephen.humphry at uwa.edu.au
Wed Feb 17 22:29:17 EST 2010


Hi all,

I think Margaret's explanation clarifies the point well.

I have a couple of minor things to add, but I stress they are minor in the general scheme of things. I take what Margaret said to mean that the parameter *estsimates* obtained from calibrations are not "sample free" in the presence of DIF. So for clarity, if the data fit the model then the parameter *estimates* are "sample free". That is, I take item parameters to be short-hand for parameter estimates as implied by Margaret's reference to the calibrations that produce estimates.

The other thing is that, assuming Margaret means what I think she does, in the case of DIF described below, using a single estimate is not the only available strategy. If the DIF is uniform then the items can be 'split' and different estimates obtained for different groups. Having done so, it is possible for data to fit the model, in which case the item parameter estimates would still be independent of the person parameter estimates and, therefore, independent of the group (sample) of persons involved in the estimations. This is what "sample free" means if one taks samples and sampling to be primary epistemological concepts. Rasch definitively did not take these things to be primary, although he stressed the distinction between parameters and their estimates.

I agree, nevertheless, that it is precaution and good practice to select a sample carefully. I agree with this whether (a) the aim is to maximise the capacity to detect DIF that is materially relevant in the situation or (b) for whatever reason it is difficult or impractical to split items (or equivalently to parameterize and estimate the difference between item locations for different groups).

Lastly, although I fully agree that some fit tests are weak in detecting model violations, it depends on the nature of misfit and the information available. Also, where possible I think it is preferable to use multiple tests of fit and graphical inspection of the data.

Regards,

Steve

________________________________
From: rasch-bounces at acer.edu.au [rasch-bounces at acer.edu.au] On Behalf Of Margaret Wu [wu at edmeasurement.com.au]
Sent: Wednesday, 17 February 2010 6:29 PM
To: 'Agustin Tristan'; 'Rasch'
Cc: 'RayAdams'
Subject: RE: [Rasch] complex sampling Rasch -- not sample free?

Dear Agustin,
When the data fit the Rasch model (e.g., you simulate some data sets that fit the Rasch model), the item parameters will be “sample free”, in the sense that the item parameters will be the “same” (up to measurement error) irrespective of which people (sub sample) you select.

If the data set does not fit the Rasch model, then the item parameters may not be sample free. For example, we have found that girls outperform boys by a great deal on reading texts that involve human relationships, but not on reading texts that are scientific. If a sample chosen consists of more girls than boys, the item parameters may differ from a calibration where there are an equal number of girls and boys. In this case, the calibrations are NOT sample free. For this reason, in surveys like PISA, the calibration sample is carefully selected (with sampling weights, etc) to represent the population, so that the calibrated item parameters should represent those if the whole population is tested.

As the Rasch model is a mathematical model, there is no guarantee that any data set you collect will fit the model. While you can check for model violations, typically the fit tests are weak in detecting model violations. For these reasons, it will always be better to select your calibration sample carefully, and not just pick any sample. This is a precaution and it is a good practice.

As for whether "Rasch parameters are sample free or not", the answer is YES, if the data fit the model. If the data do not fit the model, by running the data through a Rasch model software program, you will not have the sample free property. I hope this clears the confusion.

Margaret


________________________________
From: Agustin Tristan [mailto:ici_kalt at yahoo.com]
Sent: Wednesday, 17 February 2010 1:01 AM
To: Rasch
Cc: RayAdams; wu at edmeasurement.com.au
Subject: RE: [Rasch] complex sampling Rasch -- not sample free?

Hello!
Margaret points out: "Rasch parameters are NOT sample free". I cannot understand very well how
"properly" drawn samples (i.e., with weights to represent countries) SHOWS" this fact, certainly I need more explanattions of how properly drawn samples can be a demonstration of that, but this point can be regarded later.
At this moment the most important point concerns what we have to say when we talk about the Rasch model and sampling. In a previous message I just used the words by Ray when he wrote:
"if you useconditional maximum likelihood or a pairwise routine then the item parameter estimates are sample free and so the sample design does not influence the estimates.  If you use unconditional or marginal maximum likelihood then "in theory" there may be some minor effects, but I've never seen any evidence that it has any practical consequence" , similar form of expression was used in several papers by Wright using the common "sample free calibration".
But probably it is not the correct way to say and I assume that Ray or Ben were trying to say something in English an I'm incorrectly translating or giving an undesirable interpretation into Spanish, or probably I'm missing something important all those years.
If you can help me, I think it could be an important point to define from now on  the correct form to define or express this.
Please let me know if it is correct to say:
"Rasch model parameters are INDEED NOT sample free"?
or just
"Rasch model parameters MAY NOT BE sample free"
How do I have to say that in correct English? I don't want to commit more mistakes in the future, please.
By the way, this will finish an eternal discussion in this listserve and other forum where other people reject the Rasch model and we try (incorrectly, as I did learn now) to defend that one of its beauties is that it is sample free.
Or probably we need a new definition of "sample free calibration"?
Regards and thank you for your answer.
Agustin Tristan



--- On Mon, 2/15/10, wu at edmeasurement.com.au <wu at edmeasurement.com.au> wrote:

From: wu at edmeasurement.com.au <wu at edmeasurement.com.au>
Subject: RE: [Rasch] complex sampling Rasch
To: "Adams, Ray" <adams at acer.edu.au>
Cc: "Agustin Tristan" <ici_kalt at yahoo.com>, "rasch" <rasch at mailinglist.acer.edu.au>
Date: Monday, February 15, 2010, 5:37 PM
We need to be careful in saying that "As the Rasch model is sample
free,...", because real data sets never fit the Rasch model. The PISA data
set is no exception. (not the fault of the Rasch model, but the "fault" of
the real data set).

The fact that in PISA, the calibration had to use "properly" drawn samples
(i.e., with weights to represent countries) shows that the item responses
are NOT sample free (otherwise we can just take any sample). In fact,
there are large DIF across countries, particularly between different
language groups.

One should always take the precaution and assume that the (real) data set
is not sample free.

Margaret

> (a)  This is effectively what happens.  Item parameters in PISA are
> estimate using a "calibration sample".  The calibration sample is a
> subsample of 500 students from each OECD country.  The subsample is
> drawn using weights so that schools and students are appropriately
> represented, and then with 500 per country each country has the same
> influence on the item parameter estimates.
>
>
>
> (b) PISA illustrates a potential for quite strong position effects.  To
> control for this our test design, which has multiple test booklets,
> makes sure that each item is located once in each of four locations:
> first 30-minute cluster, second 30-minute cluster, third 30-minute
> cluster and fourth 30-minute cluster.
>
>
>
> Ray
>
>
>
>
>
> From: Agustin Tristan [mailto:ici_kalt at yahoo.com<http://us.mc1115.mail.yahoo.com/mc/compose?to=ici_kalt@yahoo.com>]
> Sent: Tuesday, 16 February 2010 1:58 AM
> To: rasch
> Cc: Adams, Ray; Randy & Shelley MacIntosh; rasch
> Subject: RE: [Rasch] complex sampling Rasch
>
>
>
> Hello! Thank your for the explanation concerng the sample design and the
> Rasch item's parameters, it is useful because we are also interested to
> use some ideas from the PISA project . In addition to Randy's question,
> I wonder if some of the following aspects are considered as important
> for the item and test calibration for the PISA project:
>
> a) Could it be convenient to get the item calibration for each
> sub-sample  (schools) and after that to combine the calibrations using
> the weight for each sample unit to obtain a "general" item calibration?
>
> b) As the Rasch model is sample free (or without a practical implication
> as Ray did explain), can we use the item calibration of the anchor items
> independently of the position of the items  in the test?
>
> Regards
>
> Agustin Tristan
>
>
>
>
>
>
> --- On Sun, 2/14/10, Adams, Ray <adams at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=adams@acer.edu.au>> wrote:
>
>
>     From: Adams, Ray <adams at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=adams@acer.edu.au>>
>     Subject: RE: [Rasch] complex sampling Rasch
>     To: "Randy & Shelley MacIntosh" <srmac at bluebottle.com<http://us.mc1115.mail.yahoo.com/mc/compose?to=srmac@bluebottle.com>>, "rasch"
> <rasch at mailinglist.acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch@mailinglist.acer.edu.au>>
>     Date: Sunday, February 14, 2010, 7:31 PM
>
>     Randy,
>
>     The answer is a bit complicated. Let's put fit aside and assume
> the
>     Rasch model and your data are compatible.  Then, it depends upon
> the
>     modelling assumptions you make and your estimation method.  If
> you use
>     conditional maximum likelihood or a pairwise routine then the
> item
>     parameter estimates are sample free and so the sample design
> does not
>     influence the estimates.  If you use unconditional or marginal
> maximum
>     likelihood then "in theory" there may be some minor effects, but
> I've
>     never seen any evidence that it has any practical consequence.
>
>     In PISA we do not take the complex sample design into account
> when
>     estimating item parameters.
>
>     Ray
>
>
>     -----Original Message-----
>     From: rasch-bounces at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au>
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au
>>  [mailto:rasch-bounces at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au>
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=rasch-bounces@acer.edu.au
>> ] On
>     Behalf Of Randy & Shelley MacIntosh
>     Sent: Monday, 15 February 2010 7:42 AM
>     To: rasch
>     Subject: [Rasch] complex sampling Rasch
>
>     I am interested in applying the Rasch model to data from a
> complex
>     sample design.
>     I was wondering if this has been done?
>     For example, is the nature of the PISA sample design explicitly
> taken
>     into account when the Rasch estimates are produced?
>
>
>     Thanks,
>     Randy MacIntosh
>
>
>
>     -------------------------------------------------
>     Please consider the environment before you print
>     _______________________________________________
>     Rasch mailing list
>     Rasch at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
>     https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
>     -------------------------------------------------
>     Please consider the environment before you print
>     _______________________________________________
>     Rasch mailing list
>     Rasch at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> <http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
>     https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>
>
>
>
> -------------------------------------------------
> Please consider the environment before you print
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
> https://mailinglist.acer.edu.au/mailman/listinfo/rasch
>



-------------------------------------------------

Please consider the environment before you print

-------------------------------------------------
Please consider the environment before you print



More information about the Rasch mailing list