# [Rasch] complex sampling Rasch -- item position matters?

Agustin Tristan ici_kalt at yahoo.com
Tue Feb 23 08:09:55 EST 2010

```Thank you Ray, I couldn't find the specification concerning the position of the clusters in the PISA design in the technical report. I shall appreciate if you can give me a more precise hint to find it in the document.
But what I've seen is that clusters are intact, but I would like to know if the positions on the booklet  of the cluster and the item are important.
Is it possible that every item of the cluster appears in different pages on each booklet?
i.e. The first anchor item may occupy page 13 (cluster1), page 4 (cluster2), page 8 (cluster3), page 21 (cluster4) or is it needed that it is in the same page in the four clusters?
It is clear that all the items of the cluster are intact and then will follow the page number according to position of the first item.
If position matters all clusters reduce to the first one and I'm affraid CAT cannot be possible as we shall not be free to move the positions of the items in different tests.

Agustin Tristan

Subject: RE: [Rasch] complex sampling Rasch
To: "Agustin Tristan" <ici_kalt at yahoo.com>
Date: Tuesday, February 16, 2010, 3:39 PM

They occupy the same position in each.  Clusters are kept intact.  You should see the pisa design in the study’s technical reports.

From: Agustin Tristan [mailto:ici_kalt at yahoo.com]
Sent: Wednesday, 17 February 2010 3:18 AM
To: rasch
Subject: RE: [Rasch] complex sampling Rasch

Hello Ray, you wrote:

"To control for this our test design, which has multiple test booklets, makes sure that each item is located once in each of four locations: first 30-minute cluster, second 30-minute cluster, third 30-minute cluster and fourth 30-minute cluster."

Does the position of the anchor item in each cluster matters? i.e. The first anchor item may occupy positions 13 (cluster1), 4 (cluster2), 8 (cluster3), 21 (cluster4) or is it needed that it occupies position 13 in the four clusters?

It is clear that the position must be inside its own tesltlet, but as items must be independent I think they may be interchaged in position inside the testlet.

I hope position will not be important on the item calibration, but please let me know.

Thank you.

Agustin Tristan

Subject: RE: [Rasch] complex sampling Rasch
To: "Agustin Tristan" <ici_kalt at yahoo.com>, "rasch" <rasch at mailinglist.acer.edu.au>
Cc: "Randy & Shelley MacIntosh" <srmac at bluebottle.com>
Date: Monday, February 15, 2010, 4:56 PM

(a)  This is effectively what happens.  Item parameters in PISA are estimate using a “calibration sample”.  The calibration sample is a subsample of 500 students from each OECD country.  The subsample is drawn using weights so that schools and students are appropriately represented, and then with 500 per country each country has the same influence on the item parameter estimates.

(b) PISA illustrates a potential for quite strong position effects.  To control for this our test design, which has multiple test booklets, makes sure that each item is located once in each of four locations: first 30-minute cluster, second 30-minute cluster, third 30-minute cluster and fourth 30-minute cluster.

Ray

From: Agustin Tristan [mailto:ici_kalt at yahoo.com]
Sent: Tuesday, 16 February 2010 1:58 AM
To: rasch
Cc: Adams, Ray; Randy & Shelley MacIntosh; rasch
Subject: RE: [Rasch] complex sampling Rasch

Hello! Thank your for the explanation concerng the sample design and the Rasch item's parameters, it is useful because we are also interested to use some ideas from the PISA project . In addition to Randy's question, I wonder if some of the following aspects are considered as important for the item and test calibration for the PISA project:

a) Could it be convenient to get the item calibration for each sub-sample  (schools) and after that to combine the calibrations using the weight for each sample unit to obtain a "general" item calibration?

b) As the Rasch model is sample free (or without a practical implication as Ray did explain), can we use the item calibration of the anchor items independently of the position of the items  in the test?

Regards

Agustin Tristan

Subject: RE: [Rasch] complex sampling Rasch
To: "Randy & Shelley MacIntosh" <srmac at bluebottle.com>, "rasch" <rasch at mailinglist.acer.edu.au>
Date: Sunday, February 14, 2010, 7:31 PM

Randy,

The answer is a bit complicated. Let's put fit aside and assume the
Rasch model and your data are compatible.  Then, it depends upon the
modelling assumptions you make and your estimation method.  If you use
conditional maximum likelihood or a pairwise routine then the item
parameter estimates are sample free and so the sample design does not
influence the estimates.  If you use unconditional or marginal maximum
likelihood then "in theory" there may be some minor effects, but I've
never seen any evidence that it has any practical consequence.

In PISA we do not take the complex sample design into account when
estimating item parameters.

Ray

-----Original Message-----
From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of Randy & Shelley MacIntosh
Sent: Monday, 15 February 2010 7:42 AM
To: rasch
Subject: [Rasch] complex sampling Rasch

I am interested in applying the Rasch model to data from a complex
sample design.
I was wondering if this has been done?
For example, is the nature of the PISA sample design explicitly taken
into account when the Rasch estimates are produced?

Thanks,
Randy MacIntosh

-------------------------------------------------
Please consider the environment before you print
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
https://mailinglist.acer.edu.au/mailman/listinfo/rasch

-------------------------------------------------
Please consider the environment before you print
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
https://mailinglist.acer.edu.au/mailman/listinfo/rasch

-------------------------------------------------
Please consider the environment before you print
-------------------------------------------------
Please consider the environment before you print

-------------------------------------------------
Please consider the environment before you print
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20100222/6e50dab7/attachment.html
```