[Rasch] sample size

iasonas liasonas at cytanet.com.cy
Fri Feb 26 03:25:47 EST 2010


Thank you very much for these ideas. I have a question though. You said,
"That is, even if each individual is not measured well, you can still have
reliable population level results if the sample is large and well chosen".
Do you mean that, even if the error of measurement for individuals is large,
this error should be random, therefore, overall, (if the sample is large and
representative), we should be able to estimate the population parameters
correctly? If yes, then we agree, although usually, the politicians 'force'
us to report individual measures as well. Is anyone aware of any methods to
'balance' the individual errors and the sample/population errors? Any
references?

Jason

-----Original Message-----
From: Margaret Wu [mailto:wu at edmeasurement.com.au] 
Sent: Thursday, February 25, 2010 1:36 PM
To: 'iasonas'; rasch at acer.edu.au
Subject: RE: [Rasch] sample size

Dear Jason,
The determination of sample size is related to the conclusion you want to
draw from your assessment. 

If you want to make inference beyond the students/raters/essays involved in
your study, then you need to consider not only sample size, but the
representativeness of the sample. This is independent to whether you use IRT
(Rasch model included) or other statistical techniques.

For example, if I want to know the average height of 10 particular people, I
can just measure their heights and take the average. If I want to use the
height of the 10 people to INFER the average height of a larger group, then
I need to make sure the sample is representative, AND that the sample size
is SUFFIICIENT so that the errors (from both measurement and sampling) are
small enough for me to make "useful" conclusions.

In your case, if you want to know if the rater judgements are sufficiently
close within the study you conducted, then you need to make sure the
measurement errors are sufficiently small for you to detect a magnitude of
difference that is of interest. Suppose you will only be concerned if the
rater difference is greater than 0.2 logits. If the measurement error is of
the order of 0.2 logits or greater, then there is no way for you to test
whether the raters are different.

Further, if you want to use your results to support a scheme in the future
that such ratings will have reliable results, then based on just a few
raters, it will be difficult to generalize your results to the behaviour of
other raters.

In general, to reduce measurement error, you need more measurements per
individual. To reduce sampling error, you need more participants (whether
students and/or raters). To balance measurement and sampling error, we often
find that sampling is more important than measurement. That is, even if each
individual is not measured well, you can still have reliable population
level results if the sample is large and well chosen.

In summary, you cannot determine an adequate sample size without first
clearly identifying the kind of conclusions you would like to make in terms
of the comparisons you want to make, and in terms of the inference you'd
like to extend your results to.

Once I asked a student what she wanted to get out of her study of raters.
She said she wanted to know "the truth". I told her I couldn't help her in
that case. 

As a rule, I always tell my students to collect as much data as they can. A
study is often not very useful without making generalizations of the
results.

I hope this helps a little.
Margaret

 


-----Original Message-----
From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf
Of iasonas
Sent: Wednesday, 24 February 2010 5:40 PM
To: rasch at acer.edu.au
Subject: RE: [Rasch] sample size

 Dear all,
I have an MSc student who collectd data on two groups of students. Each
group consists of 20 students. Each student was measured on three different
essays. Each essay was evaluated twice: once when the essay was in a draft
state, and once when it was finished. Each time an essay was evaluated, it
was evaluated by three persons: the teacher, an external teacher and a
student (peer-evaluation). This is a great opportunity of Many-Facets Rasch
model. I have many years of intense experience with Rasch models, and I do
NOT worry about the sample size; I am sure that the parameters will be
estimated with a realistic/practical precision (provided the data fit the
model - we havent yet finished data collection). However, would you also all
agree that I should not worry about the sample size? The student is
reluctant to collect more data for a number of reasons. Should I give her
the green light to go on to collect data from only 20 students per group, or
does any one of you worry that the data will be too "thin"?

Thank you
Jason 


-------------------------------------------------
Please consider the environment before you print
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
https://mailinglist.acer.edu.au/mailman/listinfo/rasch

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.733 / Virus Database: 271.1.1/2707 - Release Date: 02/24/10
21:34:00


-------------------------------------------------
Please consider the environment before you print



More information about the Rasch mailing list