[Rasch] CAT Questions

Timothy Pelton tpelton at uvic.ca
Sat Jan 30 18:28:16 EST 2010


Yes, you will require a calibrating sample to set up the item  parameters for the CAT.

Yes, the CAT can reduce the time/number of questions required to achieve a similar level of accuracy.   Or on the flip side, you can ask more informative questions (i.e., at an ideal level of difficulty for the examinee) and reduce the error estimate.

Also, if your test specification has multiple cells (e.g., more than one topic and cognitive level) - then you will need to build a pool of items that covers all of the cells appropriately and ensure that the CAT administration program pulls a minimum number of items from each appropriate cell.  Otherwise you will have difficulty in claiming it is valid.

Tim

-------------
Tim Pelton
Associate Professor
Faculty of Education
University of Victoria
office: 250-721-7803
fax: 250-721-7598
________________________________________
From: rasch-bounces at acer.edu.au [rasch-bounces at acer.edu.au] On Behalf Of ranganaths [ranganath.s at excelindia.com]
Sent: Friday, January 29, 2010 10:58 PM
To: 'Agustin Tristan'; 'Rasch'
Subject: RE: [Rasch] CAT Questions

Hello,

            Thanks for the reply. I still have a question on CAT. Now with all the mail Interactions I have understood that there is no need to pre-administer the tests to rate Examinees. I have read that both the Item parameters and the abilities of the examinees can be simultaneously obtained using Birnbaum paradigm.  I hope the same is used in Winstep to ascertain the abilities as well as the item parameters.
I have read in a magazine futureGov that it reduces the time of test based on the ability of the examinee. I assumed that it might start from some basic item and go till the point till his ability is ascertained. If this is one of the advantages of the CAT does it require computing the item parameters in prior? If it is required to compute the item parameters in prior, it require the test to be administered to sample examinees before doing that. Is this true.

Regards,
Ranganath S

________________________________
From: Agustin Tristan [mailto:ici_kalt at yahoo.com]
Sent: Friday, January 29, 2010 9:05 PM
To: ranganaths; Rasch
Subject: RE: [Rasch] CAT Questions

Hello, I can explain because I have worked in the translation of Baker´s book and the adaptation of software into Spanish. The quote concerning the limitations is clearly related to the program and the algorithms included in the software by Baker provided with the book, but not to the Rasch model itself. You may use a software like Winsteps without problems with more than several thousands of persons!
Regards
Agustin Tristan

INSTITUTO DE EVALUACION E INGENIERIA AVANZADA.
Ave. Cordillera Occidental  No. 635
Colonia  Lomas 4ª San Luis Potosí, San Luis Potosí.
C.P.  78216   MEXICO
 (52) (444) 8 25 50 76
 (52) (444) 8 25 50 77
 (52) (444) 8 25 50 78
web page (in Spanish): http://www.ieia.com.mx<http://www.ieia.com.mx/>
web page (in English) http://www.ieesa-kalt.com/English/Frames_sp_pro.html


--- On Fri, 1/29/10, ranganaths <ranganath.s at excelindia.com> wrote:

From: ranganaths <ranganath.s at excelindia.com>
Subject: RE: [Rasch] CAT Questions
To: "'Mike Linacre (RMT)'" <rmt at rasch.org>
Cc: rasch at acer.edu.au
Date: Friday, January 29, 2010, 5:28 AM
Hello,

            Thanks for the response. I am referring to the book THE BASICS OF ITEM RESPONSE THEORY by Frank Baker. In this book he says. I quote
“There are three different item characteristic curve models to choose from and several different ways to implement the Birnbaum paradigm. From these, the author has chosen to present the approach based upon the one-parameter logistic (Rasch) model as implemented by Benjamin Wright and his coworkers in the BICAL computer program. Under this model, each item has only one parameter to be estimated. The procedures work well with small numbers of test items and small numbers of examinees.”

            We have a product which administers tests to thousands of people and having hundreds of Items in the test. The above quote gives a warning for such kind of applications. I wish to know If this is true which model should be used for the kind of tests we are administering? Thanks once again for the response.


Regards,
Ranganath S
________________________________
From: Mike Linacre (RMT) [mailto:rmt at rasch.org]
Sent: Friday, January 29, 2010 2:29 AM
To: ranganaths
Subject: RE: [Rasch] CAT Questions

Thank you for your further questions, Ranganath.

1. and 2. - No and No.  No need to pre-calibrate. No need to pre-administer. See, for instance, www.rasch.org/rmt/rmt52b.htm<http://www.rasch.org/rmt/rmt52b.htm> "CAT with a Poorly Calibrated Item Bank". But the initial results will be approximate (but useful) until you have collected enough data to properly calibrate the item bank. See also www.rasch.org/memo69.htm<http://www.rasch.org/memo69.htm> and www.rasch.org/memo40.htm

<http://www.rasch.org/memo40.htm>3. and 4. - What is the "conventional methodology"?  Is it Classical Test Theory (CTT, linear analysis of scored test responses) or Paper-and-Pencil (PAP) tests?

Rasch methodology can have considerable advantages over CTT.
CAT can have considerable advantages over PAP.
But the advantages depend on the situation.

For instance, in remedial teaching, you can combine a Rasch-based CAT test with computerized instruction, so that the student is automatically tutored on the topic-areas for which the CAT test reports poor performance.

Cordially,
Mike L.

At 1/28/2010, you wrote:

Hi,
     Thanks for the response. I still have a doubt about the practical implementation of the IRT.

1. Lets assume that I have started IRT from scratch. I have a list of questions to be administered to a group. For each of the question I know only whether the question is easy, medium or hard, the values of

    the item parameters are unknown for me. Is it required to calibrate the items in the test and find out the item parameters before it is administered? If yes

2. Is it required to administer each and every new item to a sample group ascertaining the item parameters before it is given to the real examinees?

3. What advantage does the IRT implementation gives to the tutor who is supposed to take the remedial teaching, which anyway can also be taken from any kind of conventional testing methodology?

4. I have read in an article that the CAT helps in reducing the test timings and also it could be administered at time independent fashion still measuring the examinees ability on the same scale, Are these the only advantages of using IRT? IF any other advantage is there over the conventional methodology please explain.



Regards,

Ranganath S
Mike Linacre
Editor, Rasch Measurement Transactions
rmt at rasch.org www.rasch.org/rmt/<http://www.rasch.org/rmt/> Latest RMT:  23:2 Autumn 2009
-------------------------------------------------
Please consider the environment before you print

-----Inline Attachment Follows-----
_______________________________________________
Rasch mailing list
Rasch at acer.edu.au<http://us.mc1115.mail.yahoo.com/mc/compose?to=Rasch@acer.edu.au>
https://mailinglist.acer.edu.au/mailman/listinfo/rasch


-------------------------------------------------

Please consider the environment before you print

-------------------------------------------------
Please consider the environment before you print



More information about the Rasch mailing list