[RE]Re: [Rasch] item bank
Lang, William Steve
WSLang at tempest.coedu.usf.edu
Wed Oct 25 21:26:55 EST 2006
We have used an open item bank (we call it an assessment task library) in Florida. With performance tasks necessary for teacher certification, most performances and products are required to be rated on a rubric at an "minimally acceptable" level. Most allow multiple tries for the teacher candidate to pass. The items that are standards-based are known to those who are to be assessed.
The initial item analysis, calibration, and subsequent reporting are Rasch based. In Florida we have approximately 45 "tasks" and 350 "items" (criteria) that must be completed. Think of it like this...to be a pilot, you must demonstrate you can land the plane. Items are the criteria that provide evidence that you can land the plane successfully.
In another example, think of a set of items that are skilled-based like the famous Knox Cube data. Can a test-taker "cheat"?
The idea is to deal with difficulty by avoiding knowledge-based items that in favor of performance-based items. It's more difficult and expensive to perform a large number of items over a longitudinal time and deal with professional raters, but from a validity standpoint it makes sense as job-related. You can already imagine why rater-error diagnosis (FACETS) is useful.
We have found some interesting results comparing the Rasch difficulties from the first try of candidates to the second or third try. If items become "easy", it says something about the mentoring/training of a program and the "teachability" of the items. I hope that makes sense. "Lesson planning" seems to improve rapidly with training, but "classroom management" improves more slowly (some never get it). Some items are relatively easy, but necessary for teachers; like "communication with parents". Others are difficult, but also necessary; "assessment accomodations for special students". We wrote an initial report for JAM.
In a book to be releases next month: "Assessing Teacher Competency: Five Standards-Based Steps to Valid Measurements Using the CAATS Model" , we (myself and Judy Wilkerson) describe Rasch and the reasons for using Rasch to a general audience, but we expect the performance items to typically be known to all the users. http://www.corwinpress.com/booksProdDesc.nav?prodId=Book229866
University of South Florida
From: rasch-bounces at acer.edu.au on behalf of iasonas lambrianou
Sent: Wed 10/25/2006 3:22 AM
To: Agustin Tristan
Subject: [RE]Re: [Rasch] item bank
Thank you everybody for offering me so much help on this issue. The truth is that I received tens of messages from all over the world, but I would like other to contribute as well, if they have not done so already.
It seems that there is much interest, and the trends are the following:
* excellent idea to have an open ended item bank accessible by everybody for high stakes exams because it makes securityu issues non existent, the quality of the items can be monitored, the candidates know what they will face etc.
* item calibrations make no senese since item difficulties will drift immediately
* a backwash effect is expected and teachers will probably teach to the item bank
* the results/scores of the candidates will be inflated (they will have the chance to memorize responses)
Have I missed anything? Please remind us.
I will also try to think whether anyone else is using such a thing in practice, and I'll make a literature review.
I think this is progressing to a journal article. Does anyone think that this is a good idea? Does anyone think that I should not spend my time writing such an article because there may not be enough interest by the audience or the editorial boards of journals? Which are the journals that would be more likely to publish such a think?
Please help me to 'interview' experts in the fields - please suggest names.
you have been really helpful. I hope you will help me more.
P.S. Apologies for hijacking the Rasch list. If not appropriate, please let me know.
---------[ Received Mail Content ]----------
Subject : Re: [Rasch] item bank
Date : Wed, 11 Oct 2006 11:36:11 -0700 (PDT)
From : Agustin Tristan <ici_kalt at yahoo.com>
To : -Rasch <rasch at acer.edu.au>
Another thing Iasonas:
Once you have an "open" item bank, is it useful to calibrate the items? the difficulty of the items will ve reduced, even most of them will be solved by mostly all people...so the initial calibration has nothing to do with the original calibration.
This is another important topic to discuss.
FAMILIA DE PROGRAMAS KALT.
Mariano Jiménez 1830 A
Col. Balcones del Valle
78280, San Luis Potosí, S.L.P. México
TEL (52) 44-4820 37 88, 44-4820 04 31
FAX (52) 44-4815 48 48
web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com
Stay in the know. Pulse on the new Yahoo.com. Check it out.
Dr. Iasonas Lamprianou
CFAS, School of Education
The University of Manchester
Oxford Road, Manchester M13 9PL, UK
Tel. 0044 161 275 3485
iasonas.lamprianou at man.ac.uk
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Rasch