[Rasch] A practical one

Agustin Tristan ici_kalt at yahoo.com
Wed Oct 12 01:39:12 EST 2005

Concerning the same problem. 
We are testing students leaving the university, they
need to pass the test in order to be certified to
become teachers. Every year we have to build 8
different tests (8 specialities, about 100 Multiple
choice items each test), and a single student has to
pass only the test of his/her specialty (Math, Natural
Sciences, Social Sciences, Literature, etc.). We
include about 15-20 items from previous exams, the
remaining are new items of the same type and
About 2000 students are tested every year, but  some
1300 are trying to be pass the test for the first time
(called the focal group), while the 700 remaining are
trying to be certified again (called the remanent
group) (some of them are in their 3rd year trial!). It
can be seen that the test is passed  by 70% of the
focal group, while it is passed only by less than 20%
of the remanent group.
In this situation we have checked that remanent group
even with the previous contact of the test, don't have
an advantage versus the focal group due to several
reasons: They are out of the school for more than 6
months, so their knowledge and  abilities are in a
very low level; they don't study or perform additional
activities to improve their abilities in preparation
for the test; the universities are not providing
special remedial activities to recover those

A different problem but related to this issue concerns
the diffusion of the item bank. In this case it is
clear that the use of the items from a "safe" item
bank or from an "open" item bank is testing a
different thing on the students who are in contact
with the bank in comparison with those students who
are not.

Agustin Tristan

--- David Andrich <D.Andrich at murdoch.edu.au> wrote:

> To assess effects of memory, it is necessary to test
> shortly after.  We
> tested this on Raven's progressive matrices two
> weeks later. There was a
> memory effect when raw scores were calculated. But
> as always, things are
> not that simple. On closer look, the practice effect
> helped students
> answer items that were easier correctly, that is,
> they made less what
> might be seen as genuinely experience with the
> format etc errors.
> However, they did not go beyond their original level
> in terms of
> answering more difficult items correctly.  So to
> assess this in any test
> or situation, a little experiment must be conducted
> - there will be no
> universal answer.
> David
> David Andrich, BSc, MEd (UWA); PhD(Chic), FASSA
> Professor, School of Education 
> Murdoch University 
> Murdoch, Western Australia 6150 
> Email: andrich at murdoch.edu.au
> Phone +61 8 9360 2245 
> Fax +61 8 93606280 
> 	-----Original Message-----
> 	From: rasch-bounces at acer.edu.au
> [mailto:rasch-bounces at acer.edu.au] On Behalf Of
> Stone, Gregory
> 	Sent: Monday, 10 October 2005 9:15 PM
> 	To: Trevor Bond; Looveer, Juho; John Barnard
> (EPEC);
> rasch-bounces at acer.edu.au
> 	Cc: Rasch listserve
> 	Subject: RE: [Rasch] A practical one
> 	On a practical note - I would wonder about the time
> between the
> first examination and the second.  It seems evident
> that when a
> reasonable time elapses, memory fades.  Indeed,
> several suggestions have
> been made that while candidates can in fact honestly
> memorize one or
> maybe two items, the rest are lost.  Either they
> remember different
> choices, different wording, etc. They ultimately
> fail to use the
> "remembered" information to their advantage.
> 	My own experience is such that if at least 6 months
> pass between
> time 1 and time 2, there is virtually no learning or
> improvement unless
> a concerted effort is made to increase skills.  As
> an example, in one of
> my certification examinations we have used the same
> form 8 times in a
> row (twice per year).  There is no perceptable
> difference in first time
> test taker passes.  The rate of passage for the
> second time takers
> continues lower than first timers and is largely
> identical to their
> initial peformances with a few minor increases. 
> After the second time,
> the probability of passing plunges into the terrible
> odds category
> (rather like the odds of GW Bush winning the Nobel
> Peace Prize).  As
> this is a high-stakes test, it is not taken lightly.
> 	Apart from time intervals, perhaps it is a semantic
> difference,
> but what sort of "paper" is it?  Multiple choice?
> Essay?  This too would
> have an impact, as I would suspect essays are much
> easier to remember
> than MCQs.
> 	Gregory
> 	Gregory E. Stone, Ph.D., M.A.
> 	Assistant Professor, Research and Measurement
> 	University of Toledo, College of Education,
> Mailstop #923
> 	-----Original Message-----
> 	From: rasch-bounces at acer.edu.au on behalf of Trevor
> Bond
> 	Sent: Mon 10/10/2005 5:17 AM
> 	To: Looveer, Juho; John Barnard (EPEC);
> rasch-bounces at acer.edu.au
> 	Cc: Rasch listserve
> 	Subject: RE: [Rasch] A practical one
> 	It would also be interesting to know how each
> repeating person's
> 	estimate has changed? do they still fit the model?
> are ther some
> 	questions now answered which are well beyond past /
> present
> 	capability?
> 	Juho's suggestion has merit: In that case, scale
> results for the
> 	non-repeaters according to all items, and "equate"
> scores of
> 	repeaters using only the unknown questions.
> 	best wishes
> 	T
> 	  At 6:50 PM +1000 10/10/05, Looveer, Juho wrote:
> 	>You are assuming that the repeaters have an
> advantage simply by
> 	>having done the same test before.  I have seen
> many teenagers
> who do
> 	>the same driving test several times but continue
> to fail,
> despite
> 	>knowing what will be expected of them.
> 	>So, Have the repeaters scored better because they
> have advance
> 	>knowledge of the tasks that have been set and thus
> these tasks
> no
> 	>longer are good indicators of knowledge across the
> domain?  Or
> has
> 	>their ability/performance actually improved -
> perhaps they were
> not
> 	>as well prepared for the test in the first
> instance?
> 	>
> 	>
> 	>However, assuming that you have considered all
> this and have
> reached
> 	>the correct conclusion, then you now have a
> pragmatic problem:
> how
> 	>to partial out the different effects of advantage
> due to
> knowing
> 	>specific test questions and actual improvement in
> knowledge.
> This
> 	>may or may not be possible. Can you be sure which
> questions the
> 	>repeaters may have remembered? If you can't be
> sure, then
> perhaps
> 	>option 1 is the way to go.
> 	>
> 	>Since there are repeaters, I assume that the test
> has
> previously
> 	>been calibrated?  In that case, scale results for
> the
> non-repeaters
> 	>according to all items, and "equate" scores of
> repeaters using
> only
> 	>the unknown questions.
> 	>
> 	>However if you are dealing with heart surgeons or
> commercial
> pilots,
> 	>I would urge that another test must be set and
> used.
> 	>
> 	>Juho Looveer
> 	>
> 	>________________________________
> 	>
> 	>From: John Barnard (EPEC)
> [mailto:JohnBarnard at bigpond.com]
> 	>Sent: Mon 10/10/2005 5:10 PM
> 	>To: Looveer, Juho
> 	>Subject: RE: [Rasch] A practical one
=== message truncated ===>
> Rasch mailing list
> Rasch at acer.edu.au
> http://listserv3.acer.edu.au/mailman/listinfo/rasch

Mariano Jiménez 1830 A 
Col. Balcones del Valle 
78280, San Luis Potosí, S.L.P. México 
TEL (52) 44-4820 37 88, 44-4820 04 31 
FAX (52) 44-4815 48 48 
web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com

Yahoo! Music Unlimited 
Access over 1 million songs. Try it free.

More information about the Rasch mailing list