[Rasch] Rasch: analyze two versions of a test

John johnbarnard at bigpond.com
Tue Jun 25 08:10:14 EST 2013


Hi Juho

 

I know that this discussion is not primarily about CAT, but want to respond
to some of your questions. (I have been implementing CAT since the late
80s.) As you said, CAT is a different ballgame, but highly efficient. As you
know in a "pure" CAT the algorithm usually selects the single item which
provides the maximum (Fisher) information - this is of course adjusted after
each response as a new ability estimate is obtained. There are methods to
manipulate this. For example, select one item randomly from say five items
at this difficulty. This is mainly for exposure control. Otherwise you can
dictate to the algorithm which specific item is to be administered after a
certain item, but that would of course disrupt the CAT nature. This may,
however, be useful in specific cases. In your math example the hierarchy is
very useful. For example, it is unlikely that you will be able to solve a
quadratic equation if you cannot solve a linear equation. To furher guide
the process, the SEM is estimated after each item is administrated and this
can also "keep you on track" as it needs to increasingly decrease.

 

In tests in which uni-dimensionality cannot be assumed or you want to derive
measures for different strands, say Number, Measurement, Space, ., you can
run sequential CATs - one for each strand. Obviously, if you want an overall
measure you will need to weigh the individual abilities by their SEs to
compute. We also implement CATs within CATs in an integrated CAT where the
same is achieved. So there are different ways in which one can manipulate a
CAT if you want to.

 

Finally, it may sound contradictory that you can revise and change responses
in a CAT since a change would mean that the pathway of item selection should
start again from that point onwards. This in not true. We have done some
significant research and showed that if you correlate initial ability
estimates with final (after review) estimates, you consistently find r >
0.995. Ability estimates are derived through ML estimation on the basis of
the item difficulties of the items administered and whether the response was
correct or incorrect. So, in a nutshell, you can go back to any item and
change the response in a CAT. (This is usually a "political" requirement as
it can be done in paper-and-pencil.) 

 

Kindly

John 

 

Prof John J Barnard (D.Ed.;Ph.D.;Ed.D.)

     Executive Director: EPEC Pty Ltd

                      EPEC logo 3

             <http://www.epecat.com> www.epecat.com

 

 

Augustin  et al

 

We aim for local independence, but I think in many tests, some items trigger
memory and ideas that can be useful for other items.

e.g. in a secondary school mathematics test, the same algebra skills are
required to manipulate formulae, to solve equations, for use of formulae
(substitute and evaluate), etc.

I am sure that sometimes there are some triggering effects from some items
to others.

Hence, it would be preferable to have items appear in a similar order, in
similar positions, etc.

Good test making includes balancing the items in a test according to
content, skills, order, etc; not just throwing the whatever number of items
in any order that happens to fit a page.

 

But you are correct, using CAT, this is a different issue.

Nit sure if anyone has explored it yet.

However, using good reasoning, we can consider the situation, and perhaps
items should be grouped in some way as well, to indicate those to be early
in a test, or late in  test, or near the middle.

 

One difference is that with pen and paper tests, the test taker can go back
and revise a response if  they have another thought about that item; so the
order of items can be countered by a student who is prompted for one item by
their working out another item. I am not sure that this is possible with
CAT.

 

Regards

 

Juho

 

 

 

Dr Juho Looveer

Australia

 

-----Original Message-----

From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf
Of rasch-request at acer.edu.au

Sent: Monday, June 24, 2013 9:28 PM

To: rasch at acer.edu.au

Subject: Rasch Digest, Vol 95, Issue 13

 

Send Rasch mailing list submissions to

      rasch at acer.edu.au

 

To subscribe or unsubscribe via the World Wide Web, visit

      https://mailinglist.acer.edu.au/mailman/listinfo/rasch

or, via email, send a message with subject or body 'help' to

      rasch-request at acer.edu.au

 

You can reach the person managing the list at

      rasch-owner at acer.edu.au

 

When replying, please edit your Subject line so it is more specific than

"Re: Contents of Rasch digest..."

 

 

Today's Topics:

 

   1. Re: Rasch: analyze two versions of a test (Agustin Tristan)

 

 

----------------------------------------------------------------------

 

Message: 1

Date: Sun, 23 Jun 2013 16:30:26 -0700

From: Agustin Tristan <ici_kalt at yahoo.com>

Subject: Re: [Rasch] Rasch: analyze two versions of a test

To: "rasch at acer.edu.au" <rasch at acer.edu.au>

Message-ID:

      <1372030226.62614.YahooMailNeo at web163405.mail.gq1.yahoo.com>

Content-Type: text/plain; charset="iso-8859-1"

 

Hello!

What if the test is not in a paper and pencil format but it is administered
using?CAT? Do the items need to appear in the same invariant order in every
CAT version if difficulty has to be invariant too?

Agustin

?

 

INSTITUTO DE EVALUACION E INGENIERIA AVANZADA.

Ave. Cordillera Occidental No. 635

Colonia Lomas 4? San Luis Potos?, San Luis Potos?.

C.P. 78216 MEXICO

(52) (444) 8 25 50 76 / (52) (444) 8 25 50 77 / (52) (444) 8 25 50 78 P?gina
Web (en espa?ol): http://www.ieia.com.mx/ Web page (in English):

http://www.ieesa-kalt.com/English/Frames_sp_pro.html

 

 

From: rsmith <rsmith at jampress.org>

To: rasch at acer.edu.au

Sent: Sunday, June 23, 2013 11:29 AM

Subject: Re: [Rasch] Rasch: analyze two versions of a test

 

 

Excellent advice Juho!

 

One of the myths of the achievement testing world is the belief that an item
will have the same item difficulty if it is the first or last item on the
test, when in effect this is hardly evey true.? So we have to construct our
tests to encourage the data to have the invariance properties necessary so
equating with the Rasch model produces the must useful result.

 

Richard M. Smith, Editor

Journal of Applied Measurement

P.O. Box 1283

Maple Grove, MN? 55311, USA

website:? www.jampress.org

phone: 763-268-2282

fax: 763-268-2782 

 

-------- Original Message --------

> From: Dr Juho Looveer <juho.looveer at gmail.com>

> Sent: Sunday, June 23, 2013 1:06 AM

> To: rasch at acer.edu.au

> Subject: [Rasch] Rasch: analyze two versions of a test

> 

> Lucia,

> It looks like the common items may be the last 30 in one test and the

first

> 30 in the other.

> if not too late, you might check that the common items are spread in

roughly

> similar positions in both tests.

> 

> Regards

> 

> Juho

> 

> 

> 

> Dr Juho Looveer

> Australia

> 

> -----Original Message-----

> From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On

Behalf

> Of rasch-request at acer.edu.au

> Sent: Sunday, June 23, 2013 12:00 PM

> To: rasch at acer.edu.au

> Subject: Rasch Digest, Vol 95, Issue 9

> 

> Send Rasch mailing list submissions to ??? rasch at acer.edu.au

> 

> To subscribe or unsubscribe via the World Wide Web, visit ??? 

> https://mailinglist.acer.edu.au/mailman/listinfo/rasch

> or, via email, send a message with subject or body 'help' to ??? 

> rasch-request at acer.edu.au

> 

> You can reach the person managing the list at ??? 

> rasch-owner at acer.edu.au

> 

> When replying, please edit your Subject line so it is more specific 

> than

> "Re: Contents of Rasch digest..."

> 

> 

> Today's Topics:

> 

>? ? 1. Rasch: analyze two versions of a test (Lucia Luyten) ? ? 2. Re: 

>Rasch: analyze two versions of a test (Bond, Trevor) ? ? 3. Re: Rasch: 

>analyze two versions of a test (Rense Lange)

> 

> 

> ----------------------------------------------------------------------

> 

> Message: 1

> Date: Sat, 22 Jun 2013 10:18:27 +0000

> From: Lucia Luyten <Lucia.Luyten at arts.kuleuven.be>

> Subject: [Rasch] Rasch: analyze two versions of a test

> To: "rasch at acer.edu.au" <rasch at acer.edu.au>

> Message-ID:

> ??? 

> <BB8B05E9D328B04D88B8D72DD963D85D37FBA142 at ICTS-S-MBX7.luna.kuleuven.be

> >

> ??? 

> Content-Type: text/plain; charset="iso-8859-1"

> 

> Hi

> 

> I have a question about analyzing two versions of a test.

> 

> Say we have 130 items for a test. We make two versions of this test. 

> In version A, we put items number 1 to 80 and in version 2 the items 

> number

50

> to 130. So items 50-80 are in both versions. In version A, the item

numbers

> 1 to 30 are anchor items from a previous test. For these items, we 

> know

and

> use the measures from a previous Facets analysis. These anchor items

(1-30)

> occur only in version A, not in version B.

> 

> About 400 candidates take version A, and about 250 take version B. The

test

> is rated by 4 raters. Raters rate both versions, each test taker is 

> rated

by

> one random rater.

> 

> One might choose to take all candidates together for analysis. Or one 

> can choose to first analyze version A separately (using the measures 

> for the anchor items).? And then use the outcome, i.e. the measures 

> for the identical items (number 50-80) and the measures for the 

> raters, in the subsequent analyses of version B.

> 

> Which way of analyzing is preferable and why is it?

> 

> Kind regards,

> 

> Lucia Luyten

> 

> 

> ________________________________

> Lucia Luyten

> wetenschappelijk medewerker

> CNaVT / CTO / KULeuven

> Blijde-Inkomststraat 7 bus 3319

> 3000 Leuven

> 016 32 53 59

> fax 016 32 53 60

> lucia.luyten at arts.kuleuven.be
<mailto:lucia.luyten at arts.kuleuven.be%3cmailto:lucia.luyten at arts.kuleuven.be
> <mailto:lucia.luyten at arts.kuleuven.be>

> http://www.cnavt.org/

> www.cteno.be <http://www.cteno.be%3chttp:/www.cteno.be/>
<http://www.cteno.be/>

> 

> 

> 

> 

> -------------- next part -------------- An HTML attachment was 

> scrubbed...

> URL:

> 

https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130622/1b502e6

> b/attachment-0001.html

> 

> ------------------------------

> 

> Message: 2

> Date: Sat, 22 Jun 2013 13:17:56 +0000

> From: "Bond, Trevor" <trevor.bond at jcu.edu.au>

> Subject: Re: [Rasch] Rasch: analyze two versions of a test

> To: "<rasch at acer.edu.au>" <rasch at acer.edu.au>

> Message-ID: <EB8A8EA9-84C6-424F-981B-FBC8E29FA722 at jcu.edu.au>

> Content-Type: text/plain; charset="us-ascii"

> 

> Dear Lucia

> You do it both ways, expecting invariance.

> Where you don't, you look for reasons.

> Then choose.

> TGB

> 

> 

> Sent from 007's iPad

> 

> On 22/06/2013, at 12:19 PM, "Lucia Luyten"

> <Lucia.Luyten at arts.kuleuven.be
<mailto:Lucia.Luyten at arts.kuleuven.be%3cmailto:Lucia.Luyten at arts.kuleuven.be
> <mailto:Lucia.Luyten at arts.kuleuven.be>>

wrote:

> 

> Hi

> 

> I have a question about analyzing two versions of a test.

> 

> Say we have 130 items for a test. We make two versions of this test. 

> In version A, we put items number 1 to 80 and in version 2 the items 

> number

50

> to 130. So items 50-80 are in both versions. In version A, the item

numbers

> 1 to 30 are anchor items from a previous test. For these items, we 

> know

and

> use the measures from a previous Facets analysis. These anchor items

(1-30)

> occur only in version A, not in version B.

> 

> About 400 candidates take version A, and about 250 take version B. The

test

> is rated by 4 raters. Raters rate both versions, each test taker is 

> rated

by

> one random rater.

> 

> One might choose to take all candidates together for analysis. Or one 

> can choose to first analyze version A separately (using the measures 

> for the anchor items).? And then use the outcome, i.e. the measures 

> for the identical items (number 50-80) and the measures for the 

> raters, in the subsequent analyses of version B.

> 

> Which way of analyzing is preferable and why is it?

> 

> Kind regards,

> 

> Lucia Luyten

> 

> 

> ________________________________

> Lucia Luyten

> wetenschappelijk medewerker

> CNaVT / CTO / KULeuven

> Blijde-Inkomststraat 7 bus 3319

> 3000 Leuven

> 016 32 53 59

> fax 016 32 53 60

> lucia.luyten at arts.kuleuven.be
<mailto:lucia.luyten at arts.kuleuven.be%3cmailto:lucia.luyten at arts.kuleuven.be
> <mailto:lucia.luyten at arts.kuleuven.be>

> http://www.cnavt.org/

> www.cteno.be <http://www.cteno.be%3chttp:/www.cteno.be/>
<http://www.cteno.be/>

> 

> 

> 

> 

> _______________________________________________

> Rasch mailing list

> Rasch at acer.edu.au <mailto:Rasch at acer.edu.au%3cmailto:Rasch at acer.edu.au>
<mailto:Rasch at acer.edu.au>

> Unsubscribe:

> 

https://mailinglist.acer.edu.au/mailman/options/rasch/trevor.bond%40jcu.edu.

> au

> -------------- next part -------------- An HTML attachment was 

> scrubbed...

> URL:

> 

https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130622/43e7e66

> e/attachment-0001.html

> 

> ------------------------------

> 

> Message: 3

> Date: Sat, 22 Jun 2013 10:00:14 -0500

> From: Rense Lange <rense.lange at gmail.com>

> Subject: Re: [Rasch] Rasch: analyze two versions of a test

> To: <rasch at acer.edu.au>

> Message-ID: <A69FA871-D198-46BA-B8B9-EE3FF6E02140 at gmail.com>

> Content-Type: text/plain; charset="windows-1252"

> 

> 

> Is there any way to have two or more raters evaluate the same people 

> on a fairly large scale? If so, you can also check rater effects using
Facets ?

> Even if you had only very limited numbers of double/triple/ ? ratings,

large

> rater differences/biases would be a sign for caution.

> 

> Rense Lange

> 

> On Jun 22, 2013, at 8:17 AM, "Bond, Trevor" <trevor.bond at jcu.edu.au>

wrote:

> 

> > Dear Lucia

> > You do it both ways, expecting invariance.

> > Where you don't, you look for reasons.

> > Then choose.

> > TGB

> > 

> > 

> > Sent from 007's iPad

> > 

> > On 22/06/2013, at 12:19 PM, "Lucia Luyten"

<Lucia.Luyten at arts.kuleuven.be>

> wrote:

> > 

> >> Hi

> >> 

> >> I have a question about analyzing two versions of a test. 

> >> 

> >> Say we have 130 items for a test. We make two versions of this 

> >> test. In

> version A, we put items number 1 to 80 and in version 2 the items 

> number

50

> to 130. So items 50-80 are in both versions. In version A, the item

numbers

> 1 to 30 are anchor items from a previous test. For these items, we 

> know

and

> use the measures from a previous Facets analysis. These anchor items

(1-30)

> occur only in version A, not in version B.

> >> 

> >> About 400 candidates take version A, and about 250 take version B. 

> >> The

> test is rated by 4 raters. Raters rate both versions, each test taker 

> is rated by one random rater.

> >> 

> >> One might choose to take all candidates together for analysis. Or 

> >> one

can

> choose to first analyze version A separately (using the measures for 

> the anchor items).? And then use the outcome, i.e. the measures for 

> the identical items (number 50-80) and the measures for the raters, in 

> the subsequent analyses of version B.

> >> 

> >> Which way of analyzing is preferable and why is it?

> >> 

> >> Kind regards,

> >> 

> >> Lucia Luyten

> >> 

> >> 

> >> Lucia Luyten

> >> wetenschappelijk medewerker

> >> CNaVT / CTO / KULeuven

> >> Blijde-Inkomststraat 7 bus 3319

> >> 3000 Leuven

> >> 016 32 53 59

> >> fax 016 32 53 60

> >> lucia.luyten at arts.kuleuven.be

> >> http://www.cnavt.org/

> >> 

> >> www.cteno.be

> >> 

> >> 

> >> 

> >>? 

> >>? 

> >> _______________________________________________

> >> Rasch mailing list

> >> Rasch at acer.edu.au

> >> Unsubscribe: 

> >> 

> >>https://mailinglist.acer.edu.au/mailman/options/rasch/trevor.bond%40

> >>j

> >> cu.edu.au

> > _______________________________________________

> > Rasch mailing list

> > Rasch at acer.edu.au

> > Unsubscribe: 

> > https://mailinglist.acer.edu.au/mailman/options/rasch/rense.lange%40

> > gm

> > ail.com

> 

> -------------- next part -------------- An HTML attachment was 

> scrubbed...

> URL:

> 

https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130622/ee34427

> f/attachment-0001.html

> 

> ------------------------------

> 

> _______________________________________________

> Rasch mailing list

> Rasch at acer.edu.au

> https://mailinglist.acer.edu.au/mailman/listinfo/rasch

> 

> 

> End of Rasch Digest, Vol 95, Issue 9

> ************************************

> 

> _______________________________________________

> Rasch mailing list

> Rasch at acer.edu.au

> Unsubscribe:

https://mailinglist.acer.edu.au/mailman/options/rasch/rsmith%40jampress.org

 

 

 

_______________________________________________

Rasch mailing list

Rasch at acer.edu.au

Unsubscribe:

https://mailinglist.acer.edu.au/mailman/options/rasch/ici_kalt%40yahoo.com

-------------- next part --------------

An HTML attachment was scrubbed...

URL:

https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130623/e2c08b4

9/attachment.html 

 

------------------------------

 

_______________________________________________

Rasch mailing list

Rasch at acer.edu.au

https://mailinglist.acer.edu.au/mailman/listinfo/rasch

 

 

End of Rasch Digest, Vol 95, Issue 13

*************************************

 

_______________________________________________

Rasch mailing list

Rasch at acer.edu.au

Unsubscribe:
https://mailinglist.acer.edu.au/mailman/options/rasch/johnbarnard%40bigpond.
com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130625/6182407c/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2057 bytes
Desc: not available
Url : https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130625/6182407c/attachment-0001.jpe 


More information about the Rasch mailing list