[Rasch] Reliability of assessments run in circuits

Imogene Rothnie irothnie at usyd.edu.au
Mon Feb 25 13:42:56 EST 2013


Thank you for the advice.
I have run the analyses this way: 1. If you want the examinee reliability for each circuit based on their overall measures, as using only their individual circuit measures as in option 2 below creates subsets in the data, equivalent to a subset per examiner. This is unsurprising as within a circuit examiners are nested within item (and there are 5 items).
Some aspects of the result are a little puzzling to me and I wonder if they discount the use of this analysis to establish a reliability for person (applicant) separation for each circuit. Firstly the estimation concludes after 2 iterations..is this because all the elements have anchor values, and essentially I am having one 'other pass' at the data by separating the elements involved in a particular circuit out and, I guess, looking at the displacement from the original estimation values?
Secondly, and unsurprisingly now given the very small numbers of items and judges and applicants - I am getting significant results for the Model,  Random (normal) chi-square, so the applicants can't be considered to be a random sample from a normal population  - does this mean the person separation reliability cannot be used? It seems quite high in some cases and I'm wondering if this might be the cause.

I'm also getting Non significant results for the Model (All same) chi square. So I have to conclude the items/judges CAN be considered to be sharing the same measure in this case (not surprising as reliability is low for these facets)...if we are asking items of little reliable difference in difficulty and judging with a 'similar' level of severity what impact does this have on applicant separation reliability?

Cheers


From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf Of Patrick B. Fisher
Sent: Friday, February 22, 2013 1:25 PM
To: rasch at acer.edu.au
Subject: Re: [Rasch] Reliability of assessments run in circuits

No, don't delete it just use a semicolon ; - anything that appears AFTER a semicolon FACETS ignores.

Patrick
On 2/21/2013 7:48 PM, Imogene Rothnie wrote:
Thanks very much Mike, I'm not sure what you mean by 'comment out'? Delete?


From: rasch-bounces at acer.edu.au<mailto:rasch-bounces at acer.edu.au> [mailto:rasch-bounces at acer.edu.au] On Behalf Of Mike Linacre
Sent: Friday, February 22, 2013 12:47 PM
To: rasch at acer.edu.au<mailto:rasch at acer.edu.au>
Subject: Re: [Rasch] Reliability of assessments run in circuits

Thank you for your question, Imogene.

1. If you want the examinee reliability for each circuit based on their overall measures:
With Facets, analyze the 24 circuits. Output an Anchorfile=af.txt with anchor values. Comment out all the circuits except one. Use the anchor file af.txt as the specification file of a Facets analysis. The candidate reliability for that circuit is reported.

2. 1. If you want the examinee reliability for each circuit based on their measures for that circuit:
With the Facets specification file for 24 circuits, comment out all the circuits except one. Use that as the specification file of a Facets analysis. The candidate reliability for that circuit is reported.

OK?

Mike L.

On 2/21/2013 17:13 PM, Imogene Rothnie wrote:
I have run a FACETS analysis on an assessment where candidates perform 5 tasks out of a pool of 33 tasks. The 33 tasks are organized into 24 circuits of 5 items, all sharing a common anchor item.
There isn't much spread of difficulty between the tasks, and they are all actually fairly similar, and so the average difficulty of each circuit does not vary by any significant amount (important).
 HOWEVER, I am trying to ascertain the reliability of the assessment, per circuit, as these are the scores we rank candidates on, not the reliability reported for the 33 items overall as a facet of measurement .
 I have included 'circuit' as a group anchor dummy facet, so am assuming the reported (sample) reliability of 0.96 is not to be interpreted as , say, a Cronbach's alpha like we might do for other facets.  Does anyone know how I might be able to calculate or find output in FACETS that can reveal reliability of the assessment, for the candidate experience of 5 items?




_______________________________________________

Rasch mailing list

Rasch at acer.edu.au<mailto:Rasch at acer.edu.au>

Unsubscribe: https://mailinglist.acer.edu.au/mailman/options/rasch/pfisher%40sportsmeasures.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20130225/78accf03/attachment.html 


More information about the Rasch mailing list