[Rasch] Estimating Rasch Measures for Extreme Scores

Robert Hess Robert.Hess at ASU.edu
Wed May 4 07:25:32 EST 2011


Rense,
This is precisely the argument (or to be more exact, the discussion) that
Michael and I had back in 1994/95. His rationale then, as well as mine
today, was that you do not need to have raters crossed by schools (meaning
that you do not have Raters present in more than one school setting). What
you end up getting is an "adjusted" influence for each of the two facets.
Now think of the results in terms of an analysis of variance model, you have
an effect for each of these facets, but you want to establish whether or not
there is any influence unique to "schools". How would you do that, you would
look for an interaction wouldn't you? So with Facets you create another
facet, school by rater, or school*rater. In essence, what you're doing is
holding school constant, and examining to see if there's any unique factors
related to the way the raters within a given school are uniquely distinct
from the other schools. It does get somewhat complex, but Facets does handle
it quite well and the output provides all the necessary information to
confirm an effect. I am attaching a very simple command file that
illustrates the first part about searching for a possible interaction (if an
interaction is possible then simply rewrite your model to examine that
potential interaction). The command file merely formats the data set into a
Facets readable file and designates the type of output you are tentatively
seeking. The key word here being tentatively, because until you get the
results that explain the different "facets" influencing the student
scores/achievement, you may need to examine your data multiple times.
Hope this helps.
- Bob.
Also, you might want to look at these two articles:

Haladyna, T. & Hess, R.K. (2000). An evaluation of conjunctive and
compensatory standard-setting strategies for test decisions.  Applied
Assessment, Vol 6 (2), pp 129-153.

Haladyna, T. & Hess, R.K. (1994). The detection and correction of bias in
student ratings of instruction.  Research in Higher Education, 35, 6, pp.
669-687.

Robert Hess, PhD
Prof. Emeritus of Educational Measurement & Evaluation.
Arizona State University

-----Original Message-----
From: Rense Lange [mailto:rense.lange at gmail.com] 
Sent: Sunday, May 01, 2011 11:04 AM
To: Robert Hess
Cc: rasch at acer.edu.au
Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores

One would need raters common to pairs of schools, i.e., connectivity within
as well as between schools is needed. Otherwise schools and teachers will
just be their own "subset" on the Facets output. Also, if I read Facets'
manual correctly, two entities cannot be connected through the most extreme
scores only - at least one non-perfect score connection is needed.

Rense Lange


On Apr 30, 2011, at 3:55 PM, Robert Hess wrote:

> The answer to your question is - Yes. I know because I have done this in
> analyzing district results in large school systems (Mesa, Arizona - 34
> schools and Scottsdale, Arizona - 26 schools). And furthermore I have done
> this over multiple years. You may use teacher as a facet (based on the
> discussions you presented earlier, you have to use teacher a facet) but I
> strongly suggest you utilize school as a facet, since the teachers with in
a
> school may employ a uniquely different interpretation of your scales - in
> this matter, you can determine if schools are a factor in your scaling.
The
> explanation you're really seek an answer to is what contributes to the
> scoring/achievement of the students? It may well be that you actually have
> an interaction between Teachers and Schools and if so Facets is capable of
> providing an interpretation of this influence (again, something that I
have
> done). I have also done this on statewide data where raters within a
scoring
> center were examined for their potential contributions to the rating of
> student performance (on six different elements of writing performance -
> traits). It is not is difficult as it sounds. The real key will be and how
> you layout your scoring key within Facets.
> Between 1995 and 2002, I must to run over 4000 of this type of analysis
> utilizing Facets. In fact, in 2004, I ran one for the state Board of
> Education to validate the scoring strategies employed by the state
> contractor for the writing exam.
> 
> Robert Hess
> Prof. Emeritus of Educational Measurement & Evaluation
> Arizona State University
> 
> -----Original Message-----
> From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf
> Of Steven L. Kramer
> Sent: Saturday, April 30, 2011 10:31 AM
> To: liasonas at cytanet.com.cy; Patrick B. Fisher; rasch-bounces at acer.edu.au;
> rasch at acer.edu.au
> Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
> 
> A question for Rasch modelers with more experience than me:
> 
> Can one use a Facets approach with Teacher as a Facet, if teachers are 
> nested within schools?  I don't see how one would be able to judge the 
> relative severity of teachers from different schools, unless students also

> moved around among schools so that common students could be used to
compare 
> teacher severity.  Otherwise, I would expect any software attempting to
use 
> teacher as a Facet to simply crash.  Is this correct?
> 
> If only one COULD use teacher-severity as a Facet with 
> teachers-nested-within-schools, then I would expect Patrick's suggestion
to 
> help.   But I am afraid it won't help in this situation.  Again:  please 
> could a more experienced person tell me whether I am correct?
> 
> Steve Kramer
> ----- Original Message ----- 
> From: <liasonas at cytanet.com.cy>
> To: "Patrick B. Fisher" <pfisher at SportsMeasures.com>; 
> <rasch-bounces at acer.edu.au>; <rasch at acer.edu.au>
> Sent: Saturday, April 30, 2011 11:18 AM
> Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
> 
> 
> Thank you patrick. But this would not solve the problem that the student 
> himself still has a perfect score, right? This seems to be the centre of
the
> 
> problem I am afraid...
> Sent from my BlackBerryR smartphone
> 
> -----Original Message-----
> From: "Patrick B. Fisher" <pfisher at SportsMeasures.com>
> Sender: rasch-bounces at acer.edu.au
> Date: Sat, 30 Apr 2011 09:29:46
> To: <rasch at acer.edu.au>
> Subject: Re: [Rasch] Estimating Rasch Measures for Extreme Scores
> 
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe: 
>
https://mailinglist.acer.edu.au/mailman/options/rasch/liasonas%40cytanet.com
> .cy
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe: 
>
https://mailinglist.acer.edu.au/mailman/options/rasch/skramer1958%40verizon.
> net
> 
> 
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe:
>
https://mailinglist.acer.edu.au/mailman/options/rasch/robert.hess%40asu.edu
> 
> No virus found in this outgoing message.
> Checked by AVG - www.avg.com 
> Version: 9.0.894 / Virus Database: 271.1.1/3606 - Release Date: 04/29/11
> 23:34:00
> 
> No virus found in this outgoing message.
> Checked by AVG - www.avg.com
> Version: 9.0.894 / Virus Database: 271.1.1/3606 - Release Date: 04/29/11
23:34:00
> _______________________________________________
> Rasch mailing list
> Rasch at acer.edu.au
> Unsubscribe:
https://mailinglist.acer.edu.au/mailman/options/rasch/rense.lange%40gmail.co
m

No virus found in this outgoing message.
Checked by AVG - www.avg.com 
Version: 9.0.894 / Virus Database: 271.1.1/3613 - Release Date: 05/03/11
07:14:00
-------------- next part --------------
A non-text attachment was scrubbed...
Name: AERA1.KEY
Type: application/octet-stream
Size: 1180 bytes
Desc: not available
Url : https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20110503/087bbfcb/attachment.obj 
-------------- next part --------------

No virus found in this outgoing message.
Checked by AVG - www.avg.com 
Version: 9.0.894 / Virus Database: 271.1.1/3613 - Release Date: 05/03/11 07:14:00


More information about the Rasch mailing list