[Rasch] Re:Urgent question about high stakes and normality

Looveer, Juho Juho.Looveer at det.nsw.edu.au
Mon Jan 30 14:27:11 EST 2006

Just for interest . . . 
An alternate approach previously used within our state of New South Wales for matriculation exam results was generally to scale all subjects to the same mean (=60/100) and standard deviation (12.5), and then combine the marks.  In 2000 we moved to a standards based system; panels of experts assess each examination against a set of standards and determine cut off marks for varying levels of achievement (e.g 90-100, 80-89, etc).
For University entrance, Professor George Cooney has developed a complex iterative process.  Basically, you look at the results for all students who studied some specific course. Then adjust their scores in that subject according to how they performed overall.  Apply this to all courses and continue until the iteration converges sufficiently This is intended to adjust their scores according to overall performance.  see http://www.boardofstudies.nsw.edu.au/manuals/pdf_doc/cooney.pdf#search='George%20Cooney' or http://www.uac.edu.au/admin/uai.html for more details.
Professor Jim Tognolini's doctoral thesis looked at combining marks that had been scaled according to Rasch analyses.  From memory, a single Rasch score was sufficient to describe performance and rank students for over 93% of the students.
So there are various alternatives.  The key trick is being able to sell an idea politically for it to be adopted.

Dr Juho Looveer


From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On Behalf Of iasonas lambrianou
Sent: Monday, 30 January 2006 5:00 AM
To: rasch at acer.edu.au
Subject: Re: [Rasch] Re:Urgent question about high stakes and normality

Thank you Mike and Agustín.

Indeed, pretending to be able to predict a student's performance in History using his score in English and Latin is rediculuous. And I do not think that this could stand in the court as well. Human nature is too complex and our models (including our favourate, Rasch model) seem to be too simplistic to predict it (the human nature). Also, we know that students sometimes pay more attention on one test and study less for another test: so its not like height or weight, performance on different tests does not follow our 'physical rules'. 

Having said that, I spent too much time talking to people from different countries. It seems that, at least in Australia, Europe, Singapour etc the people demand that we need to leave the students free to choose to be tested on their favourate subjects, but then we need to report and use 'overall scores'. In many countries people use the t-scores and then they aggregate them in various ways. I would have hoped that somebody might be able to suggest a better way. 

Would it be a good idea to contact all those people and then write a few short papers describing our case studies and try to find out a solution to this problem? Or maybe we should just try to convinvce the people and the politicians that no statistical models can fix problems caused by irrational political decisions and bad practice?

 Things become even worse when people run Rasch analysis using 50 cases when their rating scales range from 1-100. They do not even have 1 person per scale point. By the way, Mike, how would you comment on that?


	----- Original Message -----
	From: "Agustin Tristan" 
	To: "iasonas lambrianou" , "-Rasch" 
	Subject: Re: [Rasch] Re:Urgent question about high stakes and normality
	Date: Sun, 29 Jan 2006 07:17:23 -0800 (PST)
	Hi Iasonas: I can imagine two scenarios:
	Scenario 1. Tests for different profiles of students. In this case you have a set of tests with different attributes according to a profile or speciality:
	    Profile 1. Physics and Chemistry. Students have to respond a test with thess modules: Math, Organic 
	                  Chemistry, Dynamics, English.
	    Profile 2. Socio-economics. Students have to respond these modules: Math, Statistics, History and English.
	    Profile 3.... and so forth.
	A student has to pass only one profile, as they are candidates to different specialities or schools.
	In a profile you are using "shared" modules, like Math and English that are used in two or more profiles. They are not anchor tests, but modules or blocks that have to be included as part of the profile, but under some conditions you may use the blocks as anchors.
	In this case, you get results from students in each module and also you have the total result (when I say result I can imagine the measure obtained from the Rasch model, you have measures for each module and a measure of the total of the test). If this is true, I think you can report your results as they come from your analysis, and no need to add the scores or results of the parts to get a total. Trying to be clear: 
	Profile 1: Math (20 items), Organic Chemistry (40 items), Dynamics (25 items) and English (50 items), all the Test (20+40+25+50 = 135 items). If you analyse to score the students, you will have a Rasch measure in Math (say 0.5) , another measure in O.C. (say 0.8), another in Dynamics (say 1.3)  and another in English (say 0.0), and also another for the total (say 0.9).   As you see the total 0.9 is not the addition of the parts, but you don't need to add this.
	By the way, if you get the normal Z scores of each student in each part, the Z value for all the test is not the addition of the Z of the blocks, and a mean score may probably provide wrong interpretations.
	In this case you may order your students and make decisions per profile as they are participating in different schools. You will have problems if you wish to get a general ordered listing of all the students, as you're using different measurement instruments per profile.
	Scenario 2. Testing in blocks. Let's imagine you're responsible of a new educational program in Manchester, including new didactic materials, new curriculum, a set of video-games, activities and TV shows. You need a pre and post test to have "a picture" of the students with and without your educational program. As it is very difficult, expensive and time consuming to apply a test with all the items for all the educational areas, you are producing a test made with blocks: Math, Organic Chemistry, Dynamics, English, Statistics and History. Each student will respond a version including 3 of these blocks, in such a way that all the blocks will be tested, but there is no students answering all the blocks. If this is the case, you will not have a problem to get measures of the students in the block they have worked and you will not have difficulties to analyze each block and the items included in it.
	When this kind of test is used for a nationwide evaluation, you will not need the result per student. The problem is if you wish to have a comparative result among students.
	The problem in this case is how to get a total result for a student if you wish to say how much does he know in 
	the blocks he didn't answer, just using the information provided by the blocks he really answered.
	We have some discussions in Mexico and other countries related to that. We are facing two options:
	a) To produce an equated result between students of different versions to get an overall result independently of the blocks he/she has worked.
	b) To provide the measure of the version he/she has worked without equating.
	In both cases you don't have to add the Z values of the equated measures.
	The other possibilities regard some extrapolations to imagine what is the measure in History of a student, if he doesn't take the History block, just knowing his results in the other blocks. We think this is quite hard to believe, at least for us. If someone has a solution in this listserve please let us know.
	Hope this helps.
	Agustín Tristan

	Mariano Jiménez 1830 A 
	Col. Balcones del Valle 
	78280, San Luis Potosí, S.L.P. México 
	TEL (52) 44-4820 37 88, 44-4820 04 31 
	FAX (52) 44-4815 48 48 
	web page (in Spanish AND ENGLISH): http://www.ieesa-kalt.com


	What are the most popular cars? Find out at Yahoo! Autos <http://us.rd.yahoo.com/evt=38382/_ylc=X3oDMTEzNWFva2Y2BF9TAzk3MTA3MDc2BHNlYwNtYWlsdGFncwRzbGsDMmF1dG9z/*http://autos.yahoo.com/newcars/popular/thisweek.html>  
	Rasch mailing list
	Rasch at acer.edu.au

Dr. Iasonas Lamprianou
CFAS, School of Education
The University of Manchester
Oxford Road, Manchester M13 9PL, UK
Tel. 0044  161 275 3485
iasonas.lamprianou at man.ac.uk


Search for businesses by name, location, or phone number. -Lycos Yellow Pages 

http://yp.lycos.com <http://r.lycos.com/r/yp_emailfooter/http://yellowpages.lycos.com/default.asp?SRC=lycos10>  

This message is intended for the addressee named and may contain
privileged information or confidential information or both. If you
are not the intended recipient please delete it and notify the sender.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20060130/95cb389a/attachment.html 

More information about the Rasch mailing list