[Rasch] Re:Urgent question about high stakes and normality

iasonas lambrianou liasonas at lycos.com
Tue Jan 31 03:25:29 EST 2006


Thanks

Can I have a more technical and detailed explanation of the iterative
process? Or any research, report etc?

jason

  ----- Original Message -----
  From: "Looveer, Juho"
  To: "iasonas lambrianou" , rasch at acer.edu.au
  Subject: RE: [Rasch] Re:Urgent question about high stakes and
  normality
  Date: Mon, 30 Jan 2006 14:27:11 +1100

Just for interest . . . An alternate approach previously used within our
state of New South Wales for matriculation exam results was generally to
scale all subjects to the same mean (=60/100) and standard deviation
(12.5), and then combine the marks.  In 2000 we moved to a standards
based system; panels of experts assess each examination against a set of
standards and determine cut off marks for varying levels of achievement
(e.g 90-100, 80-89, etc). For University entrance, Professor George
Cooney has developed a complex iterative process.  Basically, you look at
the results for all students who studied some specific course. Then
adjust their scores in that subject according to how they performed
overall.  Apply this to all courses and continue until the iteration
converges sufficiently This is intended to adjust their scores according
to overall performance.  see
http://www.boardofstudies.nsw.edu.au/manuals/pdf_doc/cooney.pdf#search='George%20Cooney' or http://www.uac.edu.au/admin/uai.html
for more details. Professor Jim Tognolini's doctoral thesis looked at
combining marks that had been scaled according to Rasch analyses.  From
memory, a single Rasch score was sufficient to describe performance and
rank students for over 93% of the students. So there are various
alternatives.  The key trick is being able to sell an idea politically
for it to be adopted.

Dr Juho Looveer

------------------------------------------------------------------------

From: rasch-bounces at acer.edu.au [mailto:rasch-bounces at acer.edu.au] On
Behalf Of iasonas lambrianou
Sent: Monday, 30 January 2006 5:00 AM
To: rasch at acer.edu.au
Subject: Re: [Rasch] Re:Urgent question about high stakes and normality

Thank you Mike and Agustín.

Indeed, pretending to be able to predict a student's performance in
History using his score in English and Latin is rediculuous. And I do not
think that this could stand in the court as well. Human nature is too
complex and our models (including our favourate, Rasch model) seem to be
too simplistic to predict it (the human nature). Also, we know that
students sometimes pay more attention on one test and study less for
another test: so its not like height or weight, performance on different
tests does not follow our 'physical rules'.

Having said that, I spent too much time talking to people from different
countries. It seems that, at least in Australia, Europe, Singapour etc
the people demand that we need to leave the students free to choose to be
tested on their favourate subjects, but then we need to report and use
'overall scores'. In many countries people use the t-scores and then they
aggregate them in various ways. I would have hoped that somebody might be
able to suggest a better way.

Would it be a good idea to contact all those people and then write a few
short papers describing our case studies and try to find out a solution
to this problem? Or maybe we should just try to convinvce the people and
the politicians that no statistical models can fix problems caused by
irrational political decisions and bad practice?

 Things become even worse when people run Rasch analysis using 50 cases
when their rating scales range from 1-100. They do not even have 1 person
per scale point. By the way, Mike, how would you comment on that?

Jason

  ----- Original Message -----
  From: "Agustin Tristan"
  To: "iasonas lambrianou" , "-Rasch"
  Subject: Re: [Rasch] Re:Urgent question about high stakes and
  normality
  Date: Sun, 29 Jan 2006 07:17:23 -0800 (PST)

  Hi Iasonas: I can imagine two scenarios: Scenario 1. Tests for
  different profiles of students. In this case you have a set of tests
  with different attributes according to a profile or speciality:
  Profile 1. Physics and Chemistry. Students have to respond a test
  with thess modules: Math, Organic Chemistry, Dynamics, English.
  Profile 2. Socio-economics. Students have to respond these modules:
  Math, Statistics, History and English.    Profile 3.... and so forth.
  A student has to pass only one profile, as they are candidates to
  different specialities or schools.In a profile you are using "shared"
  modules, like Math and English that are used in two or more profiles.
  They are not anchor tests, but modules or blocks that have to be
  included as part of the profile, but under some conditions you may
  use the blocks as anchors.In this case, you get results from students
  in each module and also you have the total result (when I say result
  I can imagine the measure obtained from the Rasch model, you have
  measures for each module and a measure of the total of the test). If
  this is true, I think you can report your results as they come from
  your analysis, and no need to add the scores or results of the parts
  to get a total. Trying to be clear: Profile 1: Math (20 items),
  Organic Chemistry (40 items), Dynamics (25 items) and English (50
  items), all the Test (20+40+25+50 = 135 items). If you analyse to
  score the students, you will have a Rasch measure in Math (say 0.5) ,
  another measure in O.C. (say 0.8), another in Dynamics (say 1.3)  and
  another in English (say 0.0), and also another for the total (say
  0.9).   As you see the total 0.9 is not the addition of the parts,
  but you don't need to add this.By the way, if you get the normal Z
  scores of each student in each part, the Z value for all the test is
  not the addition of the Z of the blocks, and a mean score may
  probably provide wrong interpretations.In this case you may order
  your students and make decisions per profile as they are
  participating in different schools. You will have problems if you
  wish to get a general ordered listing of all the students, as you're
  using different measurement instruments per profile. Scenario 2.
  Testing in blocks. Let's imagine you're responsible of a new
  educational program in Manchester, including new didactic materials,
  new curriculum, a set of video-games, activities and TV shows. You
  need a pre and post test to have "a picture" of the students with and
  without your educational program. As it is very difficult, expensive
  and time consuming to apply a test with all the items for all the
  educational areas, you are producing a test made with blocks: Math,
  Organic Chemistry, Dynamics, English, Statistics and History. Each
  student will respond a version including 3 of these blocks, in such a
  way that all the blocks will be tested, but there is no students
  answering all the blocks. If this is the case, you will not have a
  problem to get measures of the students in the block they have worked
  and you will not have difficulties to analyze each block and the
  items included in it.When this kind of test is used for a nationwide
  evaluation, you will not need the result per student. The problem is
  if you wish to have a comparative result among students.The problem
  in this case is how to get a total result for a student if you wish
  to say how much does he know in the blocks he didn't answer, just
  using the information provided by the blocks he really answered.We
  have some discussions in Mexico and other countries related to that.
  We are facing two options:a) To produce an equated result between
  students of different versions to get an overall result independently
  of the blocks he/she has worked.b) To provide the measure of the
  version he/she has worked without equating.In both cases you don't
  have to add the Z values of the equated measures. The other
  possibilities regard some extrapolations to imagine what is the
  measure in History of a student, if he doesn't take the History
  block, just knowing his results in the other blocks. We think this is
  quite hard to believe, at least for us. If someone has a solution in
  this listserve please let us know. Hope this helps.Agustín Tristan


  FAMILIA DE PROGRAMAS KALT. Mariano Jiménez 1830 A Col. Balcones del
  Valle 78280, San Luis Potosí, S.L.P. México TEL (52) 44-4820 37 88,
  44-4820 04 31 FAX (52) 44-4815 48 48 web page (in Spanish AND
  ENGLISH): http://www.ieesa-kalt.com

  --------------------------------------------------------------------


  What are the most popular cars? Find out at Yahoo! Autos
  _______________________________________________
  Rasch mailing list
  Rasch at acer.edu.au
  http://listserv3.acer.edu.au/mailman/listinfo/rasch



Dr. Iasonas Lamprianou
CFAS, School of Education
The University of Manchester
Oxford Road, Manchester M13 9PL, UK
Tel. 0044  161 275 3485
iasonas.lamprianou at man.ac.uk



--

_______________________________________________
Search for businesses by name, location, or phone number. -Lycos Yellow
Pages

http://yp.lycos.com

**********************************************************************
This message is intended for the addressee named and may contain
privileged information or confidential information or both. If you
are not the intended recipient please delete it and notify the sender.
**********************************************************************


_______________________________________________
Rasch mailing list
Rasch at acer.edu.au
http://listserv3.acer.edu.au/mailman/listinfo/rasch


Dr. Iasonas Lamprianou
CFAS, School of Education
The University of Manchester
Oxford Road, Manchester M13 9PL, UK
Tel. 0044  161 275 3485
iasonas.lamprianou at man.ac.uk


-- 
_______________________________________________

Search for businesses by name, location, or phone number.  -Lycos Yellow Pages

http://r.lycos.com/r/yp_emailfooter/http://yellowpages.lycos.com/default.asp?SRC=lycos10

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20060130/0af3b178/attachment.html 


More information about the Rasch mailing list