[Rasch] Solving longitudinal puzzles with Rasch?
liasonas at cytanet.com.cy
Thu Dec 25 18:22:49 EST 2014
Dear Agustin and all,
Merry Christmas and a Happy new Year!!!!!
Dear Agustin, I am afraid that I had missed your email, and just read it! Thank you for the interest. Here are some responses
(a) All the quizzes/items are nested within "books". The teachers assign books to the pupils, and while reading the books online, the pupil take the quizzes. They all tap on the same dimension (i.e. reading comprehension). There are 7 different target areas (we can imagine them as sub-dimensions) but the quizzes really seem to be unidimensional (I will explain later)
(b) pupils can read as many books as they like, therefore, they can complete as many items/quizzes as they like. This is part of an informal classroom assessment program
(c) the tests are useful because they are used for informal evaluation purposes and to motivate pupils. Yes, they are useful even if they are not calibrated, but they would be much more useful if we could calibrate them!
(d) many pupils take only one test. In this case, we cannot measure progress, but we can use their responses when calibrating the items. What I want at the moment, is the right way to calibrate the items and the measurement of progress is a natural consequence when we have a calibrated item bank in hand
(e) I have done the following trick, please let me know if it sounds robust.
Overall, there are 2,200 items and more than 100,000 pupils. I have data across many months. I chose to use data from march/april 2013 as Snapshot 1. I used data from Sept./october 2014 as Snapshot 2. I used data from March/April 2014 as Snapshot 3. Each Snapshot has a length of two months. I assume that within two months there are no measurable differences in a pupils reading comprehension ability (which sounds reasonable to me, at least!). So, I assume that the pupil ability is fixed within each Snapshot. Different pupils have taken different number of tests within a Snapshot and at different times, but I so not care because I assume that their ability is practically stable within the two-month period of a Snapshot. Then, I calibrated each of the three Snapshots independently and I compared the item estimates. The correlation between the item estimates is 0.98. Question: is this a good enough proof that there is sample invariance?
The model-data fit is not excellent. Around 10% of the items have infit mean square larger than 1.3. However, less than 0.5% have infit mean square larger than 1.6. How should i judge the model-fit? Is it acceptable or good for low stakes purposes. I use a Rating Scale model (0-1-2). I do not use the t-transformed statistics because I have many thousands of pupils and many of those are large. The analysis gave a reliability index for the items of 1.00 and strata=25.
Now, my plan is to use the item estimates of one of the calibrations (using the results of the most recent Snapshot) as an anchor, and then re-run the three analysis estimating only pupil measures. Those pupils who appear in more than one snapshot will have their progress measured. How does this sound?
Have a happy new year!
From: Agustin Tristan < agustintristan123 at gmail.com >
Date: 2014-12-15 3:43 GMT-06:00
Subject: Re: [Rasch] Solving longitudinal puzzles with Rasch?
To: rasch at acer.edu.au
Something that I need from Iasonas is to explain:
a) How are the tests A to Z designed? are they of the same topics? Do they follow a test blueprint? are they a random selection of items of any content of the course? any other idea concerning their validity could be useful.
b) why a student can take only one test or why take many tests? Is it because they receive a score and they can pass the course?
c) Are the test conceived for formative evaluation? i.e. to give some insight of deficiencies and suggest to students some areas to study and improve their learning? in such case they could be useful even if the items are not calibrated.
d) If some students can take only one test, how can we assume that they have a change on their learning? it is a curious time series, without a pattern, following the ideas by Dr, Linacre, the scenarios may grow not only regarding improvement but also regarding comparison among time if a student takes only one test.
e) Is it possible to do 2 calibrations? calibrate the items using (1) the first and (2) the last test of each student and it is supposed that several items (or perhaps all the items) will be the same in both occasions.
Hope to hear from you
2014-12-11 22:29 GMT-06:00 Iasonas Lamprianou < liasonas at cytanet.com.cy > :
I need to solve a longitudinal puzzle. I would love to use Rasch (if it is the most appropriate tool). My post is long, but my puzzle is complex!
I have data from a computerized test. The students were allowed to log in whenever they wanted to take any number of short tests.Each test had 3-7 questions. Each test consists of different questions. There is no cosnistent pattern as to which tests were completed by the students (i.e. some students completed test A first but others would complete test Z first). The tests are not of the same difficulty. The items within a test are not of the same difficulty. The tests/items are not calibrated. There are many thousands of students and tens of tests (=hundreds of items). The teachers have a vague idea of the difficulty of each test, so they tried to match the difficulty of the test with the ability of the students. But of course, as I said, the tests are not calibrated (so the teachers were not really sure how difficult each test was), and they did not really have precise measures of the students ability (but of course they knew their students). This practice lasted for a whole year. Some students were more industrious, so they used log in every week (any time during the month/year) and they used to take a test. Others logged in once a month; and others only logged in once and took only one test. Overall, the students have taken on average 4-5 tests (=15-20 items), at random time points across the year. However, the ability of the students changed across the year. My question is how can I use (if I can) the Rasch model to analyze the data? In effect, my aim is: (a) to calibrate all the tests/items so that I can have an item bank, and (b) estimate student abilities at the start and end of the year (wherever possible) to measure progress. I am ready to assume that item difficulties do not change (we do not alter the items) but student abilities do change (hopefully improve) across time.
I am not sure if this puzzle can be solved using Rasch models. I thought that I could split the year in intervals of, say, 2 months. Assume that the ability of each person during those two months is more or less the same. Also assume that each person is a different version of itself in the next two months. Then assume that item difficulties are fixed. Then run the analysis with six times the number of students (each two months the student "changes"). This has the problem that the students are not really "different" and there should be a lot of collinearity (dependence).
Any idea will be values and considered to be significant.
Thank you very much
Rasch mailing list
email: Rasch at acer.edu.au
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Rasch