[Rasch] Establishing validity for standardized case examinations

Stone, Gregory gstone at UTNet.UToledo.Edu
Sat Apr 15 08:23:02 EST 2006


I am not up on recent journal articles in this area, but the question routinely appears in textbooks.  My memory on titles is bad but there are two educational assessment texts - one by Linn and Miller (formerly by Linn and Gronlund), and one by Anthony Nitko that include standard sections that at least skim over this topic.  Both are I think from Pearson and both I believe from Prentice Hall, but I could be mistaken.

However, an assessment is an assessment.  For the MCQ, you may have 200 or 300 or 400 items, all of which are balanced on the ratios established via your task analysis.  The ratios are suggested to be representative of the content in the profession, but that is something we're all familiar with surely.  What isn't as well documented is the question you ask.  The answer, however, must appear within the same model.  The Oral or Practical (authentic) exam must be representative in a similar way that the MCQ exam represents the profession.  We could ask an infinite number of MCQ items and still not cover the content completely.  Thus we choose what we believe are most critical.  Similarly, if your dentists can handle only three or four actual tasks, then those tasks must be most important.  Clearly, all content cannot be covered, so importance must be a consideration.  In this effort, I doubt there is as "clean" a measurement model.  When we licensed dentists in Florida, those tasks were clearly elaborated by the board in terms of most importance and patient safety, as it was a licensure exam.

While I don't believe it works for standard setting, Robert Ebel's creative ratings of items may be of some help.  Ebel's wisdom demonstrated to us that items bear more than single qualities - in his model, importance and difficutly.  Such a model would likely help to define the few pieces of the puzzle that could be examined in your authentic situation.  If you gathered importance and frequency information in your task analysis, you could use importance to reclassify content in at least one direction of a grid.  Using difficulty might be useful, as may other characteristics.

Ultimately, I think my point is whether or not MCQ, oral or authentic an assessment is still an assessment...and should follow along in the same representative way.  I really think Ebel was moving in a wonderful evaluative direction with his model - and I would encourage more in that direction.  Whatever you ultimately come up with, I hope you share your work with the rest of us also struggling with this question.


Gregory E. Stone, Ph.D., M.A.

Assistant Professor of Research and Measurement
The University of Toledo, Mailstop #923
Toledo, OH 43606   419-530-7224

Editorial Board, Journal of Applied Measurement

-----Original Message-----
From: rasch-bounces at acer.edu.au on behalf of Valerie Been Lober, PhD
Sent: Fri 4/14/2006 10:16 AM
To: rasch at acer.edu.au
Subject: [Rasch] Establishing validity for standardized case examinations
In addition to administering written MCQ examinations, our organization
offers standardized patient cases.  Dentists wishing to become Board
certified must pass both examinations. We are just completing our second
practice analysis to serve as the basis for these examinations.  For the
written exam, we have test specifications that spell out the proportion of
items that should appear on the written for each patient care activity.

However, I am unclear on the best way to translate the results of the
practice analysis to the standardized-patient case examination.  Could
someone direct me to relevant articles?  Thank you.

My best regards,

Valerie Been Lober, Ph.D., Executive Director
American Board of Oral Implantology/Implant Dentistry
211 East Chicago Avenue, Suite 750-B
Chicago, Il 60611
Voice 312-335-8793  Fax 312-335-9045
diplomate at aboi.org

Rasch mailing list
Rasch at acer.edu.au

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailinglist.acer.edu.au/pipermail/rasch/attachments/20060414/5c7d7078/attachment.html 

More information about the Rasch mailing list