# [Rasch] Reliability of a vector of estimates?

Donald Bacon dbacon at du.edu
Sat Feb 7 03:24:14 EST 2009

```Hi All –

I need a method of estimating reliability that I don’t think I’ve seen before.  I’m conducting a study of measures of attribute importance.  In marketing, attribute importance is often estimated using direct ratings, e.g., “how important is safety in your new car decision? 1= not at all important, 7 = extremely important.”  These importances are measured on several attributes (e.g., safety, size, mpg, acceleration, etc.).  The traditional procedure has been to take the average score on each attribute for the entire sample and report this importance vector (one estimate for each of k attributes).  Attribute importance measures can also be estimated using preference regression or conjoint analysis.

I’m working on an alternative method that uses Rasch instead.  I’m then going to compare the convergent validity of the estimates by examining the correlation between the Rasch estimate vector and the direct ratings vector (along with two other methods).  In examining those convergent validity estimates, it also seems appropriate to examine the reliability of each vector.  I’m thinking I could split the sample, estimate the importances on each half of the sample, and then examine the correlation between these split half results.  This is analogous to traditional split half reliability, except that instead of using all persons but splitting the items, I’m using all the items but splitting the persons.  Unfortunately, the analogy is not perfect.  In the case of preference regression, coefficient estimates are not independent, and will generally be negatively correlated when the inter-attribute correlations are positive.  Thus, this split-half “vector reliability” cannot easily be adjusted to estimate the reliability of the estimates from the entire sample, but can offer an approximation of the relative reliability of the estimates of importance.

Does this make sense?  Has anyone seen this method used before, or is there a better method I should use?