Reliability

CSD 367K - Spring 1996

Reliability relates to the generalizability, consistency, and stability of a test.
  1. Test-retest reliability - Do the scores from two administrations of the test (usually about 2 weeks apart) correlate highly?
  2. Split-half reliability - Do the scores from two halves of a test correlate?
  3. Interscorer (interrater) reliability - Two examiners score the same set of tests. Do the scores from two examiners correlate?
  4. Intrascorer (intrarater) reliability - An examiner scores a set of tests, then scores them again later. Do the scores from time 1 and time 2 correlate?

Accounting for Test Error

One reason for obtaining a reliability coefficient is to estimate the amount of error that is associated with either test-retest or split-half reliability. We know that any two administrations of a test frequently result in different scores. How can we take that into account when we only test a student once? There are two ways:

  1. Estimate the child's true score. The estimated true score equals the test mean plus the product of the reliability coefficient and the difference between the obtained score and the group mean.

    ETS = M + [reliability * (obtained score - mean)].

    Example.
    Mean deviation quotient = 100
    Obtained Score = 75
    Reliability coefficient = .90

    100 + [.90 * (75-100)] = 77.5

  2. Use the Standard Error of Measurement (SEM), which is 1 standard deviation of error, to create a confidence interval. For a given probability, a confidence interval is the range within which the child's true score will occur if he was to take the test over again. 1 Standard error of measurement above and below an estimated true score is the 68% confidence interval. That is, you can be 68% sure that the child's true score will fall within that interval if he was to be tested again.

    If you want a 90% confidence interval, you multiply the SEM by +1.64 and -1.64. Add and subtract those values to and from the estimated true score.

    If you want a 95% confidence interval, you multiply the SEM by +1.96 and -1.96. Add and subtract those values to and from the estimated true score.

    Example.

    Estimated true score = 77.5
    Standard Error of Measurement - 3

    68% confidence interval = 77.5 +/- 3 = 74.5 - 80.5
    90% confidence interval = 77.5 +/-4.92 = 72.58 - 82.42
    95% confidence interval = 77.5 +/- 5.88 = 71.62 - 83.38
    Last updated February 10, 1996

    Created by: Ron Gillam

    Send comments to: lizp@mail.utexas.edu

    Syllabus | UT Austin Web Central