Site menu:

Discrepancy comparisons and confidence intervals for psychometric test results

The site contains several tools for analyzing psychometric test results, such as calculating confidence intervals and comparing discrepancies.

1. Confidence intervals of psychometric test results
2. Test for difference from fixed value
3. Test for differences in repeated measurements (Reliable Change)
4. Discrepancy of two test results of one person
5. Profile analysis: test for identity, structure and profile height of two profiles
6. Estimated Total Score of a Sum of Correlated Scales
7. References

1. Confidence intervals of psychometric test results

Psychometric test results are not absolutely accurate. Confidence intervals are estimates of the range in which the true value lies with a certain probability. The following calculation determines the confidence interval based on the standard error of estimation. The result can also be corrected for the effect of regression to the mean. In this case, the estimated value is also displayed.

The calculation is performed using the following formula (correcting for regression to the mean with estimated value z_predicted = z_score * rel; for standard estimation errors, see Krum, Amelang & Schmidt-Atzert, 2022, p. 149):

In the case of percentiles, these are first converted to z-values and then the interval limits are recalculated. Note that the procedure is only useful for reliability parameters > .5. For lower values, the standard measurement error is used.

2. Test for difference from fixed value

If you want to test against a fixed value, it is sufficient to use the one-sided confidence interval and test with z_1-α (Krum et al., 2022, p. 151 f.). However, it is still necessary to specify the direction in which the result is to be secured. It also depends on the exact formulation of the hypothesis. Because of the higher accuracy of the calculation, the regression to the mean correction should be applied again.

For example, if a person scores an IQ of 135, it can be investigated whether the result is significantly higher than a value of 130, which is considered the threshold for giftedness. In this case, the score would have to be significantly higher than 130 (the direction must be "... is higher than..."). Another question may be whether giftedness can be ruled out, for example, if the result is 122. In this case, one would hedge upwards (direction "... is lower than...") and require a non-significant result, which would correspond to the statement that giftedness cannot be ruled out.

The test is one-tailed with the standard error of measurement using the following formula (correction for regression to the mean is made with the estimated value z_predicted = z_score * rel):

In the case of percentiles, these are first converted to z-scores and then the interval limits are recalculated.

3. Test for differences of repeated measurement (Reliable Change)

.

When a test is repeated on an individual, the so-called Reliable Change Index (RCI; Jacobson & Truax, 1991; see also Krum et al., 2022, p. 153) can be determined. The RCI can be interpreted as a test variable in a z-test. It can be used to express whether there are significant differences between two test scores, e.g. whether an intervention has led to a significant change in characteristics.

As in Calculators 1 and 2, the percentiles are converted to z-scores per inverse cumulative normal distribution prior to the calculation. The formula for calculating the RCI is based on Jacobson and Truax (1991; see Krum et al., 2022, p. 153):

4. Discrepancy of two test results of a person

.

When a person is tested with different tests or scales of a test, it can be interesting to compare the results. For example, one might want to investigate whether logical reasoning is better developed than verbal comprehension, if intelligence tests do not already provide such analysis options. Or one might want to clarify whether the stress levels of different clinical symptoms differ.

As with the previous calculators, percentiles are converted to z-scores by inverse cumulative normal distribution before calculation. In general, the procedure is also suitable for raw values, provided that the population mean is known. This is given per se when norm scores are used. For both test results, a value Y_i must first be calculated. Then the test statistic z can be determined. The formulas for calculating the test statistic to compare the test results (Krum et al., 2022, p. 154f.):

5. Profile analysis: test for identity, structure and profile height of two profiles

Profiles of psychometric results can be analyzed regarding equality (= profile identity), structure (= profile shape), or magnitude (= profile height) (cf. Huber 1973, chap. 10). Such an approach can be applied, for example, to intelligence profiles or the clinical stress spectrum before and after therapy. Huber (1973, p.) gives as an example the result of a 35-year-old person in the Intelligence Structure Test (IST; Amthauer, 1953) with the following results:

Results are reported as standard scores (M = 100, SD = 10), and subtests have a uniform reliability of r = .9. Results can now be tested to determine whether the two profiles are identical and, if not, whether they differ in height, structure, or both.

6. Estimated Total Score of a Sum of Correlated Scales

If the results of several scales point in one direction, for example, if they are all above or below average, then the overall result is more extreme than the average of the individual results. The following form estimates the overall result based on the sums of the individual scales. The (average) correlation is also required. Please enter the individual results separated by spaces, specify the type of scale (T-, IQ-, z-score, or percentile rank), and enter the (mean) correlation. Averaging of correlations can be performed here. Caution is advised if the correlations are very inhomogeneous.

Use case: If all scales in an intelligence test have high or low scores, the total IQ is even more extreme. This is the result of the intercorrelations between the scales. This effect can be used: If different scales correlate only moderately with each other, such as the various subjective assessments in ADHD (the mean correlation of parent, teacher, and self-assessment is approximately r = .3), then it is possible that neither the individual scales nor the average of the scales will reach a critical threshold value. If the intercorrelation of the scales is taken into account, the overall result is more extreme and may reach the threshold required for a diagnosis.

The variance of sums or difference values of correlated variables is calculated from the sum of the variances plus the covariance multiplied by the correlation. In the case of standardized variables, the calculation can be simplified (for z scores with SD = 1):

$${\mathrm{SD}}_{\mathrm{Total}}=\sqrt{k+k(k-1)\overline{r}}$$
To predict the total value, the sum of the results is divided by the standard deviation of the sum.

References

Chi² tests are conducted with the help of jStat.

• Amthauer, R. (1953). IST-Intelligenz-Struktur-Test. Hogrefe.
• Huber, H. (1973). Psychometrische Einzelfalldiagnostik [Psychometric diagnosis for single cases]. Beltz.
• Jacobson, N. S., & Truax, P. (1991). Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12-19.
• Krum, S., Schmidt-Atzert, L., & Amelang, M. (2022). Grundlagen diagnostischer Verfahren [Basics of diagnostic procedures]. In L. Schmidt-Atzert, S. Krum & M. Amelang (Eds.), Psychologische Diagnostik (6th ed.) (pp. 39-209). Springer. doi:10.1007/978-3-662-61643-7

Citeable source:
Lenhard, W. & Lenhard, A. (2023). Confidence intervals, test of discrepancy and profile analysis for psychometric results. available: https://www.psychometrica.de/discrepancy.html. Psychometrica.

Copyright © 2023 Drs. Alexandra & Wolfgang Lenhard