Site menu:

Switch to German language

cNORM - Troubleshooting and Frequently Asked Questions



cNORM works "out of the box" in many scenarios, especially with a sufficiently large amount of data and a sufficiently large share of variance explained by the explanatory variables (e.g. a strong age trend). However, there are also cases that are more difficult to handle. In any case, we recommend checking the plausibility of the generated solutions and adjusting the parameters "manually" if necessary. Similar to factor analyses, model selection is a multi-stage process in which different models usually have to be calculated and compared with regard to different quality criteria. In the case of cNorm, there are three main quality criteria, namely the explained variance, the simplicity of the model and the model plausibility or consistency. As a rule, the latter must be taken into account especially at the extreme ends of the model.


In a nutshell:

  1. A power parameter of k = 4 is sufficient for almost all scenarios.
  2. Models with a small number of terms (e.g. 'terms = 4' to 'terms = 10' in 'bestModel') are the most robust and usually produce the best results in cross-validation. Avoid models with a lot of terms and rather accept a slightly lower R2 instead.
  3. After an initial model calculation, inspect the information functions of the models ('plotSubset') and the percentile lines ('plotPercentiles' and 'plotPercentileSeries'). In a second step, calculate the model again using those parameters (e.g. a certain number of terms at given k) that you have previously determined to be optimal.
  4. For a data set with many finely graded age groups with small samples, it may be more convenient to combine the age groups into broader intervals.



1. Intersecting percentile curves

During modeling, inconsistent models may occur, especially in extreme areas, with strong floor or ceiling effects or in the event of over-adjustment. These are expressed, for example, by the fact that the lines of the percentiles overlap ('plotPercentiles') or that a corresponding warning is issued in the modeling ('checkConsistency').

Possible solutions:

# Creating a percentile plot

plotPercentiles(data, model) # Display the current model
plotPercentileSeries(data, model) # Display a sequence of models




2. How to deal with covariates

The question of whether separate norms should be generated for different groups of people in a population (e.g. men versus women, people with high versus low education ...) goes beyond the mere statistical modeling of the data. Please consider whether such a distinction makes sense for your scenario or not.

Possible solutions:

# Include a covariate, e. g. based on the integrated PPVT4 dataset
# with sex as the covariate (1 = boys, 2 = girls)

data1 <- rankByGroup(PPVT, group="group", raw="raw", covariate="sex")
data1 <- computePowers(data1)
model <- bestModel(data1)
plotPercentile(data1, model, covariate = 1) # Specify the degree of the covariate

# Often, covariates are excluded in the modeling, since there impact is
# much smaller than that of the explanatory variable. Force the covariate
# and all interactions to be included in the model (though might results in
# unstable models):

model <- bestModel(data1, force.in = c("COV", "L1COV", "A1COV", "L1A1COV"))


# Based on your theoretically motivated expectations of the influence
# of the covariate, you might want to include a specific covariate term,
# for example a linear term and the three-way interaction:

model <- bestModel(data1, force.in = c("COV", "L1A1COV"))


# Too plot these models side by side:

c(plotPercentiles(data1, model, covariate = 1),
plotPercentiles(data1, model, covariate = 2))



3. Necessary sample size for the modeling

Our simulations indicate that a group size of 50 cases per age group or even less can produce very reliable models (see figure). More important than a particularly large number of cases is that the total sample is as representative as possible. (Please note that random samples do not become automatically more representative as the number of cases increases.) However, if the sample size is too small, not only will the power of the statistical methods decrease, but it will also be difficult to stratify the sample with regard to all relevant characteristics. In addition, the number of age groups and the strength of the explanatory variables also play a role.


Comparison of cNORM and Conventional Norming in Dependence of Number of Items per Scale and Sample Size per Age Group

cNORM is able to partially compensate for violations of representativity or lower case numbers in specific age groups, since the models are calculated on the basis of the entire data set. Nevertheless, a very high-quality data basis should be ensured when standardizing a test procedure.

Possible solutions:




4. Low share of variance explained by the explanatory variable

Since version 1.1.8 cNORM automatically checks the share of explained variance by the explanatory variable in the data preparation (functions 'prepareData' and 'computePowers') by means of a polynomial regression (up to the power parameter k). Thus, not only linear, but also more complex relations are captured, usually up to the fourth power (basic setting). If there is only a small correlation between the explanatory variable and the raw scores, cNORM issues a warning message. In this case, the question arises whether the inclusion of explanatory variables in the calculation of standard scores really makes sense or whether, for example, a single norm score table is sufficient. With a low share of variance explained by the explanatory variables, it can sometimes happen that the models are not very stable and that the default settings for the model estimation of R2 = .99 is not reached.

Possible solutions:



5. Strongly meandering percentile lines in the 'plotPercentiles' function


Possible overfit

The model probably shows an overfit, e.g. because the share of variance explained by the explanatory variable is not very high, too many predictors were used, the scale generally does not measure reliably, there are strong violations of representativeness in individual norm groups, etc... In many cases, it is simply due to small sample sizes, resulting in an overfit of the model - as it is the case in the model above with n ≤ 50 per subgroup. It might as well be that the cNORM package is not suitable to model the specific dataset. Please note the suggested solutions of the previous point. In most cases it is recommended to use a simpler structured model with fewer terms.

Possible solutions:

Reduction of k to 3 in the example above results in the following model:


Vermeidung einer Überanpassung


6. Modeling the continuous age variable does not work

If you use the function 'prepareData' for data preparation and specify a grouping variable, a continuous age variable and the width of the age interval (parameter 'width'), the ranking is carried out using a sliding window. You can as well use the function 'rankBySlidingWindow' and then 'computePowers' instead. However, for the modeling to work, the values of the grouping variable must correspond to the age variable. This means: The values of the grouping variables should reflect the mean age of the respective age group.

Possible solutions:

# Generation of a consistent grouping variable with
# 4 groups on the basis of the age variable

data1 <- rankBySlidingWindow(data, age=data$age, width = 1, nGroups = 4)



7. Is it possible to apply weights to the cases in the norming sample?

Both in the ranking and in the model calculation with 'bestModel' a weighting of the cases can be done with the parameter 'weights'. With computeWeights(), cNORM contains a specifically designed function for the post-stratification of the data on the basis of raking. This can reduce the effect of unbalanced norm samples. Please refer to the vignette "Weighted regression".



Jamovi
 Overview