Home > Validity: The Long and Short of It > Paper 3: Scales of Potential

Paper 3: Scales of Potential

This paper looked at hierarchical models of workplace competencies to establish the most useful level of the model for recruiters to maximize validity, and to provide a practical level of detail to identify distinct behaviors which are important for the role.

We did this by exploring the cross-validation of equations to forecast potential across three studies. The studies involved individuals with self-report personality questionnaire scores (from Wave Professional Styles) and independent ratings of matched behaviors and overall performance, including ‘Demonstrating Potential’. The Wave Professional Styles and Rater questionnaire assess the Wave hierarchy of 108 facets, 36 dimensions, 12 sections and four clusters.

Three samples were used in the study to attempt to answer the following questions:

  • Which level of the hierarchy performs the best in forecasting potential?
  • Which level of the hierarchy provides the greatest level of convergent and discriminant validity for measuring specific behaviors?
  • Which is the most useful level of the model for the recruiter in practical terms – balancing the above considerations?

 

Algorithms were created for each of the four scale levels (in each of the three samples) to compare how well we can forecast potential based on weighting different levels of detail. To create the equations, we correlated every scale with ratings of ‘Demonstrating Potential’ then ranked from best to worst at each level of the model – with the highest ranked receiving the biggest weighting and the lowest weighting applied to the scale with the lowest relationship with potential.

After analysis of the data, we were able to conclude that there is good forecasting at all levels of the Wave hierarchy, with slightly stronger relationships generally demonstrated at the more detailed scale level (as opposed to the broader four cluster level). The equations tended to perform better in the sample they were based on (as we would expect), but there were significant and substantial correlations for the equations cross-validated into other samples. This provides clear evidence for the generalizability of this method of creating weightings based on ranking correlations for forecasting potential.

The section level demonstrated the highest level of convergent validity (for forecasting specific behaviors) in all of the samples. Discriminant validity compares the matched, convergent validity correlations with the unmatched correlations. These were consistently lower across all levels and samples, indicating that the scales are measuring the specific behaviors they are designed to measure, rather than just being generally related to performance.

We found good levels of forecasting across the hierarchy. Forecasting potential was slightly stronger at the more detailed level, but it was noted that this can be less practical to work with and more time consuming for job-analysis. The mid-level (section level of Wave) performs well when it comes to predicting potential while balancing ease of use. By achieving good forecasting of potential while maximizing convergent validity, the sections can be particularly useful for streaming people into different jobs, flexible role fit and forecasting of specific behaviors which are important for the role.Based on this research, as well as the research discussed in earlier papers regarding the importance of both validity and time to candidates, we wanted to explore how we could utilize the validity benefits of working with the section level of the hierarchy to create a powerful assessment which delivers high validity in a short completion time.