Fascinating and thought provoking content Andrew. Clearly feeds into Talent Density which is something we spend a lot of time and effort optimizing for, but is a concept that opens up a lot more methodological questions than we can hope to cover in a comment section.
Still if you possibly could I would love to hear how your model and thus the distortion and noise within the data being reviewed would respond to adjusting two specific elements.
1. Predictive Validity of hiring assessment/selection period raised from the typical .40 modelled to a .70 that elite well architected talent acquisition practices can achieve.
2. Replacement of Likert or similar scoring systems with what I call Outlier Analysis for Performance Evaluation - multi criteria (typically 5-10 and multi-callibrated within department and by employee grade/level) but only 3 scoring options 0=Clear evidenced example(s) of standard not being met / 3=No clear data points for standard not being met or significantly exceeded / 5=Clear evidenced example(s) of standard being significantly exceeded (also was advised that for assessing the reliability of evaluation scoring done this way Cronbach"s alpha is best replaced with McDonald's omega - would you concur?).
I do agree that using a multi-criteria approach is the way to go. As a someone with Bayesian tendencies I do admit I rarely use something like Cronbach’s Alpha and successors (I hate single figures instead of distributions) - it terms of looking at the structure I am a big fan of partial correlation networks (which of course are not Bayesian but do at least paint a more nuanced picture than a single metric) A Bayesian probably treats the real performance value as a latent variable related to the observed value and a measurement error.
The point being it gets messy quite quickly but there are ways of making it a bit less messy.
There are ways of dealing with noise and I suspect redesigning processes to reduce that would be a great first step. As others have mentioned, observed performance probably has noise AND bias.
Ultimately we do have to admit that measuring performance is a good example of Goodhart’s law - “When a measure becomes a target, it ceases to be a good measure”
Fascinating and thought provoking content Andrew. Clearly feeds into Talent Density which is something we spend a lot of time and effort optimizing for, but is a concept that opens up a lot more methodological questions than we can hope to cover in a comment section.
Still if you possibly could I would love to hear how your model and thus the distortion and noise within the data being reviewed would respond to adjusting two specific elements.
1. Predictive Validity of hiring assessment/selection period raised from the typical .40 modelled to a .70 that elite well architected talent acquisition practices can achieve.
2. Replacement of Likert or similar scoring systems with what I call Outlier Analysis for Performance Evaluation - multi criteria (typically 5-10 and multi-callibrated within department and by employee grade/level) but only 3 scoring options 0=Clear evidenced example(s) of standard not being met / 3=No clear data points for standard not being met or significantly exceeded / 5=Clear evidenced example(s) of standard being significantly exceeded (also was advised that for assessing the reliability of evaluation scoring done this way Cronbach"s alpha is best replaced with McDonald's omega - would you concur?).
What a fantastic comment. Thanks a lot.
I do agree that using a multi-criteria approach is the way to go. As a someone with Bayesian tendencies I do admit I rarely use something like Cronbach’s Alpha and successors (I hate single figures instead of distributions) - it terms of looking at the structure I am a big fan of partial correlation networks (which of course are not Bayesian but do at least paint a more nuanced picture than a single metric) A Bayesian probably treats the real performance value as a latent variable related to the observed value and a measurement error.
The point being it gets messy quite quickly but there are ways of making it a bit less messy.
There are ways of dealing with noise and I suspect redesigning processes to reduce that would be a great first step. As others have mentioned, observed performance probably has noise AND bias.
Ultimately we do have to admit that measuring performance is a good example of Goodhart’s law - “When a measure becomes a target, it ceases to be a good measure”