The Bigotry Of Dunning-Kruger
Dunning-Kruger is a very seductive and compelling idea. When I first learned of it in 2016, writing my magnum cringetopia opus, I had the two reactions I’ve come to observe most people having when they learn about it. “This is so true, I was so stupidly overconfident about X back when I knew much less” and “This is so true, the people who are stupidly overconfident about X are invariably those who know the least”. Unfortunately, the Dunning-Kruger effect also applies to my knowledge of the Dunning-Kruger effect. I’ve never questioned the basic assumptions I’ve made about this; 1. Incompetent people tend to be highly confident, 2. Highly competent people will be less confident than the incompetent 3. When I was incompetent and confident it was because it was a temporary snapshot of my journey along the knowledge curve, but when others are incompetent and confident it’s because they’re just permanently incompetent stupid people, and 4. Humility is a virtue.
Take the actual Dunning-Kruger result in its entirety though:
The competent students underestimated their class rank, and the incompetent students overestimated theirs, but the incompetent students did not estimate their class rank as higher than the ranks estimated by the competent group.
It follows that assumption 1 is true, but assumption 2 is false. Incompetent students did not magically leapfrog the competent in their self-assessment, so in the graph the ‘trust me it’s complicated’ at the very minimum should be at a higher Y-axis confidence value than the ‘I know everything’. Instead it appears that the incompetent think they’re smarter than the competent, which is just not true. If I transform the variable of confidence into displacement of expectation from the actual values, where exactly on the competence continuum would I switch from +ve values (overestimation) to -ve (underestimation)? Perhaps students can’t reach the degree of competence (say a genetics expert with 40yrs experience) it might take to switch back to +ve again, which further skews the suggestion that the competent underestimate, when we have more than enough real world evidence that expertise often goes to the head. There is some inherent narrow-minded prejudice in the simplistic depiction of this graph and the resultant unattractive sneering that smart people self righteously engage in while dumping on the stupid who are too stupid to realize they are stupid. This is exemplified in assumption 3.
From personal experience, I have empathy for the overconfident ill-informed and find it unhelpful that lack of knowledge, either absolute or self-assessed, is punished. What I’d be more interested in is the Dunning-Kruger effect applied to a Bayesian updating of prior beliefs based on new information. So if you tell me the original study was tweaked such that students were given information about their percentile and yet the incompetent students chose to ignore it and stubbornly overestimated their abilities, then that’s a phenomenon that at least warrants some negativity. It seems to me that when the Dunning-Kruger effect is brought up, on Reddit for instance, it seems to indicate that the latter is what is being mocked whereas the actual study has little do with the ability to change one’s mind given contrary information.
In essence it’s become a socially acceptable way of calling someone an illiterate buffoon. It doesn’t matter if you’re clearly speaking to someone who couldn’t afford to go to an expensive university or spend hours on Reddit playing political compass quizzes to iron out their inconsistent bouquet of views into a respectable homogeneous monolith. It suddenly doesn’t matter that IQ is highly correlated with socioeconomic status, and education even more so. Just don’t call them brainless filthy casuals, say the Dunning-Kruger effect is in play here, then chuckle sensibly into your embroidered pocket square and collect the upvotes.
Assumption 4, of humility being a virtue, is an awkward side effect of the inherent bigotry against the intellectually challenged. One cannot claim humility is unnecessary at high levels of competence without somehow accepting that D-K is basically saying with gratuitously condescending tautology that low-IQ people are stupid. Otherwise it isn’t at all self-evident that humility is good. Consider that it is a very fashionable insight across science and business that smart amateurs might see lateral solutions that experts have been stuck with for a long time. Innovation and creativity thrive on a little randomness of knowledge and experience. We build diverse teams in the workplace hoping that manufacturing insights come from the sales team and marketing insights come from the engineering department. We recognize the importance of connecting ideas, left-field hypotheses and unconventional solutions, all of which imply a person exhibited high confidence in a field of expertise where he is thoroughly incompetent. There’s an underlying assumption here: overestimating your specific competence is fine as long as you’ve displayed some sort of general competence elsewhere ie general intelligence.
Ultimately if our goal is to maximize competence/knowledge and accelerate one’s journey along that curve, a variant of Dunning-Kruger I would like is to see how every unit of knowledge changes the rate of seeking/accrual of new knowledge. I know I’ve just spent a few paragraphs tearing down the curve as having nothing to do with the actual D-K effect, but for visual simplicity if I follow the D-K curve, I’d assume high confidence slows down the seeking of new knowledge and therefore rate of change of knowledge is very low, whereas low confidence nadir at ‘this is starting to make sense’ will have the highest rate of seeking and building new knowledge. The former is bad, the latter is good. But what about ‘I’m never going to understand this’, that’s arguably worse than ‘i know everything’, where at least there is the high self-esteem and regard to motivate learning in the few remotely associated ways a high confidence expert at ‘trust me it’s complicated’ continues to be motivated to keep learning, all of which are absent at the low-confidence low self-esteem quadrant, where we’d have disillusioned dropouts rather than motivated learners. If anything, I’d begin with a hypothesis that high humility / low confidence decreases the rate of acquisition of knowledge.
This is a highly biased post that came out of a feeling of insignificance while reading Hayek, wondering if there was any value in amateurs like me trying to have independent thoughts when reading difficult texts. I’ve settled on the nice comfy answer that there is indeed value, that my own learning journey is made easier and richer if I proceed with the intellectual hubris that I might have just solved the unsolvable problem of market socialism. But if this is true, why is it not true for everyone, and why is it only true in the private sphere of one’s study? Narrow-mindedness and intellectual stagnation is the real enemy, not hubris and unearned overconfidence. Rate of growth of knowledge is the real virtue, not humility and self-effacing underestimation. Another pointless reactionary virtue signal that I blame Trump for sparking into the zeitgeist.
