fix: 7+ concerns w lerner's analysis of certs
certs can be great for your job search even while correlating with weak DS&A perf
Aline Lerner is the founder of interviewing.io. They recently released an analysis of certifications and drew two conclusions that I find problematic:
“generally speaking, certifications carry a negative signal”
“People often tell me that @interviewingio should create a certification…I thought it was a terrible idea. We finally ran the numbers, and it is.”
These and other claims can be found in her article here. I originally heard about this through the tweet below:
The current article is a critical response to Lerner’s article. Before I begin, I want to be clear that I continue to admire both Lerner and the interviewing.io platform. I greatly value all of their work including the very article that I am responding to. I have used the platform to success and I recommend it as part of my standard software engineer career guidance.
This response organizes critiques into four high-level sections including a positive alternative and additional, smaller points within each section:
Misinterpretation of Results: Within Lerner’s own analytical framework, their conclusion not only fails to follow but in fact an inverse interpretation is more appropriate.
Variable Bias: Lerner’s analytical framework includes questionable elements and leaves out other important elements.
External Inconsistency: Lerner’s results are at odds with existing academic work, current industrial data, and common practice.
A Suggested Alternative: Lerner’s conclusions are comparatively dispreferred to a suggested alternative interpretation that harmonizes her data with external data and is robust to the other criticisms laid out.
I. Misinterpretation of Results
Lerner’s conclusion is that her company should not create a certificate, and, further, that displaying certificates on a LinkedIn profile causes a negative signal to recruiters. She draws this conclusion mainly from the negative correlation between people displaying certificates on their LinkedIn profiles and their technical interview pass rate on interviewing.io.
First, the causal claim is unsupported by the data. The claim that “recruiters will start to develop negative associations” is entirely disconnected from the collected evidence. It’s simply tacked on. As we will see later in this article, it’s also inconsistent with external evidence.
Second, the negative correlation is a feature of the average certificate, not a feature of all certificates. Indeed, her data actually shows that some certificates positively correlate with performance. The recognition of high-pedigree and low-pedigree credentials would be internally consistent with Lerner’s treatment of candidates and university credentials. It would also be more consistent with the external research on credentials. The idea that certificates on average present a negative signal is highly controversial. The idea that low-quality certificates present a negative signal is not.
Third, granting Lerner’s operationalization of the technical interview pass rate as the sole driver of candidate value. If we grant this, it becomes immediately clear that interviewing.io should create its own certificate! They would be the only provider of a certificate with perfect signal, in contrast to all other certificates. This would be a massive business opportunity.
II. Variable Bias
Now let’s critique the operationalization of the technical interview pass rate as the sole driver of candidate value. Lerner’s own analysis shows that certificate holders are disproportionately from a non-traditional background. We know recruiters and corporations actively seek out these candidates to support diversity initiatives. This is the main form of omitted variable bias worth noting, and it calls the main conclusion into question by itself, but there are two cases of included variable bias also worth noting.
The first form of included variable bias is the bias in the variable of technical performance. Optimizing solely on technical performance is expected to preferentially select competitive programmers. There’s good reason to think this is not an ideal way to select an employee. Personality research has found an inverse relationship between competitive personality orientation and emotional intelligence. We know emotional intelligence is key to team effectiveness and individual contributor effectiveness in the long run. Correspondingly, Peter Norvig famously claims that “being a winner at programming contests was a negative factor for performing well on the job” according to an analysis of Google data.
The sole dependent variable in Lerner’s analysis is a concern for bias, as is the sole independent variable. Suppose that Lerner can correctly model recruiter preference. It doesn’t follow that candidates shouldn’t display their credentials. Active job seekers optimize on job search performance over time, and recruiter outreach isn’t even the main route to interview for most active job seekers. Even if it were, maximizing recruiter outreach as a candidate is fundamentally different from maximizing recruiter preference.
Lerner’s data shows that there is a difference in the technical performance of about 5% between certificate presenters and non-certificate presenting candidates. LinkedIn has long held that certificates result in “LinkedIn members with certifications receive 6x more views to their profile.” If technical performance perfectly correlates with recruiter outreach rate, then candidates should absolutely display certificates, because a five-fold-plus gain dominates a five-percent reduction many, many times over.
The study would further benefit from the inclusion of four other variables, two of which have already been hinted at:
Credential prestige
Heterogeneous employer preferences
The number of credentials, rather than a binary contrast between any and none.
A clear comparison between non-college graduates with and without certification.
The pedigree controls were nice but nonidentical to this concern and a bit opaque. Pedigree controls conveyed controlling for the prestige of university education. If the same variable also attempted to capture a lack of university education, it is mixing concerns in an unusual way compared to academic standards. The level of education and the prestige of a university are typically treated as two distinct variables for very good reasons.
III. External Inconsistency
If critiques 1 and 2 are meaningfully true, we should expect Lerner’s main result to be at odds with external research. This is exactly what we find.
High-prestige university credentials improve labor outcomes like wages and employment rate.
High-prestige alternative credentials like coding bootcamps are associated with improved hirability.
Lerner’s own data, if not her conclusion, show that high-prestige certificates are positive signals.
Beyond inconsistency with the academy, Lerner’s results are inconsistent with common practice, recruiter expectations, and industry data.
Common practice is to show your certifications. Recruiter expectations are such that if they are targeting a role that requires certification, such as an AWS certification, qualified applicants should attach those to both their resume and their LinkedIn profile. Industry data indicates that some credentials are valuable, and the prima facia induction is that a valuable credential is also a valuable signal.
Here’s a quote from a recruiter named Connor Leech who gave a top-voted Quora answer on this subject. Technical interview performance expectations aren’t even mentioned in his perspective:
For me as a recruiter online certifications themselves aren’t enough to qualify a candidate but they do have value. Online learning outside of someone’s 9–5 shows they care about their craft and are passionate and interested in getting better at it. That’s a positive signal for most recruiters and companies.
As one example of credential value, Jefferson Frank found that professionals received a 27% percent salary increase on average after achieving certification. This means the credential is valuable. Is the idea that it’s valuable and yet should be hidden from the LinkedIn profile? That would be strange. More likely, AWS-certified individuals have spent time studying for the AWS exam instead of studying for technical interview prep.
It’s a tradeoff. Individuals in these roles do a bit worse in leetcode because they spent time investing in other skills. AWS professionals are signaling a different skill mix, not a strictly inferior skillset, and specific recruiters and employers are directly searching for these skills. Jefferson Frank, for example, is a recruiting agency specializing in AWS roles. Hiding your cert would disable them from finding you as a candidate.
IV. A Suggested Alternative
The essential nature of the current article is a critique, but I want it to be a positive and constructive critique, so I’ll end by presenting two alternative interpretations of Lerner’s data:
Credentials on LinkedIn operate the same way credentials in general operate. High-prestige credentials are a net positive signal on average, but most credentials are not prestigious.
The interesting question is about which credentials are high-prestige, and I thank interviewing.io for shedding some light on that.
I don’t agree that the cutoff for prestige should be the point of zero association with a difference in passing the interview. I think we have good reason to believe that LinkedIn skill certifications are a net benefit for job seekers. However, I think we can utilize the general scheme of analysis and simply shift the cutoff point. To be concrete, HackerRank and Cisco certifications seem like generally bad ideas. Maybe an individual who is particularly passionate about a Cisco-related role would even still want to present such a certification, but the average candidate would seem to harm their job search by displaying such a credential.
Credentials communicate a mix of skills. Candidates should be strategic in their display of credentials to support an efficient job search with respect to time, role, employers of interest, and personal brand.
As an example, Microsoft certifications signal higher prestige than LinkedIn certs on this analysis, and we have reason to think LinkedIn certs are a net gain to the job search. At the same time, we know that displaying a Microsoft Access certification will tend to result in contact for MS Access-related roles, so we would not want to display this cert if we are not willing to accept being contacted about such roles.