What to ask when interviewing software engineers

TL;DR: Most companies use the wrong metrics during software engineering interviews. That makes them both brutal for the candidates and inefficient for the companies. Both sides suffer as a result. In this blog post, we propose how to improve your hiring loop by using the right metric.

The problem of using the wrong metric

Before we dive in, let’s recap how a typical hiring loop looks and the rationale behind it. It usually consists of several technical interviews, followed by “meet the CEO” ones. The technical interviews can take different forms: from universally loathed on-site whiteboarding to pairing sessions using online coding solutions. The unifying theme in all those forms is that the candidate must solve all the puzzles the interviewers could come up with for the interview and do that under pressure. Ultimately the decision of whether to extend the offer depends on the candidate not making any technical mistakes.

In other words: can the candidate survive the interview.

The metrics one would get from such interviews are:

• The ability to recall the implementation details of some algorithms or data structures. A rarely used trait outside of the interview because you can always look those things up. Some companies are outright recommending “Cracking The Coding Interview” as a pre-interview practice. Let’s face the facts: how many times have you referred to that book while working? This ability is not a good predictor of whether the candidate will generate value. Learning recipes by heart doesn’t guarantee a good chef. Memorizing traffic laws doesn’t make you a safe driver. Why do you think remembering programming puzzles will produce a great software engineer?
• The ability to imagine outcomes. Questions like: what if the app has 10x more clients? What if memory pressure increases? What interviewers hope to measure is building or system design skills. Instead, they measure imagination, which is an entirely different trait. Of course, experienced candidates can share more war stories of things going wrong. But, unless you test the candidate’s design with an actual 10x load, all you get is speculation.
• How much the candidate’s thinking process aligns with that of an interviewer. They ask the same questions to many candidates and expect the usual answers. If they encounter a unique solution, they have to counteract their confirmation bias. As a brilliant Barometer legend shows, this spawns further than software engineers’ interviews. And thus, the candidate’s ability to deliver value remains unmeasured.

Typical hiring loops can generate a sporadic great hire or two, but they are not consistent in doing so. Like a coin toss, sometimes they yield the opposite results, and making a bad hire is an expensive mistake. Burned by it, the company will invest more resources trying to protect itself. The common-sense approach is to pause and investigate why the hiring loop produced the mistake in the first place. However, the hiring loop is usually the last to suspect — after all, it brought the company to its current state, so it must be working, right? That simple conclusion cloaks the hiring loop from improvement, and it remains deficient. Instead of implementing the constructive changes, the company raises the bar to entry. They start using trickier puzzles or asking the candidates to design more complicated systems, sometimes while still operating within the time frame of a typical one-hour on-site interview. That undoubtedly makes interviews more brutal and harsh. Fewer candidates survive them, and the company sees it as a positive change.

But that neither improves their filtering nor protects them from bad hires. They get the same data points, again and again, never improving the signal-to-noise ratio. If they raise the bar too high, they throw the baby out with the bathwater as they start to pass on talent. All those metrics from the above become a single one: can the candidate avoid making mistakes. Errare humanum est, remember?

If the company does that for too long, then the hiring decisions become protective and based on fear. Instead of highlighting the value that the candidate brings to the table, the interviewers will seek a reason to reject them. If they couldn’t find it, they will resort to biases and personal inclinations. Training interviewers to reject the candidates will only produce more efficient rejections.

Tell me how you measure, and I will tell you how I will behave.

Eli Goldratt, The Haystack Syndrome

Improper metrics at the core of your hiring loop will make it yield inefficient results. To fix that, we need to address the root cause and base the loop on a better set of metrics.

A better set of questions

We have refined three questions that determine a practical hiring decision for a software engineer:

1. Can the candidate do the actual work?
2. Can we work together?
3. Does the candidate want to work with me?

This set is not a silver bullet that will drive your hiring loop from zero to success. It is neither a secret knowledge nor a magic trick. Instead, it is the common sense behind a hiring loop based on a proper metrics. Let’s explore them in-depth so that you can build a similar hiring loop at your company.

Can the candidate do the actual work?

Or, as we prefer to frame it: can the candidate deliver something that works in production.

As simple as it looks, this is the main and the first thing you need to measure about the candidate. Your company is there to build your product, not solve riddles on a whiteboard. Your hiring loop should select for the ability to contribute to the former, not the latter.

It is hard to measure the ability to perform without letting candidates do the actual work. Companies that can’t measure it resort to proxy metrics and instead try to predict the future. They check what they can, and they hope the correlation will be on their side. Google used to ask ridiculous brain teasers (and disgorged a horde of imitations). They stopped doing it because it turned out to be completely useless for hiring. Surprise? No, not at all. Just another example that using improper metrics will yield unsatisfactory results.

The best way to check that someone can do real work is to allow them to do real work. That is also the first thing you need to check before you engage in more expensive interviews. The good news is that it is possible to do both.

Can we work together?

Or, in other words, the cultural fit question.

This metric is orthogonal to the previous, and you should measure it separately. There will be candidates who can deliver exceptional code but who can’t cooperate. And there will be candidates who fit your culture like a glove yet can’t ship a single line of code to production. An efficient hiring loop accurately handles both of those extremes and anything in between.

Companies usually try to measure this trait with “meet the CEO” style interviews. That is not enough, as it is prone to unconscious biases of the person performing the interview. To measure this metric correctly, you would need to do some self-reflection first. You will need to understand what is the culture that you are trying to build.

Culture is the Behavior You Reward and Punish

Jocelyn Goldfein

The tough aspect of this measurement is that it is difficult to automate. It will likely involve expensive human time, which is expensive to scale with the demand. Therefore, we recommend performing this measurement at the end of your hiring pipeline. This way, it will not become a costly bottleneck.

Does the candidate want to work with me?

The ultimate decision to join your company belongs to the candidate. They are the ones who will accept the offer or reject it. Interviews work both ways, and this phase of the loop is when the candidate measures your company. The experience you provide to them during the interview will be with them for a long time. They will share it with their friends and colleagues, tweet about it, and discuss it online.

They may forget what you said, but they will never forget how you made them feel.

Carl W. Buehner

Invest in giving them the best experience possible. Your company will stand out by being friendly, transparent, and approachable. If you want to be bold, allow them to interview you, and do not judge their questions, but answer them truthfully. Let them relax and be themselves, and you will help them picking you for their next adventure.

Conclusion

Basing your hiring loop on the right metric could be what turns a stagnating company into a thriving one. That will allow you to start bringing the best talent aboard consistently. At this point, you no longer bet on a coin toss.

The next step is to automate the assessment and put it at the beginning of your hiring loop. That will allow your interviewers to focus only on candidates that can deliver. At this point, you no longer waste resources.

What’s left is to concentrate on building the best culture you aim to have and stand out.

Marketing pitch

Okay, folks, here’s the favorite part of our blog posts, The Marketing Pitch™

AutoIterative platform solves the measurement of software engineers’ ability to do the actual work. It provides the production environment where the candidates ship their code, and it performs an automatic assessment of whether their solution works according to the requirements.

You can build your custom equivalent of it, or you can evaluate the fully functional demo account for free and start focusing on your culture immediately.