Conducting A High Quality “Problem Solving” Interview

Problem solving interviews have been a mainstay in many industries for decades. Consulting firms and investment banks are well-known for using “business case” interviews to evaluate candidates’ analytic horsepower and structured thinking. Technology companies use live coding exercises extensively in evaluating software developers — in fact, many companies use little else when evaluating engineering talent.

Problem solving interviews have an important role to play in evaluating candidates for certain types of roles, but they do have limitations. They are notoriously prone to generating false negatives (rejections of candidates who would actually be quite capable of performing well in the job). This happens because the interview format is inherently contrived—few companies actually require employees to solve problems in a matter of minutes via a high pressure, 1:1 interaction and without any supporting computational tools or reference materials.  Therefore, these interviews are best used when the cost of false negatives is lower, e.g., there is an ample supply of candidates, or with more junior-level candidates don’t have a great deal of career history to evaluate.

Problem solving interviews can be quite useful when used appropriately, and I strongly recommend them as a supplemental lens to evaluate candidates who will be hired into an intellectually or analytically demanding role. There are several important guidelines for making them work:

• KEEP IT RELEVANT. Make sure the problem you are presenting to the candidate is highly relevant to the role they are interviewing for. Avoid random brain teasers or riddles—stick with problems that closely resemble those they can expect to face on the job. In fact, the best problem solving questions are often directly derived from actual work in the company. Not only does this ensure you are testing relevant skills, but it also gives the candidate some direct exposure to the kinds of problems your company solves every day.
• MAINTAIN OPTIONS. Ideally, choose a problem in which there are multiple potential paths to solve the problem. It is common in software development “coding” interviews to present a problem that can be coded up in several different ways, some being (for example) more flexible, scalable or efficient than others. This allows you to make finer distinctions between good vs. great candidates, and increases the probability that weaker candidates will come out of the room feeling they did a good job.
• USE PHASING. Whenever possible, structure the problem in distinct phases or steps, allowing you to see different elements of the candidate’s thinking process. Many such interviews (both business and technical) are set up three steps—(1) diagnose the problem, (2) design the best solution and (3) implement/execute on it.
• KEEP IT INTERACTIVE. Stick with problems that can be broken into components and discussed interactively. Encourage your candidates to explain their reasoning and talk through the tradeoffs they are making as they go along. Much of the data you will gather about candidates will come from their process, not just their solution. An overly “black box” approach robs you of this valuable information.
• CREATE SAFETY NETS. Make sure you have plenty of “nudges” handy for candidates who may struggle with one or more parts of the problem. Doing so ensures you avoid awkwardness with weaker candidates, and ensures all candidates come away feeling as good as possible about their performance. I have seen situations where a fairly simple nudge turned a potential crash-and-burn situation into a solid overall performance.
• MAINTAIN RAPPORT. In biographical interviews, great rapport comes from showing curiosity about the candidate him/herself. In problem solving interviews, that same curiosity can be channeled into your collaborative exploration of the problem with the candidate. Even though you may have presented this problem dozens of times before, it is important to maintain believable passion and interest throughout the interview. Similarly, be very careful not to express disapproval, judgment or frustration if the candidate gets off track—candidates can detect this and it can taint the entire hiring process.
• STANDARDIZE. Maintain objectivity and consistency by using a standard set of problem solving interviews across your team. Get clear on what constitutes top 10% vs. merely top 50% performance on a given problem, and track candidates’ performance over time to ensure you are well-calibrated. Periodically check Glassdoor to ensure your questions have not inadvertently leaked—this is a frequent problem that can create obvious bias.

Problem solving interviews are not the be-all, end-all approach to vetting candidates, even in roles with a heavy analytic or problem solving component. They are merely one of many acceptable interview formats, and should be viewed as a supplement to a deep understanding of your candidate’s unique career history. When misused, problem solving interviews can deliver “noisy” data and can create a bad candidate experience. But when used appropriately, they can provide insights into an individual’s intellectual capabilities and thought processes that no other interview format can match.

How Savvy Candidates Avoid Being Interviewed

Of all of the mistakes companies make in their hiring processes (startups in particular), there is one so simple and fundamental that it boggles the mind. The mistake is...failing to interview the candidate at all. And it happens far more often than you’d think.

Let’s say you’re the CEO of a promising technology startup that recently secured its Series A. You are desperate to hire a new VP Engineering to take the reins of the development team and deliver on your ambitious product roadmap. One of your new investors is able to convince an experienced and well-known technical lead from a top-tier software company to grab lunch with you. You review her LinkedIn profile, and it is...spot on! You meet with her, and immediately she is peppering you with questions about you, your business model, your competitors and your technology—impressive questions across the board. You deliver your responses in rapid succession, addressing every subtlety, and you come away with an agreement that she will drop by your offices to meet more of the team.

You spend the next day envisioning her on board. You imagine making the introduction to the rest of the team on her first day, her decisive voice and vision in your weekly product sync meetings, and the new inspiration in every face you see in the office. When you talk to your management team members about her upcoming visit, you find yourself spending most of that time prepping them for questions they should expect rather than questions they should ask. She comes onsite, and you and your team NAIL it. The rapport was right, the energy was high, and your team’s answers were consistent and compelling. And then, three days later...she calls to let you know she's taking another opportunity. What happened?

Star performers are turned off by an overly salesy process. They actually want to be vetted, and vetted thoroughly. Why? Because they want to work with other star performers—people who were rigorously evaluated on the way in. They don't want to waste valuable years of their careers working with average colleagues who were arm-twisted into joining a mediocre organization.

There is a surprisingly simple way to avoid this pitfall. When your hot prospect expresses interest in the opportunity, simply...ask them if they would like to interview for the job! It’s absolutely fine—in fact, it’s best practice—to sell like crazy when you are first engaging with a strong prospect. But at some point, you need to convert this lead into a candidate. Phrase it however you like, but in essence, you are saying: “I think you might be an excellent fit for this role, and it sounds like you share our excitement for the opportunity ahead of us. Would you like to start the interview process?” 

If they say yes, put them through your process. Ask them the tough questions! Push them, dive deep, showcase the depth of diligence you and your team are committed to delivering with every candidate. If they say anything other than yes, ask what they would need to get to that point. Follow through on whatever they ask, and then ask them again. If they stay in this half-in "limbo zone," walk away—it's much better to chalk it up as a missed opportunity than to try to bring in someone who isn't willing to be interviewed.

There is another benefit of taking this approach, one that plays on a deeper psychological mechanism called cognitive dissonance. Let's say I'm a top-performing candidate and you are interviewing me for your relatively unknown organization. You begin asking me direct questions about my capabilities and performance, and I respond. The moment I answer you I have to justify to myself why I am doing so. I have to come up with an explanation of why you are worthy of engaging me in this activity, which compels me to elevate your importance and attractiveness in my own mind. My (subconscious) thought process: "Wow, here I am, selling my skills and capabilities to this person. Why am I doing this? It must be because he is impressive, his company is highly selective, and this opportunity doesn't come around every day!"

So the next time you find yourself in front of a superstar prospect you want to attract to your company, remember they have two decisions to make—the first is to apply for the job, and the second (should they prove worthy) is to accept your offer.  And in between those decisions is a rigorous and engaging interview process that EVERYONE goes through.

Remember—NOBODY gets into your company without being interviewed. Your superstar team will thank you for it!

The Worst Kind Of Interview Question

Take a moment to think about your greatest weakness. That thing you hate doing, or that nagging item that has reared its head on many a performance review. Maybe you are hopeless with administrative details. Maybe you really hate sticking your neck out to “cold call” new prospects. Maybe you struggle to have tough conversations with underperforming team members. Do you have yours in mind? Great.

Now imagine that an interviewer asks you a hypothetical question that plays on that very weakness. For example, you’re the type who hates to deal with underperformers, and you are asked, “How would you deal with someone on your team who was clearly missing performance expectations?” Could you come up with a good story for this? Maybe one about how you would move quickly, decisively and objectively, putting emotions aside and doing right by both the company and the underperformer him/herself? Perhaps…a story that bears ZERO resemblance to how you would actually handle this in real life? 

Hypothetical questions are terrible, because they give a candidate a license to fabricate reality. The only value you can get is if a candidate gives a clearly terrible answer. A good answer, however, tells you essentially nothing. 

Consider this—what kind of candidates are best prepared with great answers to hypothetical questions? There are two types that come to mind most often. One is the active/desperate job seeker who has been asked this question 3 times in the past month. The other is the candidate for whom the subject of your question is a significant problem, and they have been given lots of coaching and feedback on the right way to do it. In other words—great answers to hypothetical questions can actually be a negative indicator of your candidate’s potential!

When I share this point in my workshops, I often hear resistance related to problem solving or “business case” questions. So let’s be very clear about the distinction here. A problem solving/business case question requires the candidate to solve a novel (and challenging) intellectual problem in front of you. This approach has its merits, though it also has risks and should not be overused (the subject of a future blog...) A hypothetical question is one that a candidate has a good chance of having encountered or at least thought through in his/her past, such that they can paint a fictional picture for you based on what they know you want to hear.

The Truth Is In The Standouts

Take a moment to recall your top three professional victories in the last five years of your career. That product you developed that vastly exceeded expectations. That new process you championed that saved your company millions. That smart-but-struggling team member you coached who became a top performer. Got it? Now unpack each victory, identifying the underlying traits, qualities and skills you exhibited that made each of them happen. Exceptional creativity and an ability to think like a customer? Check. Top notch analytic skills and intense loyalty to the cause? Check. Empathy and a passion for seeing others shine?Check. 

Review your list—if you are like most people, you have just created an 80-90% comprehensive list of your most important professional strengths—the most critical assets you would bring to any would-be employer. The same goes for your biggest mistakes by the way…if you list your three biggest whoppers over the past five years and what was behind each one, you should have a reasonably comprehensive list of your most important professional liabilities.

Now think about your top three victories related to generating revenue from new customers in the past 2 years. Hmmmm…a little harder, right?  There was that sales pitch you joined that went pretty well. There was that new role you created that made things a little more efficient in Marketing. And of course, that product tweak that slightly improved adoption amongst millennials. So…what were the traits or skills behind these “victories?” This list of skills/traits is likely to be far less comprehensive, and far less representative of who you truly are, than the first list.

What’s the point of all of this? When we interview, we should be seeking standout data—the most prominent events in a candidate’s career that really jump out as especially good or bad. These events are the ones that speak to our real strengths and limitations—and the way to get them is to ask broad, open-ended questions! For example, “What was your greatest accomplishment during your time at Google?” is a broad question that is likely to generate meaningful and highly-representative information about your candidate. Conversely, “What was the most difficult internal negotiation you had with a peer who resisted your influence?” is a far narrower question—one that may generate a less insightful, more contrived and less “representative” response from your candidate. In fact, the more "filters" you apply to your question, the more likely you are to hear a completely fabricated response!

It is tempting to put lots of filters on our questions to try to steer the candidate towards the skills and traits we care about. But this can lead to lackluster, uninsightful and mundane data points. A more powerful approach is to keep the questions broad to find out what makes the candidate really stand out, and narrow our questions only if we need to (e.g. to fill the few remaining gaps or unknowns in the story).