How to Create a Good Problem-Solving Interview
Things I wish I'd known when I started interviewing people. This article is part one in a two-part series.
"Hey, wanna interview someone?"
You are developer and one day your manager comes to you, asking if you would be willing to take part in the recruitment process of the next member of your team. Being a great team player, you are excited to contribute to the endeavor, so you promptly answer “Yes, of course!” and, before you have time to finish your latest pull request, Outlook is ringing with a new invite to interview the next candidate.
While your involvement in the interview process shows how efficient your company is and how much you are trusted, that’s when you realize that you've never led a technical interview before. The doubts start to settle in:
- “Will I know what to ask?"
- “What if I can’t tell if the candidate is doing well?”
- and worst of all, “What if the candidate is actually way smarter than me and I look like a fool and I make them think badly of my team?”
Cue the panic attacks.
Embed provider giphy not yet supported.I had these same doubts two years ago while beginning to help with recruitment at Dashlane. Since then, I've become a strong technical interviewer thanks to the coaching of my teammates. To help others through similar journeys, in this article I'll share my tips for conducting a great first technical interview and to help you shut down this impostor syndrome.
First, we will learn what to look for during a problem-solving interview and how to create an exercise which will maximize the chances that you get all the information you need about your candidates. In the second article of this series (coming soon!) we will then focus on how to approach your first interview in a stress-free way and gain confidence while doing so.
A note about the hiring process at Dashlane
At Dashlane our interview process consists of several remote and on-site interviews tailored for different positions. For more on that, you can read How We Interview and Hire at Dashlane, which was written by our wonderful People Team.
For engineers specifically, the process includes several technical interviews, which are usually led by one or two team members. It is customary for one of these sessions to be the "problem-solving interview.” I believe the problem-solving interview is the best setting in which to start being an interviewer because it can be led by any engineer, regardless of their seniority in the company.
The problem-solving interview at Dashlane is a classical whiteboard interview. It lasts one hour, during which time we expose a short problem to the candidate, who will have to find an algorithmic solution to this problem. It is used to interview engineering candidates of all levels of seniority and attempts to assess different technical and behavioral qualities of the candidate.
What to look for in a problem-solving interview
The first key to leading a problem solving interview is to know what you are trying to assess. This is something which should be defined by your team, your hiring manager and your talent team when your company tailors its recruitment process. Make sure you have a good understanding of your team’s expectations by sitting down with the people who regularly conduct candidate interviews. Ask them how candidates are evaluated, making sure to get clarity about what criteria to use.
Even though it varies slightly from team to team, these interviews tend to incorporate a consistent set of criteria. Unlike what you might think, we care about more than simply making sure the candidate knows how to code.
- Methodology:
The first thing to look for in a problem-solving interview is the engineer’s methodology. A big part of our day-to-day job is about finding solutions to the problems which arise, be it determining the cause of a bug, finding the source of an alert in production, detailing the best way to implement a feature or structuring a solution to tackle a problem. We are constantly trying to understand a situation, determine which parameters to take into consideration, and which actions to take to reach our goal.
We design problem-solving interviews to first evaluate whether or not a candidate can reformulate the problem. This will tell us if they understand the issue being presented to them. Next, we want to know the candidate can come up with a high-level plan to solve the issue. Accomplishing these two things means the candidate has gotten half way through the challenge without even having to write a single line of code.
On the contrary, if the candidate doesn’t take time to think about the issue and to communicate with me how they understand the problem and what they think they will do, I get more concerned because it shows one of two things:
- Either they have trouble organizing their thoughts and it might be a real issue in their day-to-day job (even though the team might help them to improve on that)
- or—and that’s the worst-case scenario—they are very smart and will have no trouble solving the problem but they can’t communicate that with other people. In a team poor communication skills is a big problem because as soon as you start working with another person, your job is not to only code but to also share ideas and thoughts with your coworkers.
This leads us to the second key thing to evaluate: Communication.
- Communication
During the interview I really want to see what is going on in the candidate’s head: Are they struggling on a particular issue without finding an idea? Are they trying to choose between two options and are not sure which one is the best? Did they realize that they didn’t understand the problem but don’t want to admit it?
If the candidate is able to communicate their thought process during an interview, chances are that they will be able to work well with the team, sharing their ideas and making proposals to improve our projects. It’s also important that candidates can admit when they don’t understand something or when they are wrong, because it happens to all of us in our job. Being unable to share your struggles with your coworkers is usually a recipe for disaster.
Some people get shy during the interviews and avoid asking questions they would normally ask, so it’s important to encourage them to do so and to help them understand they're in a safe space where they will not be penalized or criticized for not knowing something.
- Programming skills
The last important point we want to validate during a problem-solving interview is the actual technical skills of the candidate. We want to validate that they have the algorithmic skills necessary to work on our projects, that they understand the programming topics we deem important to perform well in the team and that they understand the underlying concepts of a programming language. This covers a range of scenarios, like:
- Which kind of data structure is the most adapted to search in, and why?
- What is the concept of scope?
- What does it mean to return a value from a function?
How to choose the exercise
Now that you have a clear idea of what you want to evaluate in your candidates, it is time to think about how you will conduct this assessment. If this is the first time your team is doing a technical interview, you will need to come up with the right exercise to give your candidates. Sure, it sounds easy: all you need to do is to browse Leetcode’s curated list of algorithmic questions, Codingame’s puzzles or even maybe the Project Euler problems and just pick one more or less randomly. I personally believe there is a little more to it than that. Here are some tips for selecting the perfect exercise.
- Don’t aim for complexity
You might think “I’m looking for top quality engineers, so I want a very hard exercise to make sure that only the top engineers succeed.” You might also think that you want to show off to the candidates how smart your team is, and furthermore boast that you have people who can solve very complex algorithmic challenges, which will attract the more technically versed engineers. While not entirely false, this reasoning has the flaw of dramatically narrowing the pool of candidates who will be considered for several reasons:
- First, not all great engineers perform well under the pressure of an interview. I’ve seen candidates of all levels of seniority getting blocked by the stress of being scrutinized. This is completely normal. It's a situation where our "fight or flight" reflex kicks in! Tailoring an exercise which is not too complex but allows for deeper questions, sets up the candidate for success. In the end, you want a candidate to be able to succeed in the interview so you can evaluate their thought process, problem solving skills, and their ability to code, not their ability to handle stress.
- Even if the candidate is not particularly stressed out, if your exercise is too complex and focused on a particular concept that they never used, you don’t allow them to show their skills. Just because someone cannot come up with a Monte Carlo algorithm implementation in under one hour doesn’t necessarily mean they don’t have great algorithmic skill; it could just mean they are not familiar with this particular problem but would still be able to learn quickly when given the opportunity.
- Ideally you want to review several things during the interview. Can the candidate come up with an algorithm? If so, great! Next, can they think of edge cases? Finally, are they able to identify potential bottlenecks and suggest optimizations? If your problem is too complex, chances are that you will only focus on one aspect of the solution and finish the interview without evaluating the full range of your candidate skills.
A good rule of thumb for a one-hour interview is to find a problem which can be solved in a calm setting in about 15 minutes. This should give the candidate enough time to find a solution and iterate on it a few times, while also thinking about some follow up questions.
- Interview your coworkers
It can't be said enough: Two (or more) heads are better than one. Some of your teammates have already faced problem-solving interviews, some might even do coding competitions for fun, so they have experience with algorithmic challenges and, in any case, they have firsthand knowledge of the skills required to perform well in your team. That makes your teammates the best people to consult with when proposing problem ideas or to when reviewing your own ideas.
As in all aspects of our job, collaboration will allow you to detect flaws in your current solution, to get ideas of improvement and to get validation.
- Standardize your interview
The last thing you want is to spend ten hours leading ten interviews before you realize that all your candidates did great on different aspects of the interview which makes it impossible for you to identify which one is the best fit. “So this first one came up with an algorithm in 20 minutes but was completely unable to optimize it afterward. The next one found all the edge cases but they never finished their algorithm, and the following one used extremely complex concepts but couldn’t find a simple solution... Where does that leave us?”
Once again, let’s borrow from our field of expertise to come up with a better interview process. When you evaluate a new tool to be added to your stack, you usually make sure that first and foremost, it solves your problem. Then, you use different criteria to make an informed choice while evaluating several different options at one time.
A standard interview exercise is as follows: Ask the candidate to create an algorithm to perform a set of tasks, such as sorting, searching, merging, or other permutation. Then, ask follow up questions to learn more about their thought process.
When reviewing the algorithm itself, you might consider:
- Is the candidate able to create a working algorithm during the interview? This is table stakes.
- There are probably several approaches which would give the same output: Is the candidate able to justify why, for example, they chose a recursive approach rather than a loop?
- Does the candidate think of the off-by-one errors?
- Do they see the two or three main pitfalls which might arise? Did they take time to mention these edge cases?
This sort of discussion is a satisfactory litmus test for whether the candidate passed the interview or not. If they can’t find a solution, they probably are not a good fit for the position. (However, if only very few candidates find a solution it could be a sign that you need to lower the complexity of your exercise). Once you identify the candidates who did well, you can use your follow up questions to make a choice between them. You should try to have a list of three to five such questions. Here are some examples:
- Make the candidate think about the edge cases of their algorithm: “Now instead of ten lists to merge you have one million lists. Does your solution still work? If not, what could you change to make it work?” “Is your solution resilient to bad or unexpected inputs?”
- Ask them about the asymptotic complexity of what they just wrote. Depending on their seniority and the role, it could be fine if they are not familiar with the Big O notation but they should still have an idea of the time or the resources the code will take to execute.
- Add a twist: “Now your sorting algorithm must work in real time instead of just using already existing inputs. What should you change to make this possible?”
Asking these questions offers the candidates an opportunity to stop writing code and instead discuss their ideas and give insight into to how they think.
I like having these follow up questions ready because it allows me to have an idea of how much a candidate could do during the interview: If I had time to ask one or two questions to a junior candidate it usually means that they did well at the purely algorithmic part; on the other hand, if I interview a senior candidate and I don’t even have time to ask these questions it usually shows that they didn’t perform very well. This shouldn’t be your only criterion as there are other things to take into consideration but it’s still a good rule of thumb to compare two candidates for a same position and level of seniority. As the interviewer, you're responsible for keeping a healthy pace to the conversation so the candidate has enough time to both work through the algorithm and give their thoughts on process in the same period of time.
- A note about the importance of doing a whiteboard exercise
One of the long-standing debates in the world of technical interviewing is the question of whether to conduct algorithmic interviews on a computer with a real programming language, or on a white board with pseudo code.
This is a decision you should make collectively with your team. My experience with Dashlane's hiring process is that it has allowed us to recruit a talented and capable engineering team. Given that fact, here are my thoughts on the format of algorithmic interviews:
- Firstly, I believe the question is not computer vs. whiteboard but rather IDE vs. 'another way to write code.' At Dashlane when we conduct problem-solving interviews. Assessing a candidate's knowledge of a programming language's syntax is not our main goal. Using an IDE with autocompletion, syntax highlighting, linting and these kinds of features is a no-brainer in our day-to-day work. By contrast, doing an interview using an IDE forces the candidate to focus on things we are not interested in. A candidate who passes all of our interviews already knows how to use their favorite coding tools or will learn to use better ones. It’s not important if the candidate makes a syntax error or misspells a variable name as long as they are able to create clear and concise code.
- At Dashlane we sometimes interview candidates who are not familiar with the languages we know well within the team. So, pseudo code is a great way to avoid distracting the interviewer or the candidate with these kinds of irrelevant details.
- Finally, letting the candidate run their code could actually make it harder to evaluate them. Some candidates will try to execute their code really often to validate their assumptions. While this can be a valid approach in a day-to-day job, I think it prevents the candidate from thinking ahead of their algorithm. That makes it harder to understand their thought process, which is something we really want to see in the moment.
Hence, what we do at Dashlane is that we mostly use pseudo code on a white board when we do on-site interviews. This past year most interviews were online so we use coderpad.io with code execution disabled, so the candidate can use an actual programming language or a pseudo code.
What Have We Learned? Recap time!
- What are we evaluating during a problem-solving interview? We want to see that candidates can understand and rephrase problems presented to them. Furthermore, they need to be able to discuss their thought process to ensure good collaboration and not just good coding skills, which are necessary but not sufficient to succeed in an engineering role at Dashlane.
- How do we choose exercises and structure the interview? We believe that the problem presented to a candidate should be technical enough to demonstrate their ability to produce an algorithm of some variety, but not so difficult that it leaves no room for discussion of important topics like optimizations and edge cases. When conducting the interviews, we prefer to do so without the use of an IDE so that we can again focus on the underlying logic and thought process of the candidate, and avoid distractions caused by syntax and the temptation to constantly run the code.
With these tips you should now have all the tools necessary to create a relevant problem-solving exercise. In part two of this series (link coming soon!) we will focus on how to employ this exercise properly during your interviews.
Sign up to receive news and updates about Dashlane
