How To Build a Best-In-Class Technical Hiring Process

Comparing Your Technical Hiring Process To Industry Leaders

‍

Hiring software developers is really, really hard. That’s why we sit down with hiring managers every day to understand what’s working, and not working, when screening for technical talent.

‍

Our surprising finding? There is one thing that the best hiring managers do differently that causes them to fill engineering roles 30% faster than their counterparts.

‍

Before we talk about the tactics… How do you think you stack up? How long do you think it would take for you to hire a great-fit engineer for your next role?

‍

After talking to hiring managers across North America, we developed a hiring simulator that runs 1,000 simulations based on your hiring process to determine how long it should take you, on average, to find a great-fit candidate based on the most important factors to time-to-hire.

‍

You can access our hiring simulator here. Give it a try, then come back to this blog to learn what we’ve seen from hiring managers across the United States and Canada who are able to beat the market by up to 50% on time-to-hire.

‍

What World-Class Technical Screens Do Differently

So, what sets the best-in-class technical screens apart from the rest? After talking to engineering leaders and hiring professionals across North America, we believe it comes down to enabling evidence-backed decisions and moving the process along quickly.

‍

Unfortunately, it’s really hard to balance those two competing interests. It takes time to gather evidence that someone would be a good hire, but if you take too long to gather the evidence, you run the risk that your most talented applicants will get snatched up.

‍

According to our research, taking the simple step of reviewing every resume the day it is submitted reduces your time-to-hire an average of 10%. What’s more, if you are able to review resumes and get technical screens out to all qualified applicants within 24 hours of application, your time to hire is cut by nearly 30%. This ends up having a much larger impact on your overall time-to-hire than overall applicant volume. In fact, for the average software engineering hiring cycle, you would need to triple your candidate volume to have the same impact on time-to-hire as this one simple process change.

‍

If you are able to review resumes and get technical screens out to all qualified applicants within 24 hours of application, your time to hire is cut by nearly 30%

‍

In an episode of our podcast, Scaling Software Teams, we sat down with engineering leaders across the country to learn how they build a technical screen that gets the results they need fast, so they can start interviewing and start hiring.

‍

‍

Listen to the Full Episode Here:

Featured in This Episode:

• Jon Lavender, CTO at Dragos
• Adam Perelman, CTO at Affinity
• Ryan Carroll, VP of Engineering at Zylo
• Ryan Rusnak, CTO at Airspace Technologies
• Jeff Stickel, CTO at Kerauno

‍

‍

Here’s what we learned:

‍

1. Programming exercises are essential to hiring success.

Every single engineering leader we talked to listed programming exercises as an important step. Several leaders use their exercise as a way to gauge interest in the role – not just technical ability.

 

Jon Lavender, the CTO of Dragos, walked us through his company’s technical screening process.

‍

“Our technical screening process consists of a variety of hands on and verbal exercises that challenge candidates to think critically, engage our team, and demonstrate their skills and experience. We typically have a short screening session out front with various members of the engineering team, typically in the same discipline as the candidate, meaning someone from systems engineering will usually interview someone else who’s interviewing for a systems engineering position.

‍

After that, depending on [the candidate’s] discipline, we typically ask candidates to go through a hands on exercise. As an example, for software developers we ask that folks do a programming exercise that’s either tailored towards backend server application development or frontend UI development. This has proven to be a great tool to not only get a feel for the candidates skill level, but also to engage how interested a person is in the job. We’ve had folks drop out at just about every stage of that process for one reason or another – it’s been very helpful for us to help weed folks out.

‍

Lastly, we typically follow-up with a face to face interview; that’ll either be a video call or an in-person interview, where we typically have a couple of our subject matter experts ask additional technical questions, in the case of developers we typically follow-up on the exercise that they did. Sometimes we’ll bring in folks from other areas other than engineering to gauge the candidate’s knowledge outside the primary discipline they’re interviewing for – that’s proven to be very helpful for us, as well.”

‍

Jon’s screening process also identifies candidates who aren’t as interested in the job, but the challenge is striking the balance between screening tool and gatekeeper.

‍

If we over-rely on the heuristic of “who wants the job most”, we’ll screen out some of our most in-demand candidates. We also need a way to measure the “will” half of the skill/will bullseye. Candidates who aren’t willing to do our job make worse teammates.

‍

There is a middle-ground that works for both great candidates and hiring managers, and it brings us to our second point…

‍

2. The best technical interviews look like actual work.

The most common complaints about technical interviews fall into a couple different buckets: Sometimes, the candidate is upset because of how long the assessment takes. Other times, they’re frustrated because assessments resemble riddles, not work.

 

Does our company sell Fibonacci sequences? If not, then how do we know that our interview predicts success on the job?

‍

This point was echoed by Adam Perelman, CTO at Affinity. Here’s what he had to say about his technical screening process:

‍

“At Affinity, as we’ve been designing and refining our technical interview process, we’ve found that the best technical interview process should be something that the candidate might actually be asked to do on the job in the real world.

‍

So for example, a lot of interview questions ask about things like, reversing a link list or dynamic programming, or things like that. We’ve found that those aren’t really things that software engineers have to do day to day; most software engineers, most of the time, are dealing more with problems of de-buggability, handling edge cases, designing clean abstractions, and so on.

‍

So, we’ve tried to design an interview process that more closely reflects those skills that really matter day to day, on the job. And so that’s sort of the high level philosophy – in terms of the process, our standard interview process is pretty simple: we’ll start with a 60 minute technical coding interview over the phone, we’ll ask the candidate to build a simple application that hopefully is somewhat realistic but that still fits inside the 60 minute time constraint.

‍

And then if that goes well we’ll move to an onsite where we do a mix of technical and cultural interviews and of course, try to use the onsite as an opportunity to sell the candidate on Affinity, as well.

‍

The type of work matters and so does the environment. Whiteboard interviews are great at predicting performance writing code on a whiteboard in front of people. Is that our environment?

‍

If not, we introduce some confounding factors. Interviews are particularly stressful for certain people based on their personality. Anything resembling public speaking in front of strangers is also stressful for certain folks. Combining those is especially tough on introverts. It’s also a challenge if the candidate doesn’t come from an environment like ours that values psychological safety.

‍

Whiteboard interviews also have an opportunity cost… if we wait until the in-person interview to do a technical assessment, we may miss great candidates. After all, plenty of great engineers may not have the perfect resume. Many of us struggle with “culling the herd.” We get a ton of applicants for a given role, but just have a list of past employers to help us narrow the field. If we want to sort candidates by aptitude, we can use a technical screening tool at the beginning of this process.

 

Someone who has seen great results by leading with a technical screen is Ryan Carroll, the VP of Engineering at Zylo.

‍

“At Zylo, our technical screen is really designed to replicate the work you’re going to be doing here as an employee. Our project is a take home test where we work with the candidates’ schedule to agree on a mutual timeline. We’ve created three different projects based on the candidate’s background and the open role.

‍

The three projects are no GSAPI, a react single-page application, and a more theoretical, object-oriented project for New College, or bootcamp graduates. We have tried to speed up the set-up process where possible and create detailed guides and instructions, and even doc our environments where possible.

‍

We ask that the candidate spends around 2-4 hours on the project, and we emphasize that more isn’t always better. Our team will then use that code submission during the technical onsite interview to ask questions about how they solved the problems, and what else they would do to improve their project.

‍

Ultimately, we aren’t looking for specific answers – we really want to understand how someone tackles problems. If we could take one aspect of this and make it a little bit better, it would be to add some collaboration with the current team. Everything we’re doing on the engineering team at Zylo is around the team, and this current process really doesn’t take that into account.”

‍

Because Ryan puts the technical screen early in the process, he’s able to ask really concentrated, focused questions during the first phone screen. That helps him find the applicants who really know their stuff.

‍

Here’s Jon Lavender of Dragos again, talking about what he wished the hiring process could look like…

‍

“If I had a magic wand to improve one aspect of the process, it would be the initial selection of candidates. It would be incredibly helpful to narrow down a list of tens to hundreds of candidates, to a select few that best fit the role – there are a lot of tools and services out there to help with this.”

‍

In a perfect world, all of our candidates would come in and magically be sorted into “yes” and “no” piles for us. Is past job title or school part of your magical criteria? My guess is we’d focus more on ability to do the job and ability to fit in well with our team.

 

For some of us, doing the job requires top-notch programming skills. For others, it might be the ability to communicate well and collaborate with teammates. Ryan Rusnak, the CTO of Airspace Technologies, pointed out one huge limitation that the traditional interview process has:  identifying grit and heart.

‍

“At Airspace Technologies, we have a pretty lengthy screening process – and it’s not because we just want to haze the candidates, but I know that’s kind of what it’s turned into; like, I really got grilled in a technical interview, so I need to grill everybody else. And that’s not what it’s about.

‍

But how our process works is we start of with a phone call – make sure it’s a pretty good cultural fit, that they’re interested [in the job], and that they’ve actually programmed in these languages, and that they want to continue programming these languages. We’re just trying to align our values to make sure that what they want to do is what we need them to do.

‍

After that we send them a homework assignment, because I think grilling someone on a whiteboard is certainly not the best way to see if someone is a great programmer. So the homework assignment is a really low pressure way to see how they code – you can even see their commits, things like that.

‍

We then go over their homework, and then we’ll bring them in for an onsite.

Onsite we do pair programming, so you sit with another Airspace engineer and work on an assignment. [It’s a] great way to hear them talk, see how they pair; it’s a much more natural way [than] grilling someone on a whiteboard.

‍

After that, we do an architecture interview, so we’ll actually go to the whiteboard and chat and see if they can think through problems, and then after that they do a final interview with our Director of Engineering. If we think we want to highlight more architecture stuff, or we want to highlight more Ruby code-specific things, he kind of fills in the gaps.

‍

And the reason we do this is that some people are paralyzed when it comes to architecture, and some people really shine in pairing while some people don’t. So we really want to give people a chance to shine in one of those, because none of those are weighed too heavily. Let’s say someone really doesn’t do great in the pairing but their homework was amazing they poured a ton of effort into it, then we still might move forward.

‍

So although it’s a lengthy process, we make sure we’re covering all of our bases to ensure we’re giving the candidate the chance to shine in their best way possible.”

 

This is an incredibly common wish. Jeff Stickel, the CTO of Kerauno, told us that he would trade his magic wand in for a crystal ball so he could see the future.

‍

“If I had an interviewing superpower, it would be that I could look into a crystal ball and see how a candidate would be performing in a year if I decided to hire that person.

‍

Irrespective of how rigorous and complete and accurate we think our interviewing process is, we can never be sure that we’re making the best decision. Sometimes candidates that pass all of the steps in the screening process still turn out not to be the best fit for our team.”

‍

The third takeaway that we had from our conversations with engineering leaders was the importance of reciprocal communication… That means…

‍

3. Our interview process should give just as much information to the candidate as it does to the hiring manager.

‍

Ultimately, the hiring process is about finding the right fit for all parties involved. There are plenty of talented candidates who may not be the right fit for a specific role. That’s fine! We want them to learn that early.

‍

Here’s Adam Perelman from Affinity again…

‍

“At the end of the day, when we’re designing our interview process, what we’re really trying to test is, “Would this person be a great co-worker to have on the engineering team?” Day in, day out would they be someone who we’d be super excited to work with and who would bring a lot of value to the team. Obviously, any kind of artificial interview process that lasts at most a few hours is gonna have a really hard time telling you that.

‍

And so some companies, historically, have tried to go straight the source, right? So for instance Weebly I know used to do this I don’t know if they still do this, but they actually had engineering candidates come in and work with the engineering team for a week or something like that, as a way of getting really good information on both sides. If you actually work with somebody for a week it’s much easier to tell if this is someone who would fit in well with the team, who would add value, and who would contribute meaningfully.

‍

And obviously, for the candidate as well it’s much easier to tell, “Is this a company where I’d enjoy working, and is this is a company where I’d be excited to wake up every day and go to work?”

‍

If we could, we would do that for every single candidate, the problem, of course, is logistically it’s pretty tough to fit a week-long interview process into most people’s lives – especially if they’re interviewing with multiple companies or if they’re currently employed and sort of opportunistically looking.

‍

So if I could wave a magic wand I would compress time a little bit so that we could fit a week’s worth of working alongside somebody into an onsite that lasts an afternoon.”

‍

The ultimate goal of any hiring process is to match the right people with the right job, whether that’s at our organization or not. The best tech screening processes are built with that reality in mind.

‍

This means building a screening process that can identify who has the right technical skills for the job, but it also means finding folks who are aligned with our values and mission as a company. Here’s how Ryan Rusnak of Airspace Technologies put it:

‍

“If I had a magic wand that could make one part of this process better, I think it would be to just look at someone’s resume and know the caliber of employee they’re going to be. I don’t even necessarily need to know the caliber of programmer [they’d be], but interviews are just not a great way to assess someone’s heart or effort.

‍

I joke around that I lost faith in interviews since starting Airspace and hiring lots of engineers, because there are people that I interviewed and I thought they were good, and they were going to come in and be effective, and then they absolutely blew me away.

‍

So, I really wish that somehow in the interview process, there was some sort of exercise that would better show that someone just pours all of their heart and energy into their work – and you just can’t get that through a resume, and interview, or any sort of programming assessment.”

‍

 

August 9, 2019

 

‍