Auditions – Testing for Technical Competency

In our last article, we covered the three goals of an audition and the basic guidelines for using one to hire technical talent. If you happened to miss it, do go back and read it before you dig in here.

In this article, we’re going to cover how you, as a non-technical founder or product owner, test for technical competency as a part of an audition. Over the next two articles, we’ll cover assessing team fit and growth potential.

To properly assess each candidate you’ll need to do have some help lined up, do some initial homework, and have a solid plan for the audition projects.

There’s a lot to cover, so hang in there.

You’re Going to Need Help

Let’s be honest. If you’re not an experienced developer yourself, you won’t be able to assess someone else’s capability to write quality code or design great solutions. There’s too much knowledge that you simply don’t have yet.

What you need is a trusted advisor who can help you through this part of the audition. I recommend that, before you start the hiring process, you identify someone in your network who would be your ideal CTO if you could hire them, and ask them to help you evaluate the candidates. You’re looking for someone who’s willing to spend an hour or so with you on each candidate both assessing the audition work and asking more detailed questions that you might not know to ask.

Do Your Homework

Though you might not be technical, you can still learn a lot simply by poking around in the candidate’s publicly available work. Here’s what I check out first:

  1. Ask the candidate what they’ve published or built that is publicly available. This would include things like Ruby gems, Node NPM modules or any other open source project they’ve started or contributed heavily to. Once they’ve sent you the links to those projects, find out how widely it’s been used. In the case of a gem or npm, you’ll see the download count right there on the page. In the case of an open source project on Github, check out the level of documentation and issue list for the project. What’s the level of activity? What are other people saying about the code? If the developer you’re interviewing doesn’t have anything like this, it’s an immediate red flag. It’s very rare these days that really good developers aren’t working on multiple side projects, even if just for fun, and sharing what they’re up to.
  2. Find out their Stack Overflow username and browse their activity. Stack Overflow is basically Google for developers. It’s where we all hang out and answer each other’s questions. You’re looking for where your candidate has answered questions, and how the community has voted on those answers. Lots of answers, and lots of up votes are a very strong signal.
  3. Browse their connections on LinkedIn and look for recommendations. What are people saying about them, and what roles did the people have who recommended them? Reach out to those that are most interesting directly and ask if you can chat about the candidate.

Remember to ask your trusted advisor for help along the way. If there’s something you don’t understand, ask.

Once you’ve done your homework, you’re ready to create the test project.

Create the Test Project

The best way to assess technical competency and fit for your company is to work on a real problem together. From our last article, we know that a good test project has four characteristics:

  • It’s relevant to your business
  • It’s about 4-8 hours worth of work
  • It involves both design and implementation
  • It tests working like your company works

Keep in mind that you might need your trusted advisor to help you design the project, and also that you might want to tweak the parameters of the project based on the candidate. Sounds complicated, right? Let me show you a couple examples.

The “Tell Me What’s Wrong” Project

In this project, we’re giving access to our codebase to the prospective developer and asking them to tell us what they would change if they were in charge. A few words of warning here. You need to have the developer sign an NDA prior to granting them access, and you should only grant them read access to your project so that they can’t mistakenly check some new code in. I’ll ask the developer to do the following:

  • Get the project running on their machine using only the documentation provided. This is a great test because, often times, documentation is incomplete, so the developer has to dig in and figure some things out on their own.
  • Once they’ve project running, do a complete code review and tell me what’s wrong with the code / what they would do differently. I want to see a long list for two reasons: (1) I want a review of the entire project and (2) I want them to care passionately about the code. Nearly every developer will have a long list of things they don’t like when looking at anyone else’s work because, well, that’s just how we are.
  • After the review, I’ll ask them do design a solution to one of the bigger issues they found. How would they go about implementing the change without breaking existing functionality, etc.? Can they work inside an existing project, or is the immediate response to “throw the whole thing out and start over”?

If the developer has a lot of questions / needs a lot of help getting things running, or comes back with a short list of things that they would do differently, it’s almost always an immediate no.

You want to hear things like:

  • “I got the project running and am curious why your previous developers decided to use X instead of Y.”
  • “I see you’re using <some methodology / package>. Have you thought about <this other package>? I’ve used it on similar projects and I’ve found it’s better for the following reasons.”
  • “Did you know you’ve got a huge security hole because you’re not using one-way SALTs for your passwords?”

You don’t have to understand everything you get back. Your advisor can help you sift through the list and evaluate it for substance. This is a great project for any level of hire.

The Design a New Feature Project

The good thing about this project is that it doesn’t involve granting access to your current codebase / project if that makes you uncomfortable. Here’s how it works:

  • Pick a new feature off of your list that hasn’t been designed or implemented. It should be a small enough feature that you can design and implement a rough first version in the time allotted.
  • Bring the team together that typically works on new features. UI / UX designer, other developers, etc.
  • Using a whiteboard, sticky notes, etc., work through designing the new feature with the candidate. Cap the design phase at 2 hours tops
  • Once the design is done, the candidate goes off and builds the initial version, collaborating with the team as necessary during the process

What you’re looking for:

  • The ability to assimilate and understand your business domain quickly
  • The ability to understand the new feature and how it will benefit your customer, and contribute meaningfully to the design
  • The ability to take the agreed-upon design and implement, in simple form, a working solution that meets the requirements for the feature
  • The ability to think independently and adapt things that were missed in the design
  • Hit the agreed-upon deadline!

I like this kind of project because it lets you see how the candidate will function in a fast-paced, real world situation. Can they think and operate under pressure, and can they produce well-thought out solutions in short time windows. After all, this is what you’ll be asking them to do every single day.

Just Remember

We’ve covered a lot of ground in this post, and we’re only on assessing technical competency. What I want you to remember as you prepare for the next article on assessing team fit is:

  • You can’t do this alone. You’re going to have to have a trusted advisor to help you assess your first technical hires
  • You can do a lot of research on your own, and by doing so you’ll start to understand this world a bit more. Get to know your way around Github, Stack Overflow and other sites where developers hang out. It’s another way you can “speak the language.”
  • We covered two kinds of test projects in this article: The “Tell Me What’s Wrong” Project and the Design a New Feature Project. We’ll cover more in the next couple of articles, and talk about how you adapt projects for different kinds of candidates.

In our next article, we’ll cover how you assess team fit as a part of the audition.