Finding awesome developers in programming interviews
In a job interview, I once asked a very experienced embedded software developer to write a program that reverses a string and prints it on the screen. He struggled with this basic task. This man was awesome. Give him a bucket of spare parts, and he could build a robot and program it to navigate around the room. He had worked on satellites that are now in actual orbit. He could have coded circles around me. But the one thing that he had never, ever needed to do was: display something on the screen.
Some people have a knack for asking the right questions to spot awesome developers in a job interview. Other interviewers dread it, come in with their tail between their legs, ask a few questions from the Internet and just go along with the group decision. But interviewing is an essential skill for most developers. A bad hire has terrible long term consequences, because eventually a sub-par employee may bring others into the organization. On the other hand, unfairly excluding an awesome candidate also hurts.
A programming interview includes at least three parts. In part I, we prove any assumptions we have after reading the resume. In part II, we get a sense for how much true experience the candidate has. Finally, we test this experience using a few spot checks and a coding question.
Part I: Testing assumptions from the resume
Once I was intervewing a candidate along with a fellow co-worker. When it was done, I thought the candidate had done okay, but not brilliantly. My co-worker, on the other hand, seemed angry. "He lied about technology X. He obviously has not worked with it. Definately a no-hire." Technology X was not even important to us. "If he lied about that," my co-worker went on, "I don't trust anything else on the resume."Candidates should use the resume to portray themselves in a positive light. (See The Completely Honest Resume). However, there is a line where this positive portrayal becomes misrepresentation. In the example above, I wasn't as concerned as my colleague, because I already assume that everything on the resume is false until proven otherwise. If the resume says, "expert in technology X", then I will assume that the candidate merely knows the name of technology X. If the resume says, "Worked in a group that created a multi-threaded stock trading platform," then I will assume that the candidate merely chose the colours for the background. I used to be less strict until I met the guy with 10 years of experience who couldn't write code. If someone says that they wrote the text formatter in OpenOffice, or has a Ph.D, it is easy to make assumptions about their skills. Assume nothing. All must be tested.
For each relevant item on the resume, I first try to get a sense of what the candidate actually did. Then, I get him or her to prove it by talking about it.
- Created a real-time operating system as a course project.
How large a group did you work in? A group of 15? Oh, okay then, what specific part did you work on? The message queue? Great! Can you describe what happens when a high priority task sends a message to a low priority task? - Developed from scratch an audio transfer protocol for wireless security systems.
How large was your team? Just you? Wow, how did you test it? Why didn’t you use RTP? - Fixed bugs in the XYZEngine.
Can you describe a bug that you found particularly challenging, and how you fixed it?
Part II: Finding true experience
Having more experience is a good indication of awesomeness. Experienced developers have made mistakes. They know when, and when not to apply design patterns. They have a sixth sense about what part of requirements will probably change, and what part will probably stay the same. They know when to be lazy and when to be pedantic. It is true experience which makes the gap between awesome and mediocre programmers so wide.
But not all experience is the same. It is certainly possible for someone to gain solid skills in a couple of years, simply by working on lots of different things, writing and rewriting countless lines of code, and making many mistakes. On the other hand, it is also possible for someone to spend a decade writing one-line changes to a single project, without learning anything new.
Finding hidden time
There are lots of great developers who started coding when they were in their second year of university. By the time they get out of school, they will have had a few years of experience. On the other hand, some awesome developers started learning their art at an early age. I know several people who wrote some non-trivial programs in their teens or earlier. This information is nowhere to be found on their resume, and must be coaxed out during an interview.- Why did you get into the software development field?
- What's the first programming language that you ever learned?
Density of Experience
Many awesome programmers do all of their coding at work. These are great, well, rounded individuals that you should definitely hire. However, doing personal programming projects outside of work or class is a pretty good indicator of awesomeness. A candidate with personal programming time simply has more flight time under his or her belt, and will be better for it. No personal projects? These other indicators will also count for some points:
- Working on smaller teams or groups.
- Working on a wide variety of projects
- Detailed knowledge of several layers of abstraction on a large project
- Being the main contributor in a group project
Part III: Verifying experience
After gaining a sense of the candidate's true level of experience, it is important to verify that experience testing their programming abilities. A few minutes of time is completely inadequate for a true test, but that's all that's available. We can get an idea of the breadth and depth of knowledge of the candidate by asking questions about different areas of software development. Of course your perception of the candidate's skills will be biased by your own experiences. You cannot judge the correctness of answers in topics that are unfamiliar to you. That's why there are several interviewers.
The specific topics depend on the job requirements. Nevertheless, some example areas are:
- data structures and algorithms
- multithreading
- bit manipulation
- memory allocation
- objects and inheritance, design patterns
- recursion
- compilation and how computers run programs
Each area that I choose has a selection of basic questions (“What’s a semaphore?â€). These are so basic that if the candidate has done any work at all in the area he or she would be able to answer. Each area also has some more detailed follow-up questions. The way in which a candidate answers can prove or disprove awesomeness. For example something is amiss if you ask a seasoned embedded programmer to convert 0x4c to binary, and they start by writing down 4 x 16 + 12.
The Coding Question
Usually, after all of the above, I have a very good idea whether the candidate will pass or fail, but the coding question removes all doubt. It is so important, that even phone interviews are not exempt. To be useful, a coding question requires careful thought and planning before the interview. Asked the wrong way, the response will be useless.
First, one must choose a question based on what the candidate has had experience with. You may have a clever problem that becomes easy if you think of converting everything to intersecting 3D planes. Save it for the lunch hour with your colleagues. If the job does not involved 3D graphics, candidates would be unfairly excluded.
The question must be precisely worded. "Write a function to shuffle a deck of cards" is woefully ambiguous. Provide the function header and avoid misunderstandings, which are all too common. If you are not careful, the candidate will answer a harder or easier problem than the one you asked. The harder one is nice, unless it causes him or her to freeze up. The easier one provides no information. To prevent a huge waste of time, ask for a verbal outline of the solution after a few minutes, to check if the candidate is on the right track.
The influence of the order of questions
The order in which you ask questions can profoundly influence the thought processes of the candidate. For example, I used to ask question about hash tables when I thought the candidate knew about them. Later on in the interview, I would ask a coding question that had nothing to do with hash tables. Candidates would invariably decide to use a hash table in their implementation, with the keys being unique, consecutive integers starting at 0. If I avoided talking about hash tables, the candidates would instead choose to use an array.
Candidates are also strongly influenced by you in their choice of language. For example, if you say the job primarily involves Java, every candidate will swear that, by golly, Java is his best and favourite language to work in. He will choose to use it for all coding questions, realizing too late that he can't remember how to declare variables in the language he is "best" at.
Avoid language bias
It's terribly easy to be biased toward a specific programming language that you use at your company. By fixating on a particular tool, you throw away a lot of awesome developers. Do not try to determine if the candidate awesome at programming in C or Java or whatever. Instead, you should be trying to find out if the candidate awesome at programming in the language that he or she knows best.
Going further
The guidelines above do not address everything. They focus on experience, and they might miss awesome developers that have little experience, but a lot of innate ability. In particular, interviewers may want to test problem solving ability using puzzles that don't require any coding.
The interviewing technique that I have described here is based on proving a hypothesis, probability, and gut instinct. The hypothesis is that the candidate is an awesome developer. What traits does an awesome developer have? You cannot directly measure those traits, so you have to instead ask: What is the probability that the candidate has those traits given that he or she can answer a particular question quickly? It is not possible to assess a candidate within an interview with 100% success, but by asking thoughtful questions, you can come close.
Further Reading
Semaphores are a specific type of synchronization primitive and are common in the operating systems I have used.
Linux, FreeBSD: sem_init
Windows: CreateSemaphore
OS X: semaphore_create
Possibly related to the take-home problem, also ask for a one page sample of technical English. You're interested in grammar, spelling and logic. Just my opinion, but if a programmer can't clearly explain something technical, they can't design or code either.
I'd be careful with the specific questions (your Part III lead-in). Just because buddy says he knows concurrent programming, that doesn't mean he can (or ought to be able to) define a semaphore. And don't ask a candidate about a specific data structure if you can truthfully admit that you yourself had to look it up in the last couple of years. And given that a multitude of programmers use design patterns without ever having read all that much about them, don't assume that not knowing what a Builder or a Flyweight or a Strategy pattern is means that the guy doesn't know how to employ these patterns to solve problems.
I don't think your guidelines are that far off the mark, mind you.
> construct and not a part of any language, then I
> don't want them wasting my time on "probation".
> So I'm going to ask a LOT of "stupid questions".
> And I do expect them to do the same. Next.
Ummm, which OS uses the term semaphore? Do you think everyone you interview will have used that OS? I've written C for more than one OS that didn't call them semaphores.
If someone doesn't know that a semaphore is an OS construct and not a part of any language, then I don't want them wasting my time on "probation". So I'm going to ask a LOT of "stupid questions". And I do expect them to do the same. Next.
If someone asked me to convert hex to decimal or binary manually, I'd ask why? In 15+ years of development work, I've never had to do it. And yes, I know the difference between hex and decimal.
I've also been on the other side of the table, and it's tough sometimes to find what the interviewee is all about. I have used the white board approach with some good success. For example, if I'm looking for a .Net developer, I might ask them to write a few lines of code to perform a certain function which pertains to the job requirements. One young interviewee who claimed to have good experience in ASP was asked to create a recordset from a command object, and couldn't do it. Really? No job for you.
Either way, on both side of the table, it's part skill and part art. I have found that team players with a positive attitude and a willingness -- even a desire -- to learn make the best hires, regardless of whether they can convert hex to decimal.
Languages are tools; they come and go. (Ghu, I wish I had the time back I spent teaching myself Ada...for nothing.) The real gold is knowledge of algorithms, data structures, models--the things that transcend any language. Problem solving, not codeslinging _per se_. Codeslinging comes fast for any experienced developer--I can ramp up in any language I've used before but not used in 5 years in a day or two; a new one may take a few more days. But I probably couldn't write compilable code on that first interview. I probably don't remember all the library calls--I'll remember they *exist*, but not the syntax.
On that basis, you wouldn't get me in the door. And it'd be your loss.
Don't hit them for a particular language and a compilable example; demand algorithmic pseudo-code and give them something that requires understanding the issues at hand (e.g., interprocess communication, etc.)
The interview process works both ways. It is vital to determine if you can work for these manglers. My favorite memories are of hacks who lied constantly, backed out of commitments, agreed to project schedules & then ignored the schedule and demanded the finished products immediately, as if the plan is all that it required, changed specs almost daily and best of all, refused to honor compensation agreements as soon as they got the finished product.
Get references about the people you'll be working for; if you are smart enough to do this you can hack it. Otherwise you might be more suited as a mangler trying to guess if a programmer is any good or not. Unless you are thick skinned and cognisant of the blame games C students play before you go in, a bad management experience can damage your love for your craft and your confidence.
Save your best ideas until you have successfully gone around the SDLC without being stabbed in the back. The last few years things have gotten out of hand, with alleged outsourcing costs dropping and MS releasing a new crop of doodads which seldom live up to possibly inaccurate marketing claims... made by other C students who didn't make it through any symbolic math classes due to the rigorous integrity this requires. Stupid is forever & idiots you can't delegate away from will always wear you down.
The less you say the better. Say enough to get the job, don't impress them too much. They'll be skinning you and hanging your head on their wall before you get out of the room. If they start drooling show them something shiny, refer to magic & run for your life.
If you get a real crooked bunch they may try to encumber your IP for the rest of your life. You should be able to screw them up enough to leave you alone but they sometimes jump out of the bushes years later, claiming what you are doing is based on what you did for them. Better to never work professionally than write one line of code for these pimps.
I am rather fond of 1 patent lawyer. He never wrote any code, he's 'awesome' with this body of law. Coders who became lawyers make me nervous, like Lestat, and for the same reasons. Keep your personal IP lawyer on retainer so your employer can't hire him away from & undermine your support team.
If you want to find out if a programmer is any good, once you've vetted them let them spend between 20 minutes to 1 hour with the developers they'll be working with. Let them sample the kinds of projects they'll be working on, if not the actual project. Stick your head in toward the 2nd half & observe the interaction, then after they leave ask your people what they think. Unless you are Catbert this is much easier & accurate than devising labyrinthine one time pad interview trciks, or catchall methods that sort of work some of the time.
Poaching is for C students. If your company is any good the A+ students will find you. Try to be smart enough not to throw them out, or give them a broom. Believe me, it comes back.
An alternative version of this: "Ten years of experience or one year repeated ten times?"
Jaco> If you ask me a bunch of stupid questions I might lose interest in your company and go somewhere else.
I used to think like that. Then I sat on the other side of the table too many times. Now I know that one bad hire costs a hell of a lot more than missing several good ones.
At this point, I intend to start with one question: "Do you have any publicly readable code we could look at?" If the answer is yes, I'm biased towards hiring; if the answer is no, I'm biased against hiring.
I have come to the conclusion that there is no such thing as innate programming ability (or innate ability in general). Read the book "Talent is overrated" as well as "Outliers" by Malcom Gladwell. Persistent practice trumps what appears to be innate talent. And persistence is generally driven by interest. That's why I like to look at personal side projects - stuff the interviewee is working on in their free time without being paid for it.
Very true.
@Zenja:
haha! Have seen poaching work wonderfully when done at the right time && with the right carrots on sticks.
@Steve:
>>test problem solving ability using puzzles that don\'t require any coding
A few firms that I know use this method to recruit people by truckloads, off colleges. But only a few of those turn out to be good at programming as I have seen. IMHO, the capability to solve puzzles would be a necessary but not a sufficient condition.
Again, this is just my opinion, but I don't think you can condense 20 years of experience into a 60 minute interview. That's why we have things like probation periods.
qb.js: An implementation of QBASIC in Javascript
Play NIBBLES.BAS in your browser. I re-implemented a small part of QBASIC as a compiler in Javascript, so it runs in a webpage.Compressing dictionaries with a DAWG
A practical, memory efficient way to store and search large sets of words.Coding tips they don't teach you in school
Some time-saving shortcuts for C code that will make your coworkers scream. In Awe.Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US
After many months, your software sale is complete! You've got a purchase order, sent the invoice, delivered the software. You're already handling some support issues from users at BigCorp. Then BANG! Martha from Procurement emails back, as a favour, just to let you know that BigCorp has not received your W8 form with a valid tax id, and therefore will be withholding 30% of the purchase price of your multi-thousand dollar product for taxes.You can cheat so your web site seems faster than it is
You can make your web site seem faster without actually being faster.Why Perforce is more scalable than Git
Branching on Perforce is kind of like performing open heart surgery. But here's why git can't hope to compete with it.Zwibbler: A simple drawing program using Javascript and Canvas
Now it's a commercial product, but Zwibbler was once a fun side-project, and here's some details on its implementation.Stock Picking using Python
Python can tell you which stocks to buy. It's a sure thing!VP trees: A data structure for finding stuff fast
Let's say you have millions of pictures of faces tagged with names. Given a new photo, how do you find the name of person that the photo most resembles?In the cases I mentioned, each record has hundreds or thousands of elements: the pixels in a photo, or patterns in a sound snippet, or web usage data. These records can be regarded as points in high dimensional space. When you look at a points in space, they tend to form clusters, and you can infer a lot by looking at ones nearby.