Hate UML?

Draw sequence diagrams in seconds.
http://www.websequencediagrams.com

Finding awesome developers in programming interviews
Posted on: 2010-09-13 18:00:00

In a job interview, I once asked a very experienced embedded software developer to write a program that reverses a string and prints it on the screen. He struggled with this basic task. This man was awesome. Give him a bucket of spare parts, and he could build a robot and program it to navigate around the room. He had worked on satellites that are now in actual orbit. He could have coded circles around me. But the one thing that he had never, ever needed to do was: display something on the screen.

Some people have a knack for asking the right questions to spot awesome developers in a job interview. Other interviewers dread it, come in with their tail between their legs, ask a few questions from the Internet and just go along with the group decision. But interviewing is an essential skill for most developers. A bad hire has terrible long term consequences, because eventually a sub-par employee may bring others into the organization. On the other hand, unfairly excluding an awesome candidate also hurts.

A programming interview includes at least three parts. In part I, we prove any assumptions we have after reading the resume. In part II, we get a sense for how much true experience the candidate has. Finally, we test this experience using a few spot checks and a coding question.

Part I: Testing assumptions from the resume

Once I was intervewing a candidate along with a fellow co-worker. When it was done, I thought the candidate had done okay, but not brilliantly. My co-worker, on the other hand, seemed angry. "He lied about technology X. He obviously has not worked with it. Definately a no-hire." Technology X was not even important to us. "If he lied about that," my co-worker went on, "I don't trust anything else on the resume."

Candidates should use the resume to portray themselves in a positive light. (See The Completely Honest Resume). However, there is a line where this positive portrayal becomes misrepresentation. In the example above, I wasn't as concerned as my colleague, because I already assume that everything on the resume is false until proven otherwise. If the resume says, "expert in technology X", then I will assume that the candidate merely knows the name of technology X. If the resume says, "Worked in a group that created a multi-threaded stock trading platform," then I will assume that the candidate merely chose the colours for the background. I used to be less strict until I met the guy with 10 years of experience who couldn't write code. If someone says that they wrote the text formatter in OpenOffice, or has a Ph.D, it is easy to make assumptions about their skills. Assume nothing. All must be tested.

For each relevant item on the resume, I first try to get a sense of what the candidate actually did. Then, I get him or her to prove it by talking about it.

  • Created a real-time operating system as a course project.
    How large a group did you work in? A group of 15? Oh, okay then, what specific part did you work on? The message queue? Great! Can you describe what happens when a high priority task sends a message to a low priority task?
  • Developed from scratch an audio transfer protocol for wireless security systems.
    How large was your team? Just you? Wow, how did you test it? Why didn’t you use RTP?
  • Fixed bugs in the XYZEngine.
    Can you describe a bug that you found particularly challenging, and how you fixed it?

Part II: Finding true experience

Having more experience is a good indication of awesomeness. Experienced developers have made mistakes. They know when, and when not to apply design patterns. They have a sixth sense about what part of requirements will probably change, and what part will probably stay the same. They know when to be lazy and when to be pedantic. It is true experience which makes the gap between awesome and mediocre programmers so wide.

But not all experience is the same. It is certainly possible for someone to gain solid skills in a couple of years, simply by working on lots of different things, writing and rewriting countless lines of code, and making many mistakes. On the other hand, it is also possible for someone to spend a decade writing one-line changes to a single project, without learning anything new.

Finding hidden time

There are lots of great developers who started coding when they were in their second year of university. By the time they get out of school, they will have had a few years of experience. On the other hand, some awesome developers started learning their art at an early age. I know several people who wrote some non-trivial programs in their teens or earlier. This information is nowhere to be found on their resume, and must be coaxed out during an interview.
  • Why did you get into the software development field?
  • What's the first programming language that you ever learned?

Density of Experience

Many awesome programmers do all of their coding at work. These are great, well, rounded individuals that you should definitely hire. However, doing personal programming projects outside of work or class is a pretty good indicator of awesomeness. A candidate with personal programming time simply has more flight time under his or her belt, and will be better for it. No personal projects? These other indicators will also count for some points:

  • Working on smaller teams or groups.
  • Working on a wide variety of projects
  • Detailed knowledge of several layers of abstraction on a large project
  • Being the main contributor in a group project

Part III: Verifying experience

After gaining a sense of the candidate's true level of experience, it is important to verify that experience testing their programming abilities. A few minutes of time is completely inadequate for a true test, but that's all that's available. We can get an idea of the breadth and depth of knowledge of the candidate by asking questions about different areas of software development. Of course your perception of the candidate's skills will be biased by your own experiences. You cannot judge the correctness of answers in topics that are unfamiliar to you. That's why there are several interviewers.

The specific topics depend on the job requirements. Nevertheless, some example areas are:

  • data structures and algorithms
  • multithreading
  • bit manipulation
  • memory allocation
  • objects and inheritance, design patterns
  • recursion
  • compilation and how computers run programs

Each area that I choose has a selection of basic questions (“What’s a semaphore?”). These are so basic that if the candidate has done any work at all in the area he or she would be able to answer. Each area also has some more detailed follow-up questions. The way in which a candidate answers can prove or disprove awesomeness. For example something is amiss if you ask a seasoned embedded programmer to convert 0x4c to binary, and they start by writing down 4 x 16 + 12.

The Coding Question

Usually, after all of the above, I have a very good idea whether the candidate will pass or fail, but the coding question removes all doubt. It is so important, that even phone interviews are not exempt. To be useful, a coding question requires careful thought and planning before the interview. Asked the wrong way, the response will be useless.

First, one must choose a question based on what the candidate has had experience with. You may have a clever problem that becomes easy if you think of converting everything to intersecting 3D planes. Save it for the lunch hour with your colleagues. If the job does not involved 3D graphics, candidates would be unfairly excluded.

The question must be precisely worded. "Write a function to shuffle a deck of cards" is woefully ambiguous. Provide the function header and avoid misunderstandings, which are all too common. If you are not careful, the candidate will answer a harder or easier problem than the one you asked. The harder one is nice, unless it causes him or her to freeze up. The easier one provides no information. To prevent a huge waste of time, ask for a verbal outline of the solution after a few minutes, to check if the candidate is on the right track.

The influence of the order of questions

The order in which you ask questions can profoundly influence the thought processes of the candidate. For example, I used to ask question about hash tables when I thought the candidate knew about them. Later on in the interview, I would ask a coding question that had nothing to do with hash tables. Candidates would invariably decide to use a hash table in their implementation, with the keys being unique, consecutive integers starting at 0. If I avoided talking about hash tables, the candidates would instead choose to use an array.

Candidates are also strongly influenced by you in their choice of language. For example, if you say the job primarily involves Java, every candidate will swear that, by golly, Java is his best and favourite language to work in. He will choose to use it for all coding questions, realizing too late that he can't remember how to declare variables in the language he is "best" at.

Avoid language bias

It's terribly easy to be biased toward a specific programming language that you use at your company. By fixating on a particular tool, you throw away a lot of awesome developers. Do not try to determine if the candidate awesome at programming in C or Java or whatever. Instead, you should be trying to find out if the candidate awesome at programming in the language that he or she knows best.

Going further

The guidelines above do not address everything. They focus on experience, and they might miss awesome developers that have little experience, but a lot of innate ability. In particular, interviewers may want to test problem solving ability using puzzles that don't require any coding.

The interviewing technique that I have described here is based on proving a hypothesis, probability, and gut instinct. The hypothesis is that the candidate is an awesome developer. What traits does an awesome developer have? You cannot directly measure those traits, so you have to instead ask: What is the probability that the candidate has those traits given that he or she can answer a particular question quickly? It is not possible to assess a candidate within an interview with 100% success, but by asking thoughtful questions, you can come close.

Further Reading

Want more programming tech talk?
Add to Circles on Google Plus
Subscribe to posts

Post comment

Real Name:
Your Email (Not displayed):

Text only. No HTML. If you write "http:" your message will be ignored.
Choose an edit password if you want to be able to edit or delete your comment later.
Editing Password (Optional):

Jaco Pretorius

2010-09-14 06:04:04
I think you're missing an important point here - if I'm coming in for an interview I'm interviewing you as much as you are interviewing me. If you ask me a bunch of stupid questions I might lose interest in your company and go somewhere else. Great developers can choose who they work for, not the other way round. For example, how many languages still actually use semaphores?

Again, this is just my opinion, but I don't think you can condense 20 years of experience into a 60 minute interview. That's why we have things like probation periods.

Zenja

2010-09-14 06:40:16
Good recruiters poach. They don't depend on the luck of the draw to get a decent applicant. Besides, all the excellent applicants are already working for your competitor, and the only time you'll ever see them is when they're relocating, or they've just had enough of X and now want to try Y. However, I still stand by my assertion that if you really want to recruit talented staff, you have to poach them.

raghava

2010-09-14 10:21:12
@Jaco:

Very true.

@Zenja:

haha! Have seen poaching work wonderfully when done at the right time && with the right carrots on sticks.

@Steve:

>>test problem solving ability using puzzles that don\'t require any coding

A few firms that I know use this method to recruit people by truckloads, off colleges. But only a few of those turn out to be good at programming as I have seen. IMHO, the capability to solve puzzles would be a necessary but not a sufficient condition.

Dave Vandervort

2010-09-14 13:34:11
You have some good points here but overlook the elephant in the corner. When I'm interviewing candidates, I'm not concerned with finding awesome coders. I've found too many who were completely worthless. I'm thrilled just to find someone adequate.

Tom Phillips

2010-09-14 13:41:07
"They focus on experience, and they might miss awesome developers that have little experience, but a lot of innate ability"

I have come to the conclusion that there is no such thing as innate programming ability (or innate ability in general). Read the book "Talent is overrated" as well as "Outliers" by Malcom Gladwell. Persistent practice trumps what appears to be innate talent. And persistence is generally driven by interest. That's why I like to look at personal side projects - stuff the interviewee is working on in their free time without being paid for it.

Tommy McGuire

2010-09-14 17:15:57
> But not all experience is the same.

An alternative version of this: "Ten years of experience or one year repeated ten times?"

Jaco> If you ask me a bunch of stupid questions I might lose interest in your company and go somewhere else.

I used to think like that. Then I sat on the other side of the table too many times. Now I know that one bad hire costs a hell of a lot more than missing several good ones.

At this point, I intend to start with one question: "Do you have any publicly readable code we could look at?" If the answer is yes, I'm biased towards hiring; if the answer is no, I'm biased against hiring.

Geert De Pauw

2010-09-15 04:24:32
Never forget to check his/her communication skills, you can be an excellent developer but without knowledge how to communicate with managers,customers,users. They don't understand your technical answers buy want to hear the answers in their understanding language.

Old Guys Rule

2010-09-15 04:27:29
Awesome? Just don't call me a rockstar. I'd rather smash you in the head with a rockstar than be expected to wag my tail and instantly produce release grade products on demand. If you don't understand the work involved I'm not going to allow you to put me on a pedestal. All I ask is absolute control of my projects.

The interview process works both ways. It is vital to determine if you can work for these manglers. My favorite memories are of hacks who lied constantly, backed out of commitments, agreed to project schedules & then ignored the schedule and demanded the finished products immediately, as if the plan is all that it required, changed specs almost daily and best of all, refused to honor compensation agreements as soon as they got the finished product.

Get references about the people you'll be working for; if you are smart enough to do this you can hack it. Otherwise you might be more suited as a mangler trying to guess if a programmer is any good or not. Unless you are thick skinned and cognisant of the blame games C students play before you go in, a bad management experience can damage your love for your craft and your confidence.

Save your best ideas until you have successfully gone around the SDLC without being stabbed in the back. The last few years things have gotten out of hand, with alleged outsourcing costs dropping and MS releasing a new crop of doodads which seldom live up to possibly inaccurate marketing claims... made by other C students who didn't make it through any symbolic math classes due to the rigorous integrity this requires. Stupid is forever & idiots you can't delegate away from will always wear you down.

The less you say the better. Say enough to get the job, don't impress them too much. They'll be skinning you and hanging your head on their wall before you get out of the room. If they start drooling show them something shiny, refer to magic & run for your life.

If you get a real crooked bunch they may try to encumber your IP for the rest of your life. You should be able to screw them up enough to leave you alone but they sometimes jump out of the bushes years later, claiming what you are doing is based on what you did for them. Better to never work professionally than write one line of code for these pimps.

I am rather fond of 1 patent lawyer. He never wrote any code, he's 'awesome' with this body of law. Coders who became lawyers make me nervous, like Lestat, and for the same reasons. Keep your personal IP lawyer on retainer so your employer can't hire him away from & undermine your support team.

If you want to find out if a programmer is any good, once you've vetted them let them spend between 20 minutes to 1 hour with the developers they'll be working with. Let them sample the kinds of projects they'll be working on, if not the actual project. Stick your head in toward the 2nd half & observe the interaction, then after they leave ask your people what they think. Unless you are Catbert this is much easier & accurate than devising labyrinthine one time pad interview trciks, or catchall methods that sort of work some of the time.

Poaching is for C students. If your company is any good the A+ students will find you. Try to be smart enough not to throw them out, or give them a broom. Believe me, it comes back.

Dave Ihnat

2010-09-15 09:41:35
Good as far as it goes, but I'm really not a fan of "write this program" on interviews. I've been a developer for over 30 years, a consultant for most of that time. I've forgotten more programming languages than most people have ever heard of. Over that time, there have been periods in which I was eyeball-deep in assemblers, C, C++, Java, Javascript--you name it. For each language, I ramped up *at that time*, and when I moved to something else, I necessarily let my day-to-day expertise in the others lapse. So as much time as I spent in 68000 or Intel assembly, today I'd have to look up specific opcodes; as much time as I spent in C++, today I'd need a reference text for the first day or two.

Languages are tools; they come and go. (Ghu, I wish I had the time back I spent teaching myself Ada...for nothing.) The real gold is knowledge of algorithms, data structures, models--the things that transcend any language. Problem solving, not codeslinging _per se_. Codeslinging comes fast for any experienced developer--I can ramp up in any language I've used before but not used in 5 years in a day or two; a new one may take a few more days. But I probably couldn't write compilable code on that first interview. I probably don't remember all the library calls--I'll remember they *exist*, but not the syntax.

On that basis, you wouldn't get me in the door. And it'd be your loss.

Don't hit them for a particular language and a compilable example; demand algorithmic pseudo-code and give them something that requires understanding the issues at hand (e.g., interprocess communication, etc.)

Frank Silbermann

2010-09-15 10:39:41
I would be quite annoyed by a question like, "Write a program to shuffle a deck of cards." I would ask, "Sure, but what is the API of the computer-controlled card deck shuffler? Most computers are not sold with peripherals that manipulate playing cards."

Steve Taylor

2010-09-15 11:58:40
I've had interviewers ask me the most absurd questions. Like: Tell me in detail how you'd make a sandwich. I know he was looking for how detail oriented I was, but I sensed that to the interviewer it was kind of a game. BTW, it was an interview with Microsoft and I didn't get the job, mainly because it was a support job and they didn't think I was a fit. Glad they thought so because I went on to better things.

If someone asked me to convert hex to decimal or binary manually, I'd ask why? In 15+ years of development work, I've never had to do it. And yes, I know the difference between hex and decimal.

I've also been on the other side of the table, and it's tough sometimes to find what the interviewee is all about. I have used the white board approach with some good success. For example, if I'm looking for a .Net developer, I might ask them to write a few lines of code to perform a certain function which pertains to the job requirements. One young interviewee who claimed to have good experience in ASP was asked to create a recordset from a command object, and couldn't do it. Really? No job for you.

Either way, on both side of the table, it's part skill and part art. I have found that team players with a positive attitude and a willingness -- even a desire -- to learn make the best hires, regardless of whether they can convert hex to decimal.

Rodrigo Silveira

2010-09-15 12:18:14
Excellent guidelines. On a personal note, many years ago I managed an engineering team and was charged by the CEO to hire a number of world class engineers. We knew we wanted them but we did not know how to find and get them. Most of the people hired under that program were losers. There is no question as to the importance of ensuring that the basics are covered as suggested in the article. But I go beyond, looking for character, motivation, and staying power (remember those death marches, they are still around!).

Alan

2010-09-15 14:10:24
@Jaco

If someone doesn't know that a semaphore is an OS construct and not a part of any language, then I don't want them wasting my time on "probation". So I'm going to ask a LOT of "stupid questions". And I do expect them to do the same. Next.

Jerry

2010-09-16 01:52:00
> If someone doesn't know that a semaphore is an OS

> construct and not a part of any language, then I

> don't want them wasting my time on "probation".

> So I'm going to ask a LOT of "stupid questions".

> And I do expect them to do the same. Next.

Ummm, which OS uses the term semaphore? Do you think everyone you interview will have used that OS? I've written C for more than one OS that didn't call them semaphores.

Arved Sandstrom

2010-09-16 05:10:54
In my experience, save the coding question for a take-home problem that a candidate can work on in a 24-48 hour timeframe. Ask for some tests as well. A well-chosen problem will provide a fair amount of evidence about coding skills, even without any direct knowledge of how much time it took the candidate nor of how much Googling they had to do.

Possibly related to the take-home problem, also ask for a one page sample of technical English. You're interested in grammar, spelling and logic. Just my opinion, but if a programmer can't clearly explain something technical, they can't design or code either.

I'd be careful with the specific questions (your Part III lead-in). Just because buddy says he knows concurrent programming, that doesn't mean he can (or ought to be able to) define a semaphore. And don't ask a candidate about a specific data structure if you can truthfully admit that you yourself had to look it up in the last couple of years. And given that a multitude of programmers use design patterns without ever having read all that much about them, don't assume that not knowing what a Builder or a Flyweight or a Strategy pattern is means that the guy doesn't know how to employ these patterns to solve problems.

I don't think your guidelines are that far off the mark, mind you.

maninblack

2010-09-16 16:54:42
@Jerry

Semaphores are a specific type of synchronization primitive and are common in the operating systems I have used.

Linux, FreeBSD: sem_init

Windows: CreateSemaphore

OS X: semaphore_create

Alex

2013-09-03 18:59:53
I am very impressed with the information presented here. I do believe that every developer/programmer should have side projects. It should not be about making them rich either! If you do something you love, it should not be a problem. Because the programmer is employed, there will be a need to hire a virtual assistant to assist with the administrative tasks of the side project. Some VAs are very knowledgeable in software and programming technology. One website that software developers can hire VAs is VAnow.net. It is one of best that I know has certified, skilled VAs.
Email
steve.hanov@gmail.com

Other posts by Steve

Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US How Asana Breaks the Rules About Per-Seat Pricing 5 Ways PowToon Made Me Want to Buy Their Software How I run my business selling software to Americans 0, 1, Many, a Zillion Give your Commodore 64 new life with an SD card reader 20 lines of code that will beat A/B testing every time [comic] Appreciation of xkcd comics vs. technical ability VP trees: A data structure for finding stuff fast Why you should go to the Business of Software Conference Next Year Four ways of handling asynchronous operations in node.js Type-checked CoffeeScript with jzbuild Zero load time file formats Finding the top K items in a list efficiently An instant rhyming dictionary for any web site Succinct Data Structures: Cramming 80,000 words into a Javascript file. Throw away the keys: Easy, Minimal Perfect Hashing Why don't web browsers do this? Fun with Colour Difference Compressing dictionaries with a DAWG Fast and Easy Levenshtein distance using a Trie The Curious Complexity of Being Turned On Cross-domain communication the HTML5 way Five essential steps to prepare for your next programming interview Minimal usable Ubuntu with one command Finding awesome developers in programming interviews Compress your JSON with automatic type extraction JZBUILD - An Easy Javascript Build System Pssst! Want to stream your videos to your iPod? "This is stupid. Your program doesn't work," my wife told me The simple and obvious way to walk through a graph Asking users for steps to reproduce bugs, and other dumb ideas Creating portable binaries on Linux Bending over: How to sell your software to large companies Regular Expression Matching can be Ugly and Slow C++: A language for next generation web apps qb.js: An implementation of QBASIC in Javascript Zwibbler: A simple drawing program using Javascript and Canvas You don't need a project/solution to use the VC++ debugger Boring Date (comic) barcamp (comic) How IE <canvas> tag emulation works I didn't know you could mix and match (comic) Sign here (comic) It's a dirty job... (comic) The PenIsland Problem: Text-to-speech for domain names Pitching to VCs #2 (comic) Building a better rhyming dictionary Does Android team with eccentric geeks? (comic) Comment spam defeated at last Pitching to VCs (comic) How QBASIC almost got me killed Blame the extensions (comic) How to run a linux based home web server Microsoft's generosity knows no end for a year (comic) Using the Acer Aspire One as a web server When programmers design web sites (comic) Finding great ideas for your startup Game Theory, Salary Negotiation, and Programmers Coding tips they don't teach you in school When a reporter mangles your elevator pitch Test Driven Development without Tears Drawing Graphs with Physics Free up disk space in Ubuntu Keeping Abreast of Pornographic Research in Computer Science Exploiting perceptual colour difference for edge detection Experiment: Deleting a post from the Internet Is 2009 the year of Linux malware? Email Etiquette How a programmer reads your resume (comic) How wide should you make your web page? Usability Nightmare: Xfce Settings Manager cairo blur image surface Automatically remove wordiness from your writing Why Perforce is more scalable than Git Optimizing Ubuntu to run from a USB key or SD card UMA Questions Answered Make Windows XP look like Ubuntu, with Spinning Cube Effect See sound without drugs Standby Preventer Stock Picking using Python Spoke.com scam Stackoverflow.com Copy a cairo surface to the windows clipboard Simulating freehand drawing with Cairo Free, Raw Stock Data Installing Ubuntu on the Via Artigo Why are all my lines fuzzy in cairo? A simple command line calculator Tool for Creating UML Sequence Diagrams Exploring sound with Wavelets UMA and free long distance UMA's dirty secrets Installing the Latest Debian on an Ancient Laptop Dissecting Adsense HTML/ Javascript/ CSS Pretty Printer Web Comic Aggregator Experiments in making money online How much cash do celebrities make? Draw waveforms and hear them Cell Phones on Airplanes Detecting C++ memory leaks What does your phone number spell? A Rhyming Engine Rules for Effective C++ Cell Phone Secrets