Hate UML?

Draw sequence diagrams in seconds.

Asking users for steps to reproduce bugs, and other dumb ideas
Posted on: 2010-05-27 21:00:00

A common misconception about software development:

  1. When a bug occurs, users will it into a tracking system with detailed information on how to reproduce it.
  2. A developer walks through the given steps to reproduce the issue, finds the problem, and submits a fix

That's based on several bad assumptions. Most users will not bother to enter bugs into your system. It is an unselfish act of altruism to enter a bug report. The user knows that it could be months, or even years before the bug is fixed, but she needs to be finished with your app by 5pm today.

The second problem is that many bugs are not reproducible. Maybe the bug depends on something the user was doing, the fact that she always clicks on the okay button instead of pressing enter, or because she installed a printer driver that replaced part of the OS with a modified, out of date library. Maybe your application is the first thing she uses in the morning while the hard drive is still chugging and 99 other programs are trying to update themselves at the same time.

Even worse: sometimes the problem just goes away on its own.

It's tempting to dismiss bug reports that you cannot reproduce. As a developer, you have enough work to do. The least they can do is tell you how to reproduce the problem then you'll have a chance at fixing it. Thousands of open bugs are closed or left to languish for years because they cannot be reproduced.

Refusing to fix a serious problem until you have reproducible steps is a cop-out excuse for lazy developers, who would rather get back to working on the physics engine for that secret flight simulator easter egg. That excuse might work for non-commercial software, but in the commercial software business, it will lose customers and get you fired.

How to fix non-reproducible bugs

Use a logging system

The best way to fix non-reproducible issues is to have adequate logging in the first place. Whenever the user does something, selects a menu, clicks "cancel", or inhales, record that action somewhere. Keep the file small, so the old parts scroll away. Take care to scrub anything that would violate privacy. Then, when a problem occurs, you can ask the user to attach the log file to the problem report.

You can use a well designed log as input to a test framework. You can then automatically reproduce the issue as many times as you need to test the fix, and ultimately make it part of your regression test suite.

Otherwise, fix by inspection

In 1981, Mark Weiser studied how experts debug software. The very best programmers create a mental slice of the program, so they only have to think about a few functions at a time. Weiser defined a program slice as the minimal program that still reproduces the problem. He developed an automatic method for finding the program slice to make debugging easier.

Unfortunately, thirty years later we are still doing things manually, and debugging requires lots of creative detective work. Use source control to ensure that you are looking at the same version of the software that your customer is using. Work backwards from the error message, keeping careful notes of the reverse call graph. If the X variable was set, what were the possible values of Y? Could this else clause have run? It's grueling work, and it takes days, but at the end of it, you will have a list of potential paths through the system that caused the error. And now you can methodically fix each one of them, without ever having reproduced the problem.

Sometimes, though, you will find that the problem logically could not have happened. In that case, you can either add more logging, or detect and recover, or do both.

Otherwise, detect and recover

Okay, so you had adequate logging. You've mentally traced through the source code for days. You've written additional tests for different theories and failed to reproduce the problem. The only way it could have happened is if a hole in the universe opened up and changed the laws of physics for a moment, or cosmic rays rewrote some register values. You will just have to detect and recover.

If slow memory leak causing the application to crash after three weeks, make sure your application restarts itself every few hours. If your complicated data structure somehow gets into an inconsistent state, write a function to go through and fix it after every single change. You get the idea.

Detecting and recovering from a mysterious bug is sometimes the only way to turn a major show-stopper into a minor annoyance. It is not the final solution, but it is a way to give you more time to find the real cause.

Get to it

The next time you get a serious bug that you can't reproduce, don't close it. Fix it, as if your job depended on it.

Want more programming tech talk?
Add to Circles on Google Plus
Subscribe to posts

Post comment

Real Name:
Your Email (Not displayed):

Text only. No HTML. If you write "http:" your message will be ignored.
Choose an edit password if you want to be able to edit or delete your comment later.
Editing Password (Optional):

Gaurav kumar

2013-08-17 10:14:30
I am not able to reproduce the bug, what should i do? Please give tell me. I am a Tester on Mobile application.

Other posts by Steve

Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US How Asana Breaks the Rules About Per-Seat Pricing 5 Ways PowToon Made Me Want to Buy Their Software How I run my business selling software to Americans 0, 1, Many, a Zillion Give your Commodore 64 new life with an SD card reader 20 lines of code that will beat A/B testing every time [comic] Appreciation of xkcd comics vs. technical ability VP trees: A data structure for finding stuff fast Why you should go to the Business of Software Conference Next Year Four ways of handling asynchronous operations in node.js Type-checked CoffeeScript with jzbuild Zero load time file formats Finding the top K items in a list efficiently An instant rhyming dictionary for any web site Succinct Data Structures: Cramming 80,000 words into a Javascript file. Throw away the keys: Easy, Minimal Perfect Hashing Why don't web browsers do this? Fun with Colour Difference Compressing dictionaries with a DAWG Fast and Easy Levenshtein distance using a Trie The Curious Complexity of Being Turned On Cross-domain communication the HTML5 way Five essential steps to prepare for your next programming interview Minimal usable Ubuntu with one command Finding awesome developers in programming interviews Compress your JSON with automatic type extraction JZBUILD - An Easy Javascript Build System Pssst! Want to stream your videos to your iPod? "This is stupid. Your program doesn't work," my wife told me The simple and obvious way to walk through a graph Asking users for steps to reproduce bugs, and other dumb ideas Creating portable binaries on Linux Bending over: How to sell your software to large companies Regular Expression Matching can be Ugly and Slow C++: A language for next generation web apps qb.js: An implementation of QBASIC in Javascript Zwibbler: A simple drawing program using Javascript and Canvas You don't need a project/solution to use the VC++ debugger Boring Date (comic) barcamp (comic) How IE <canvas> tag emulation works I didn't know you could mix and match (comic) Sign here (comic) It's a dirty job... (comic) The PenIsland Problem: Text-to-speech for domain names Pitching to VCs #2 (comic) Building a better rhyming dictionary Does Android team with eccentric geeks? (comic) Comment spam defeated at last Pitching to VCs (comic) How QBASIC almost got me killed Blame the extensions (comic) How to run a linux based home web server Microsoft's generosity knows no end for a year (comic) Using the Acer Aspire One as a web server When programmers design web sites (comic) Finding great ideas for your startup Game Theory, Salary Negotiation, and Programmers Coding tips they don't teach you in school When a reporter mangles your elevator pitch Test Driven Development without Tears Drawing Graphs with Physics Free up disk space in Ubuntu Keeping Abreast of Pornographic Research in Computer Science Exploiting perceptual colour difference for edge detection Experiment: Deleting a post from the Internet Is 2009 the year of Linux malware? Email Etiquette How a programmer reads your resume (comic) How wide should you make your web page? Usability Nightmare: Xfce Settings Manager cairo blur image surface Automatically remove wordiness from your writing Why Perforce is more scalable than Git Optimizing Ubuntu to run from a USB key or SD card UMA Questions Answered Make Windows XP look like Ubuntu, with Spinning Cube Effect See sound without drugs Standby Preventer Stock Picking using Python Spoke.com scam Stackoverflow.com Copy a cairo surface to the windows clipboard Simulating freehand drawing with Cairo Free, Raw Stock Data Installing Ubuntu on the Via Artigo Why are all my lines fuzzy in cairo? A simple command line calculator Tool for Creating UML Sequence Diagrams Exploring sound with Wavelets UMA and free long distance UMA's dirty secrets Installing the Latest Debian on an Ancient Laptop Dissecting Adsense HTML/ Javascript/ CSS Pretty Printer Web Comic Aggregator Experiments in making money online How much cash do celebrities make? Draw waveforms and hear them Cell Phones on Airplanes Detecting C++ memory leaks What does your phone number spell? A Rhyming Engine Rules for Effective C++ Cell Phone Secrets