Hate UML?

Draw sequence diagrams in seconds.
http://www.websequencediagrams.com

Four ways of handling asynchronous operations in node.js
Posted on: 2011-09-30 21:30:00

Javascript was not designed to do asynchronous operations easily. If it were, then writing asynchronous code would be as easy as writing blocking code. Instead, developers in node.js need to manage many levels of callbacks.

Today, we will examine four different methods of performing the same task asynchronously, in node.js. We will read in all the files in the current folder,

  1. In parallel, using callback functions
  2. Sequentially, using callback functions
  3. In parallel, using promises
  4. Sequentially, using promises

This will help you decide which to use for your particular situation. It is simply a matter of taste. If you want to know which is the best method -- the absolute best way to go -- is probably to switch to Golang.

Reading the files in parallel using callbacks

Our toy problem is to read all of the files in the current folder.

With our toy problem, we can't do everything in parallel. To read the files in the folder, we first need to know which files are in the folder. Thus we start with the readdir() function. We wait for the operation to complete. Then, for each file, we use the readFile() to get the contents of the file.

Here are the documentation for the functions, from node.js.

fs.readdir(path, [callback])

Asynchronous readdir(3). Reads the contents of a directory. The callback gets two arguments (err, files) where files is an array of the names of the files in the directory excluding '.' and '..'.

fs.readFile(filename, [encoding], [callback])

Asynchronously reads the entire contents of a file.

The callback is passed two arguments (err, data), where data is the contents of the file.

If no encoding is specified, then the raw buffer is returned.

All of the readFile() operations happen at once, and then we wait for the results to come in. We simply count how many times a readFile() operation has completed. When all of the files have been read, we know we are done.

    // Read all files in the folder in parallel.
    var fs = require("fs");

    fs.readdir( ".", function( err, files) {
        if ( err ) {
            console.log("Error reading files: ", err);
        } else {
            // keep track of how many we have to go.
            var remaining = files.length;
            var totalBytes = 0;

            if ( remaining == 0 ) {
                console.log("Done reading files. totalBytes: " +
                    totalBytes);
            }

            // for each file,
            for ( var i = 0; i < files.length; i++ ) {
                // read its contents.
                fs.readFile( files[i], function( error, data ) {
                    if ( error ) {
                        console.log("Error: ", error);
                    } else {
                        totalBytes += data.length
                        console.log("Successfully read a file.");
                    }
                    remaining -= 1;
                    if ( remaining == 0 ) {
                        console.log("Done reading files. totalBytes: " +
                            totalBytes);
                    }
                });
            }
        }
    });

Reading the files sequentially using callbacks

It is usually most efficient to to the above. Order the computer to do everything at once, and let the operating system sort it out. But that's not always what you want. Sometimes you need to impose an order and do things sequentially.

Here is an example of reading each file one at a time. The for loop is gone. It is replaced by a recursive function. The function checks to see if it has reached the last file. If so, it is done. Otherwise, it calls itself to process the next file in the list.

    // Read all the files in the folder in sequence, using callbacks
    var fs = require("fs");

    fs.readdir( ".", function( error, files ) {
        if ( error ) {
            console.log("Error listing file contents.");
        } else {
            var totalBytes = 0;

            // This function repeatedly calls itself until the files are all read.
            var readFiles = function(index) {
                if ( index == files.length ) {
                    // we are done.
                    console.log( "Done reading files. totalBytes = " + 
                        totalBytes );
                } else {

                    fs.readFile( files[index], function( error, data ) {
                        if ( error ) {
                            console.log( "Error reading file. ", error );
                        } else {
                            totalBytes += data.length;
                            readFiles(index + 1);
                        }
                    });
                }

            };

            readFiles(0);
        }
    });

Reading the files in parallel using Promises

A promise (also known a future, and sometimes a channel) is a concept from the 1970's that has recently become popular for Javascript programming. Promises are implemented in the node.js module promise. This module is not included unless you add it.

When you call a function and expect a return value, and the value is not yet available, the function instead returns a promise. The caller can then store the promise for later or schedule a subsequent operation when it completes. Promises can be seen as a more specialized form of node.js's EventEmitter, where the only two events are reject or resolve. Instead of using "on" to listen for the events, we use "then".

Here is a really simple example of promises being used.

var promise = doSomeAsynchronousOperation();
promise.then( function(result) {
    // yay! I got the result.
}, function(error) {
    // The promise was rejected with this error.
}

function doSomeAsynchronousOperation()
{
   var promise = new Promise.Promise();
   fs.readFile( "somefile.txt", function( error, data ) {
        if ( error ) {
            promise.reject( error );
        } else {
            promise.resolve( data );
        }
    });

    return promise;
}

Promises may be easier to deal with for some people, because functions that return promises are harder to misuse. You could forget whether a callback belongs in the 3rd or 4th parameter, but you can't make that mistake with a return value. Another advantage is that they encapulate the recursive function loop above. You can easily construct a super-promise from an array of promises, what will resolve only when each of its members resolve. That's we do in the code below, with Promise.all()

    var fs = require("fs");
    var Promise = require("promise");

    // Wrap the io functions with ones that return promises.
    var readdir_promise = Promise.convertNodeAsyncFunction(fs.readdir);
    var readFile_promise = Promise.convertNodeAsyncFunction( fs.readFile );

    p = readdir_promise( "." );
    p.then( function( files ) {

        // Create an array of promises
        var promises = [];

        for ( var i = 0; i < files.length; i++ ) {
            promises.push( readFile_promise( files[i] ) );
        }

        Promise.all( promises ).then( function(results) {
            var totalBytes = 0;
            for ( i = 0; i < results.length; i++ ) {
                totalBytes += results[i].length;
            }
            console.log("Done reading files. totalBytes = " + totalBytes);
        }, function( error ) {
            console.log("Error reading files");
        });

    }, function( error ) {
        console.log( "readdir failed.");

    });

Reading the files sequentially using Promises

By default, promises don't support sequential operations very well. But we can build on them using a PromiseSequence, which adds the ability to define a series of steps and loops, which are performed sequentially.

Here is the program again, reading a file. Instead of indenting the code by many levels, we are able to write it in more of a sequential style. Also, the above examples had two places where errors had to be handled. With a promise sequence, the errors for any operation in the sequence are handled in one place.

We first add the readdir() operation to the sequence, and then add a loop() to the sequence. The loop executes repeatedly until exitLoop() is called. Since there are no further step, the argument to exitLoop resolves the promise and the program ends.

    // Read all the files in the folder in a sequence, using Promises
    var fs = require("fs");
    var Promise = require("promise");
    var PromiseSequence = require("./PromiseSequence").PromiseSequence;

    // Wrap the io functions with ones that return promises.
    var readdir_promise = Promise.convertNodeAsyncFunction(fs.readdir);
    var readFile_promise = Promise.convertNodeAsyncFunction( fs.readFile );

    var seq = new PromiseSequence();
    var index = 0;
    var totalBytes = 0;
    var files = null;

    seq.add( function() {
        return readdir_promise( "." );
    });

    seq.loop( 
        // The "next" function of the loop takes the result of the readdir and
        // reads the file. It is executed when the loop is entered, and again after
        // each time the body is executed.
        function( files_arg ) {
            files = files_arg;
            if ( index == files.length ) {
                seq.exitLoop(totalBytes);
                return;
            } else {
                console.log("Reading file " + files[index]);
                return readFile_promise( files[index++] );
            }
        },

        // The "body" function of the loop is called with the result of the "next" function.
        // It simply sums the length of the file.
        function( contents ) {
            totalBytes += contents.length;
        }
    );

    seq.run().then( function(total) {
        console.log("Done reading file. Total bytes: " + total);
    }, function(error) {
        console.log("Error reading files: ", error);
    });

Why does my brain hurt?

I have been programming for over two decades. I know lots of languages, but only Javascript makes my brain hurt when I have to do simple things. If you have a lot of asynchronous operations to perform, and you have the choice, please consider your language selection very carefully.

Want more programming tech talk?
Add to Circles on Google Plus
Subscribe to posts

Post comment

Real Name:
Your Email (Not displayed):

Text only. No HTML. If you write "http:" your message will be ignored.
Choose an edit password if you want to be able to edit or delete your comment later.
Editing Password (Optional):

Chinthaka, MIT, UOK, SL

2011-11-19 07:08:11
Maxaa, For my application, this helped a lot, thanks a lot

David

2012-05-16 18:58:55
for complex asynchronous operations I recommend using the async module, just google "caolan/async"

Sînicã

2014-01-28 06:23:13
I want to show you my new node module, called harvests. You could try harvests and get simpler syntax compared with promises and flow libraries. github.com/salboaie/harvests
Email
steve.hanov@gmail.com

Other posts by Steve

Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US How Asana Breaks the Rules About Per-Seat Pricing 5 Ways PowToon Made Me Want to Buy Their Software How I run my business selling software to Americans 0, 1, Many, a Zillion Give your Commodore 64 new life with an SD card reader 20 lines of code that will beat A/B testing every time [comic] Appreciation of xkcd comics vs. technical ability VP trees: A data structure for finding stuff fast Why you should go to the Business of Software Conference Next Year Four ways of handling asynchronous operations in node.js Type-checked CoffeeScript with jzbuild Zero load time file formats Finding the top K items in a list efficiently An instant rhyming dictionary for any web site Succinct Data Structures: Cramming 80,000 words into a Javascript file. Throw away the keys: Easy, Minimal Perfect Hashing Why don't web browsers do this? Fun with Colour Difference Compressing dictionaries with a DAWG Fast and Easy Levenshtein distance using a Trie The Curious Complexity of Being Turned On Cross-domain communication the HTML5 way Five essential steps to prepare for your next programming interview Minimal usable Ubuntu with one command Finding awesome developers in programming interviews Compress your JSON with automatic type extraction JZBUILD - An Easy Javascript Build System Pssst! Want to stream your videos to your iPod? "This is stupid. Your program doesn't work," my wife told me The simple and obvious way to walk through a graph Asking users for steps to reproduce bugs, and other dumb ideas Creating portable binaries on Linux Bending over: How to sell your software to large companies Regular Expression Matching can be Ugly and Slow C++: A language for next generation web apps qb.js: An implementation of QBASIC in Javascript Zwibbler: A simple drawing program using Javascript and Canvas You don't need a project/solution to use the VC++ debugger Boring Date (comic) barcamp (comic) How IE <canvas> tag emulation works I didn't know you could mix and match (comic) Sign here (comic) It's a dirty job... (comic) The PenIsland Problem: Text-to-speech for domain names Pitching to VCs #2 (comic) Building a better rhyming dictionary Does Android team with eccentric geeks? (comic) Comment spam defeated at last Pitching to VCs (comic) How QBASIC almost got me killed Blame the extensions (comic) How to run a linux based home web server Microsoft's generosity knows no end for a year (comic) Using the Acer Aspire One as a web server When programmers design web sites (comic) Finding great ideas for your startup Game Theory, Salary Negotiation, and Programmers Coding tips they don't teach you in school When a reporter mangles your elevator pitch Test Driven Development without Tears Drawing Graphs with Physics Free up disk space in Ubuntu Keeping Abreast of Pornographic Research in Computer Science Exploiting perceptual colour difference for edge detection Experiment: Deleting a post from the Internet Is 2009 the year of Linux malware? Email Etiquette How a programmer reads your resume (comic) How wide should you make your web page? Usability Nightmare: Xfce Settings Manager cairo blur image surface Automatically remove wordiness from your writing Why Perforce is more scalable than Git Optimizing Ubuntu to run from a USB key or SD card UMA Questions Answered Make Windows XP look like Ubuntu, with Spinning Cube Effect See sound without drugs Standby Preventer Stock Picking using Python Spoke.com scam Stackoverflow.com Copy a cairo surface to the windows clipboard Simulating freehand drawing with Cairo Free, Raw Stock Data Installing Ubuntu on the Via Artigo Why are all my lines fuzzy in cairo? A simple command line calculator Tool for Creating UML Sequence Diagrams Exploring sound with Wavelets UMA and free long distance UMA's dirty secrets Installing the Latest Debian on an Ancient Laptop Dissecting Adsense HTML/ Javascript/ CSS Pretty Printer Web Comic Aggregator Experiments in making money online How much cash do celebrities make? Draw waveforms and hear them Cell Phones on Airplanes Detecting C++ memory leaks What does your phone number spell? A Rhyming Engine Rules for Effective C++ Cell Phone Secrets