Steve Hanov's Programming Blog
Windows, C++ programming techniques, Mobile technology,
neat tricks you can do with Computer Science algorithms.
2007-04-15T16:29:29Z
tag:smhanov,2007:1
Copyright (c) 2007, Steve Hanov
Example Toolkit
I found Security Vulnerability in your web application
tag:smhanovtechblog,2007:id149
2022-01-28T10:54:57-05:00
2022-01-28T10:54:57-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<blockquote>
I found Security Vulnerability in your web application. For security purpose can we report vulnerability here,then will i get bounty reward in PayPal or Bitcoin for Security bug ?
</blockquote>
Is it just me, or are security consultants swarming web sites, looking for bugs unasked, and emailing you demanding thousands of dollars in bounties for security flaws? If you search the web for the exact email above, you will get hundreds of hits. Whether the cost is to an unsolicited consultant, or due to a data breach, flaws in your product can be pricey. In this page, I'm going to write down some basic things to check in your website to make attacks harder, so at least you're not being shaken down for stupid mistakes.
<p>
I have ceased any bounty payments, because the things they generally find are not very likely to happen. I have to make a business decision: Do I want a perfect web application, or can I live with a few possible attacks? I am not Apple. I am not a bank. This is going to sound foolish, but at the end of the day I simply cannot afford to pay out half of my revenue each and every month for these things, because I still have to provide for my family.
<p>
Still, here is a list of the things I have learned. I am sure there is a larger list somewhere, but I have not found it.
<h2>Don't put sensitive information in URLs</h2>
When a user is logged in, what is in the URL at the top of the screen? Is there an identifier in there that should not be shared?
<ul>
<li>If the user pasted any URL on your site, while logged in, to Twitter, could anybody get access to something they should not? Could other users edit this user's files?
<li>If a script on your page that you have no control over reads window.location and sends it back to its creator, is this a problem?
</ul>
When possible, store sensitive identifiers in cookies that cannot be accessed in Javascript. More importantly, always verify on the server that the user has access to the data that he or she is requesting.
<h2>Verify any information you include in generated emails</h2>
One consultant found a way to send password reset emails from me to any user, and replace the URL in the email with his own. How did he do this? Because I was lazy.
<p>
In my code, I had abstracted the password reset emails into a reusable library. The code needed to know which web site it was resetting the password for when it created the email. So I just included it in the POST request. And this resulted in a costly bounty I had to pay.
<h2>Rate limit generated emails</h2>
On that note, can anyone make you send an unlimited number of emails? Any code that automatically sends an email needs rate limiting.
<p>
<h2>Prevent password guessing</h2>
It's a fact that most people's passwords can be guessed in less than 100 tries. That's why they should use a password manager and create longer and more random passwords.
<p>
So one of the first things a consultant will do is figure out your API call to login and run it through their script file of passwords. If they don't get stopped right away, then this will result in a significant bounty.
<p>
You need to rate login attempts, in at least two ways. You should of course limit the attempts for a particular login name. But then the consultant will just cycle through different login names. You will need to also limit it based on IP address or other information as well.
<h2>Put limits on file uploads</h2>
Can you accept an image on your website? Do you resize it on the server? Great, let me dig out my 18 megapixel .png file for you. It's only 1KB so your code will accept it. Without proper checks, it will crash your server.
<h2>Protect "secret" web urls</h2>
Do you have any secret web urls in your app for checking its status, or performing administration? Maybe you use go, and left in the code for /debug/pprof? All secret urls can be easily guessed using automated tools that just try every word or combination of characters and see if they get a 404 error.
<p>
Assuming the consultant can list all the secret urls on your web site, can they do anything bad?
<h2>UI Redressing</h2>
If you allow your web site to be placed into an IFRAME it opens up a lot of attacks. If the attacker can trick a user into clicking on their url, then they will open up your web site inside their own page, and super-impose their login button over yours, and steal ther user's passwords.
<p>
Of course, anyone can do this without the iframe permissions by creating an exact copy of your site. But the point is to make it a little harder for them.
Here is some <a href="https://stackoverflow.com/questions/2896623/how-to-prevent-my-site-page-to-be-loaded-via-3rd-party-site-frame-of-iframe">information</a> on IFRAME breaking.
<ul><li><a href='?id=58'>Email Etiquette</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=31'>Free, Raw Stock Data</a></ul>
How to detect if an object has been garbage collected in Javascript
tag:smhanovtechblog,2007:id148
2020-05-25T10:35:28-05:00
2020-05-25T10:35:28-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
If you are writing an application in Javascript, soon you will have to worry about memory leaks. But it is difficult to even know if a memory leak exists. This handy method can help.
<h3>WeakMap</h3>
At first, you may think that WeakMap will do it. WeakMap/WeakSet will hold onto things for you, but don't prevent an object from being garbage collected. The instant an object is GC'd, it is removed from the WeakMap or WeakSet.
<p>
So, the obvious solution is to check if an object is still inside the WeakMap. If it is missing, it has been GC'd. This won't work.
<p>
The problem is that WeakMap and WeakSet are designed so that you cannot get at what's inside without already knowing it is there. In order to lookup an item, you need to <i>already have that item</i>. These collections don't even have a <tt>length</tt> method.
<p>
To check if an object is inside of a WeakMap, you must already have a reference to it, and therefore you are preventing it from being garbage collected.
<p>
So, what good are they? <em>WeakMap is best used to link objects together.</em> For example, if you have a bunch of <img> elements and you want to associate some data with them, you could simply do <tt>img.myextraproperty="blah"</tt>. But your IDE may complain because the HTMLImageElement does not have this property. Instead, you can use WeakMap. If the extra property is a single value <tt>true</tt> then use a WeakSet.
<h3>The Real Solution</h3>
Some browsers, including Chrome but not Firefox, have the ability to check the amount of Javascript memory used. So the solution to test if an object is there, is to make it sufficiently large so that it has a noticeable impact on memory.
<p>
In the code below, I use a WeakMap to associate a 1 gigabyte object with whatever you pass in. When the object is freed, and the garbage collector is run, you would expect at least 1 GB of memory to be freed as well. That is what the code checks for. The process takes at least 10 seconds, because it seems that Chrome only runs the garbage collector every 10 seconds.
<p>
<a href="https://codepen.io/smhanov/pen/bGVZKYo">View on CodePen</a>
<pre>
/** Determines if an object is freed
@param obj is the object of interest
@param freeFn is a function that frees the object.
@returns a promise that resolves to {freed: boolean, memoryDiff:number}
@author Steve Hanov <steve.hanov@gmail.com>
*/
function isObjectFreed(obj, freeFn) {
return new Promise( (resolve) => {
if (!performance.memory) {
throw new Error("Browser not supported.");
}
// When obj is GC'd, the large array will also be GCd and the impact will
// be noticeable.
const allocSize = 1024*1024*1024;
const wm = new WeakMap([[obj, new Uint8Array(allocSize)]]);
// wait for memory counter to update
setTimeout( () => {
const before = performance.memory.usedJSHeapSize;
// Free the memory
freeFn();
// wait for GC to run, at least 10 seconds
setTimeout( () => {
const diff = before - performance.memory.usedJSHeapSize;
resolve({
freed: diff >= allocSize,
memoryDiff: diff - allocSize
});
}, 10000);
}, 100);
});
}
let foo = {bar:1};
isObjectFreed(foo, () => foo = null).then( (result) => {
document.write(`Object GCd:${result.freed}, ${result.memoryDiff} bytes freed`)
}, (error) => {
document.write(`Error: ${error.message}`)
})
</pre>
<h3>How I use it</h3>
I use this method as part of my test suite for <a href="https://zwibbler.com">Zwibbler</a>, my Javascript drawing app. It has a <a href="https://zwibbler.com/docs/#destroy">destroy() method</a> that is supposed to remove all resources. But occasionally, I would have some event listener that was not removed, that would keep a reference to the entire application. So when using it inside something like React or Angular, where it can be repeatedly shown and hidden by the view framework, it is vitally important that resources be completely freed.
<ul><li><a href='?id=59'>When a reporter mangles your elevator pitch</a><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=74'>Blame the extensions (comic)</a><li><a href='?id=52'>Automatically remove wordiness from your writing</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a></ul>
My favourite Google Cardboard Apps
tag:smhanovtechblog,2007:id147
2017-02-15T10:46:40-05:00
2017-02-15T10:46:40-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
I have never been a gamer. The most I've played was Super Mario Bros (the original). I then took a break for a decade or so and spent a few weeks with Simcity 4.
<p>
Something happened last week. Overnight, I've become addicted to games. The cause was this:
<p align=center><img src="cardboard.jpg"></p>
It arrived the next day after I <a href="https://vr.google.com/intl/en_ca/cardboard/get-cardboard/">ordered it from Google.</a> This is very surprising as I am in Canada.
<p>
I now have nearly a hundred games on my old Samsung S5. Here are a few of my favourites.
<h2>First, Some Cardboard Basics</h2>
It took me a few hours to figure out how to navigate in various games. I'll save you the trouble. There are three methods of control.
<p>
<h3>Stare-to-click</h3>
The most frequently used is the stare. Stare at a button on the screen, and a circular indicator will count down for a second or so. Keep looking until it runs out to click.
<p>
<h4>Stare and click</h4>
An alternative method of control is a button on the cardboard unit itself. The original Cardboard has a slider on the left side that moves a magnet. However, the newly redesigned units have a button on the top right that you can press. It moves a lever that taps the top of the screen. (As a side effect, if somebody sends you a Facebook message while you are playing, it will send them a thumbs up.)
<p>
<h4>Walk in place</h4>
A minority of games require you to walk in place to move. Simply bounce your head up and down slightly and you will travel in the direction that you are looking.
<p>
<h4>Other control methods</h4>
You don't need any other controller. Only a couple of games require a Bluetooth or USB gamepad to move. One game studio uses <a href="http://www.realiteer.com/build-realtrigger/">the device's camera and a large QR code</a> that you point in front of you to move around. However, it falls back to the point-and-stare method.
<p>
<h4>How to sit</h4>
It is best to play in a darkened room on a swivel chair. This lets you look around without getting too tired or bumping into things. Also, my phone tends to drift slowly to the right, so I can keep spinning to keep up.
<p>
Don't forget to wear headphones. The sound is 3D too, so your ears will tell you where things are happening.
<h2>Terrifying experiences</h2>
Quite a few games consist of wandering around a haunted house. However, the act of walking makes gameplay difficult. That's why I like these two games. You sit in one place and stuff happens around you. Remember to keep looking around, because the ghouls will patiently wait for you to look away before terrorizing you from behind.
<p>
In <a href="https://play.google.com/store/apps/details?id=com.RyanBousfield.AChairInARoom">Chair in a room</a>
you can select from two stories. They each take about 15 minutes to play. You are fixed in one spot, sitting on a chair in the dark. You must gradually piece together the story from what is happening around you.
<p align=center><img src="chairinaroom.webp"></p>
<p>
<a href="https://play.google.com/store/apps/details?id=com.otherworld.Sisters">Sisters</a> includes two experiences as well. In the Blair Witch trailer, you are standing outside an abandoned house in the forest and stuff happens. When you have completed it, play Sisters. You are sitting on a couch during a storm, and two creepy dolls sit on the mantlepiece. The power goes out. You're on your own.
<p align=center><img src="blairwitch.webp"></p>
<h2>Wandering around</h2>
<a href="https://play.google.com/store/apps/details?id=com.software.mick.aaplus">Alien Apartment</a> sets itself apart by its remarkable attention to detail. Using a unique control method -- tilt your head to walk, and tilt again to stop -- wander around a neat, modern apartment. The massive living room windows overlook an alien world. The textures, subtle lighting, and spacy soundtrack make for a visual feast.
<p align=center><img src="alienapartment.webp"></p>
<p>
Alien Apartment only one scene of a larger work, <a href="https://play.google.com/store/apps/details?id=com.mick.we.ep0">Whispering Eons</a>, but I've not tried it yet.
<p align=center><img src="whisperingeons.webp"></p>
<h2>Sit and watch</h2>
In <a href="https://play.google.com/store/apps/details?id=com.creanet3d.atis2free2">A Time in Space 2</a>, a cute robot leads you on a short space adventure. For some reason, the stereoscopic image is full-screen rather than tailored for cardboard, so it makes me a little nauseous. But the experience is worth it.
<p align=center><img src="atis2.webp"></p>
<p>
<a href="https://play.google.com/store/apps/details?id=com.vw.vrcave">VR Cave</a> stands out because of its incredible detail and lends itself well to 3D. You float through a cave on a predefined track. It is stunning to see the crystals whiz by inches from your face while looking down vast bottomless caverns of rock. Sit on a swivel chair and slowly spin while you are doing it, so that you can trick yourself into feeling your feet drag on the rock as you go.
<p align=center><img src="vrcave.webp"></p>
<p>
I tried a few roller coasters. <a href="https://play.google.com/store/apps/details?id=com.frag.vrrollercoaster">VR Roller Coaster</a> is reasonable.
<p>
Need to chill out? Install <a href="https://play.google.com/store/apps/details?id=com.donytamazone.androidsdream.cardboard">Android Dreams</a> and sit back while your driver serenely lands your craft in a futuristic city at night. Gaze out the window as you cruise between giant billboards and skyscrapers.
<p align=center><img src="androiddreams.webp"></p>
<a href="https://play.google.com/store/apps/details?id=com.shakingearthdigital.vrsecardboard">Within</a> has a few dozen VR movies to choose from. My favourites are the two science documentaries on gravity waves and the robots of Boston Dynamics. You can experience several VR music videos, or sit in the audience during a taping of Saturday Night Live.
<p>
The <a href="https://www.youtube.com/channel/UCzuqhhs6NWbgTzMuM09WKDQ">Youtube Virtual Reality Channel</a> includes many short videos in VR. You can try them without cardboard by dragging the screen around, but the headset makes them shine.
<p>
The <a href="https://play.google.com/store/apps/details?id=com.velotech.veer">Veer app</a> has a curated selection of videos that were much better than the Youtube ones.
<p align=center><img src="veer.webp"></p>
<h2>Puzzle</h2>
I feel like I'm in the movie Tron when I'm playing <a href="https://play.google.com/store/apps/details?id=com.VRMersive.GravityDrop">Gravity Pull</a>. Solve puzzles by putting boxes onto weight sensitive pads, unlocking each door to the next room.
<p align=center><img src="gravitypull.webp"></p>
<h2>Action</h2>
Sorry, you're going to have to part with some cash, because the best action games cost a couple of bucks.
<p>
<a href="https://play.google.com/store/apps/details?id=com.ZeroTransform.ProtonPulse">Proton Pulse</a> is brickbreaker. Move your head to bounce the ball off the glass prism and destroy all of the floating bricks.
<p align=center><img src="protonpulse.webp"></p>
<p>
Install <a href="https://play.google.com/store/apps/details?id=com.endspacevr">Minos Starfighter</a>. In your swivel chair, you'll feel like you're in an X-Wing. Make sure you look up and down as you spin because the attacking ships are coming at you from all directions. They explode with a satisfying fireball under the wrath of your cannons.
<p align=center><img src="minosstarfighter.webp"></p>
<h2>WTF</h2>
You have to to see this to believe it. For someone completely uninitiated to Japanese anime, <a href="https://play.google.com/store/apps/details?id=com.dlsite.nagomimi">Nagomi's Earcleaning VR</a> is a constant stream of WTF moments. In the game, you are visiting your young, attractive cousin, and in dialog laced with innuendo, she beckons you to lean on her lap. Then you hear audible scratching as she proceeds to clean your ears. She looks hurt and mortified if you try to escape. Better lay down and ponder the quirkiness of Japanese culture.
<p align=center><img src="earcleaning.jpg"></p>
<h2>What my kids like</h2>
Do you have small kids? At ages five and seven, Mine like these apps:
<ul>
<li><a href="https://play.google.com/store/apps/details?id=com.archiactinteractive.LamperVRCardboard">Lamper VR: Firefly Rescue</a>
<li><a href="https://play.google.com/store/apps/details?id=com.sidekick.Aquarium">Aquarium</a>
<li><a href="https://play.google.com/store/apps/details?id=com.Sidekick.vrbowling">Bowling VR</a>
<li><a href="https://play.google.com/store/apps/details?id=com.planner5d.planner5d">Planner 5D</a>
<li><a href="https://play.google.com/store/apps/details?id=com.Geomedia.DinoTrekVRECardboard">DinoTrek VR</a>
</ul>
<p>
<h2>What are your favourite VR apps?</h2><ul><li><a href='?id=138'>Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US</a><li><a href='?id=115'>Compressing dictionaries with a DAWG</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=57'>Coding tips they don't teach you in school</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=10'>Detecting C++ memory leaks</a></ul>
O(n) Delta Compression With a Suffix Array
tag:smhanovtechblog,2007:id146
2017-01-24T10:16:32-05:00
2017-01-24T10:16:32-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<h3>ABSTRACT</h3>
The difference between two sequences A and B can be compactly stored using COPY/INSERT operations. The greedy algorithm for finding these operations relies on an efficient way of finding the longest matching part of A of any given position in B. This article describes how to use a suffix array to find the optimal sequence of operations in time proportional to the length of the input sequences. As a preprocessing step, we find and store the longest match in A for every position in B in two passes over the suffix array that has been enhanced with longest common prefix information (LCP).
<h3>INTRODUCTION</h3>
The days of losing work due to a power outage are over. In modern applications, users expect their work to be saved immediately. They also make mistakes, such as accidentally deleting large sections, and they expect to be able to restore their work to a prior state. To facilitate this, we need a way to retrieve or reconstruct every version of a document. We can take advantage of the similarities between the versions to minimize the space they consume.
<h3>INSERT/DELETE ALGORITHMS</h3>
Given two sequences of items A and B, we can compute a set of operations that will transform A into B. That way, only these operations need be stored. Two main classes of differencing algorithms are commonly used. Version control tools that deal with source code are often based on the longest common subsequence. For example, given the text:
<p>
A: The quick brown fox jumped over the lazy dog.
<p>
B: The lazy dog jumped over the quick brown fox.
<p>
The longest common subsequence is:
<p>
(The )(jumped over the )(.)
<p>
Although the text is written out here for clarity, only the position and length of each matching block is used. Using this information, one can directly compute the ranges where the text changed and thus derive INSERT / DELETE operations.
<p>
The example above can be encoded:
<p>
<pre>AT 5 DELETE "quick brown fox "
AT 5 INSERT "lazy dog"
AT 30 DELETE "lazy dog"
AT 30 INSERT "quick brown fox "
</pre>
While INSERT/DELETE changes are easy to see visually in side-by-side comparison tools, they are suboptimal for file storage because they cannot exploit out-of-sequence similarities.
<p>
We can fix this by adding a MOVE operations. It is possible to change pairs of INSERT/DELETE commands to MOVE as a post-processing step. However, a more pressing issue is that any algorithm based on the longest common subsequence has a O(N^2) runtime in its worst case. Modern differencing tools often use Meyer's O(ND) algorithm, which is proportional to the length of the strings and the number of differences between them. Of course, when the two texts share little similarity, that will take a very long time.
<p>
<h3>THE COPY/INSERT ALTERNATIVE</h3>
<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.470.9110&rep=rep1&type=pdf">Tichy describes system based only on COPY/INSERT</a>. Starting with an empty string, it is possible to recreate a B by copying from various positions of A. Portions that do not exist in A can be created using the INSERT command. It can also be used when the encoding of the insert command would be smaller than the equivalent copy.
<p>
A: The quick brown fox jumped over the lazy dog.
<p>
<pre>COPY (The )
COPY (lazy dog)
COPY (jumped over the )
COPY (quick brown fox)
INSERT (.)
</pre>
Finding COPY/INSERT operations is much simpler than LCS based algorithms.
<p>
<pre>
# the position of the string.
q = 0
while q < len(B):
find p and l such that (p.q.J) is a maximal block move
position, length = findLongestMatch(q)
if length > 0:
If insertFrom >= 0:
outputInsertCommand(insertFrom, q - insertFrom)
insertFrom = -1
outputCopyCommand(position, length)
q += length
elif insertFrom == -1:
insertFrom = q
if insertFrom >= 0:
outputInsertCommand(insertFrom, q - insertFrom)
</pre>
<p>
The runtime of the algorithm is dependent on the findLongestMatch function. Given a position in B, it finds the position in A with the longest matching sequence.
<p>
The brute force solution, which simply compares each possible position of A each time, performs surprisingly well when the sequences are mostly similar. It is not called very often, because the matches it finds are long. For sequences that are different, it again devolves into an O(N^2) running time.
<p>
The algorithm presented by <a href="http://www.xmailserver.org/xdfs.pdf">MacDonald (2000)</a> for use in the XDFS file system uses a preprocessing step to achieve O(N) operation. At each offset in A, the next 16 characters are placed into a lookup table and mapped to that position. FindLongestMatch is then:
<p>
<pre>
code = next 16 characters in A
If code exists in the table,
Return the length of the longest match at A[table[code]:] and B[index:]
Else
No match at this position.
</pre>
This algorithm is very fast and reasonably thorough. However, it is often not optimal. To guarantee O(N) time, the algorithm makes a tradeoff. When the table of positions is built, existing entries are "clobbered" by later ones. Only one position for each code is stored. If all of the positions were stored, then the algorithm would have to check each one, which would make the overall runtime O(N^2) with certain combinations of inputs.
<p>
<h3>THE SUFFIX ARRAY</h3>
A suffix array is an ordered list of positions in the string. Modern suffix array construction algorithms will bucket sort certain positions in the string, and then use the information to "induce" the positions of the other characters. With this clever trick, a suffix array can be created from a sequence in O(N) time where N is the length of the sequence.
<p>
It is often useful to build an enhanced suffix array. In addition to the suffix array of length N, the longest common prefix (LCP) array of length N-1 is built. IT contains the length of the longest common prefix between each entry and the next. By exploiting the commonalities in the sequence, these prefixes can also be computed in O(N) time, either during the suffix array creation, or as a separate step.
<p>
To find commonalities between two different sequences, they are appended together, separated by a character not found in either string, and a suffix array is constructed. An example is here:
<p>
A: "mississippi" + "u0001"
<p>
B: "sips and misses" + "u0000"
<p>
<pre>
String Index Lcp 15 characters
B 15 0 |"u0000"
A 11 0 |"u0001sips and missesu0000"
B 4 1 |" and missesu0000"
B 8 0 |" missesu0000"
B 5 0 |"and missesu0000"
B 7 0 |"d missesu0000"
B 13 0 |"esu0000"
A 10 1 |"iu0001sips and missesu0000"
A 7 2 |"ippu0001sips and misses"
B 1 1 |"ips and missesu0000"
B 10 3 |"issesu0000"
A 4 4 |"issippiu0001sips and mis"
A 1 0 |"ississippiu0001sips and "
B 9 4 |"missesu0000"
A 0 0 |"mississippiu0001sips and"
B 6 0 |"nd missesu0000"
A 9 1 |"piu0001sips and missesu0000"
A 8 1 |"ppiu0001sips and missesu0000"
B 2 0 |"ps and missesu0000"
B 14 1 |"su0000"
B 3 1 |"s and missesu0000"
B 12 1 |"sesu0000"
<span style="background:yellow">A 6 3 |"sippiu0001sips and misse"
B 0 2 |"sips and missesu0000"
A 3 1 |"sissippiu0001sips and mi"</span>
B 11 2 |"ssesu0000"
A 5 3 |"ssippiu0001sips and miss"
A 2 0 |"ssissippiu0001sips and m"
</pre>
<p>
Note 1. The position of sequence 1 and 2 can be easily determined by its offset into the concatenated string.
<p>
Note 2. The LCP at index i efers to the commonality with position i+1 in the suffix array.
<p>
Note 3. The LCP between any two entries in the suffix array is the minimum of the LCP of all adjacent entries between them.
<p>
The common parts of the combined string AB are near each other in the the suffix array. However the common parts of A and B are not necessarily adjacent. If either string has commonalities with itself, then there will be two or more adjacent entries in the suffix array belonging to the same string.
<p>
Examining the suffix array above, we see that the location "sips and misses" in B has below it a match in string A of length 2. However, above it is a match of length 3. Our algorithm must consider both possibilities.
<p>
We do this in two passes. In the forward pass, we find the longest common prefixes between B and any part of A previous to it in the suffix array. In the reverse pass, we find the LCPs between B and any part of a that is after it in the suffix array.
<p>
<pre>
def longestMatches(self):
# returns, for every position in B, a tuple with the longest matching
# position in A and the length of that match.
result = [None] * self.length2
# forward pass
lcp = 0
aIndex = 0
for i in range(len(self.sa)):
if self.sa[i] < self.length1:
# string in A
lcp = self.lcp[i]
aIndex = self.sa[i]
else:
# string in B.
result[self.sa[i] - self.length1] = (aIndex, lcp)
lcp = min(lcp, self.lcp[i])
# reverse pass
lcp = 0
aIndex = 0
for i in range(len(self.sa)-1, -1, -1):
if self.sa[i] < self.length1:
# string in A
aIndex = self.sa[i]
if i > 0:
lcp = self.lcp[i-1]
else:
# string in B.
lcp = min(lcp, self.lcp[i])
bIndex = self.sa[i] - self.length1
oldAIndex, oldLcp = result[bIndex]
if lcp > oldLcp:
result[bIndex] = (aIndex, lcp)
return result
</pre>
<h3>ANALYSIS</h3>
Even though building the suffix array is an O(N) algorithm, it will always be slower compared to the XDFS method because it requires several passes through the data. If optimal encoding is required, and the algorithm must be O(N) for all inputs, then the suffix array method may be considered. <a href="https://neil.fraser.name/writing/diff/">Preprocessing operations</a>, such as removing the common prefix and suffix from the input, are important to reduce the problem size.
<h3>RESOURCES</h3>
Here is the <a href="https://gist.github.com/smhanov/89ddefcd00f3dcc76cef23319cdc433d">Python source code</a> that I used to test this technique. It contains a transcription of the <a href="http://zork.net/~st/jottings/sais.html">SAIS suffix array construction algorithm</a> from C into Python.
<ul><li><a href='?id=11'>Cell Phones on Airplanes</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=65'>Drawing Graphs with Physics</a><li><a href='?id=85'>barcamp (comic)</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a></ul>
Finding Bieber: On removing duplicates from a set of documents
tag:smhanovtechblog,2007:id144
2014-11-10T08:00:00-05:00
2014-11-10T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
So I have two million song lyrics in a big file. Don't ask me how I got it. The point is that I want to find the most poetic phrase of all time.
<p>
Problem is, the origins of this file are so sketchy it would make a <i>Pearls Before Swine</i> cartoon look like a Da Vinci. There could well be thousands of copies of Justin Bieber's <i>Eenie Meenie</i>, all frenetically typed in by a horde of snapchatting teenagers using mom's Windows Vista laptop with the missing shift key.
<p>
I don't want my analysis to include the copies and covers of the same song. So I have two problems to solve:
<ol>
<li>How can we know whether two songs are actually the same?
<li>And how can we do this quickly, over the whole collection?
</ol>
<h3>But first</h3>
When dealing with text, or images, or sound files, or whatever kind of media tickles your pickle, we want to transform them into numbers that computers can use. We turn them into <i>feature vectors</i>, or what most Ph.D toting natural language processing experts call, when they really want to get technical for a stodgy old formal publication -- one that will put them on the tenure track -- when they want to choose the most technically precise phrase, they call them: "bags of words". I am not making this up.
<p>
Lets say we had this paragon of poesy:
<pre>
Eenie, meenie, miney, mo
Catch a bad chick by her toe
If she holla
If, if, if she holla, let her go
</pre>
<p>
A bag of words is set of words, and their counts. The ordering of the words is lost to simplify things. Order is rarely important anyway.
<p>
{a: 1, bad: 1, by: 1, catch: 1, chick: 1, eenie: 1, go: 1,her: 2, holla: 2, if: 4, let: 1, meenie: 1, miney: 1 mo:1, she: 2, toe: 1 }
<p>
We could go even simpler and remove the counts, if we feel they aren't important.
<p>
{a, bad, by, catch, chick, eenie, go, her, holla, if, let, meenie, miney, mo, she, toe}
<p>
As we process each document from the database, the first thing we do is turn it into the bag of words. In python it's a one-liner.
<pre>
def makeBag(song):
return set(song.replace(",", " ").split())
</pre>
<h3>Comparing two bags</h3>
Let's say we had three sets. One is the song above. In the other, the teenaged transcriber thought "miney" should be spelled "miny". The third is Frank Sinatra's </i>Fly me to the moon</i>. We would like a distance function, so that if two songs are differ by only one word, then the distance would be small, and if they are completely different, the distance is large.
<p>
To find the answer, we have to travel to 1907, and accompany Swiss professor Paul Jaccard on his trip to the Alps to do some serious botany. He noticed that there were different clusters of plants in different regions, and wondered if these clusters could be used to determine the ecological evolution of the area. He writes:
<blockquote>
In order to reply to this question I have considered, in an alpine district of fair size, various natural sub-divisions, presenting, besides numerous resemblances in their ecological conditions (i.e. conditions dependent on soil and climate), a few characteristic differences, and I have sought to determine, by comparison, the influence of these resemblances and differences on the composition of flora.
</blockquote>
<p align=center><img src="http://gs.riverdale.k12.or.us/~suzannal17/swiss/alpsmount.jpg"></p>
He counted all of the different plants in different regions, and came up with this formula to compare how similar two different regions are:
<p>
Number of species common to the two districts / total number of species in the two districts.
<p>
This gives 0 if the sets share no common elements, and 1 if they are the same. But that's the opposite of what we need, so we subtract it from one to obtain a distance function.
<pre>
def Jaccard(A, B):
intersection = len(A & B)
union = len(A | B)
return 1.0 - float(intersection)/union
</pre>
<p>
Now we have a distance function. To find all the duplicate songs, we just run it on every pair of songs that we have. With two million songs, that's only, umm, <i>four trillion comparisons</i>. If we can do 10000/second we could be done in about three years. Maybe we could split it up, use some cloud instances, pay a few thousand dollars for compute time and be done in a day.
<p>
Or we could use algorithms.
<h3>Time to get LSH'd</h3>
<p>
I have two little girls and coincidentally, they have approximately two million individual socks, with no two pairs alike. Yet it doesn't take me three years to sort them, because I use a locality sensitive hash.
<p align=center><img src="http://2.bp.blogspot.com/_ZptPwpCkJ5g/TTtGhTPa_cI/AAAAAAAAAsw/BcdxxDDvsDY/s1600/socks.jpg"></p>
<p>
I take a sock, and if it's pinkish, I put it in the top left. Purple goes in the top right, and colours in the middle go in between. Blues go on the bottom, greens have their own spot. By the time I run out of socks to sort, the pairs are of near each-other on the carpet. Then it's a simple matter to join them together.
<p>
Actually, over the years, I have further refined the system, because "pinkish" is ambiguous. Children's socks are a mix of shapes, eyes, cats & dinosaurs of all colours. If the sock as any blue at all, no matter how small, it goes top left. Otherwise, if it has any red, other colours notwithstanding, it goes bottom left. Otherwise, if it has any green whatsoever, top right. Otherwise, bottom left.
<p>
This is known as:
<h3>MinHash</h3>
<p> <img align=right width=120px src="http://4.bp.blogspot.com/_CNrtanQnRc0/SO7PQ9XIxQI/AAAAAAAAAIY/rbi802mOxmY/s320/lance-ito.jpg">
Now let's travel to 1997. <i>Titanic</i> and <i>The Full Monty</i> are in theaters. Some people pay to see the film 9 or 10 times. (<i>Titanic</i>, I mean) This is unsurprising because the only thing on TV is the OJ Simpson trial. On the WWW, then known as the <i>World Wide Wait</i>, AltaVista is one of the top search engines for finding the status of the <a href="http://en.wikipedia.org/wiki/Trojan_Room_coffee_pot">Trojan Room Coffee Pot</a>.
<p>
Computer Scientist Andrei Broder, who has been with AltaVista from near the beginning, is working on the <i>duplicates problem</i>. As the web was expanding, a lot of search results that come up are duplicates of other pages. For search, this is Very Annoying. Broder devises a way of quickly searching these millions of pages for duplicates.
<p>
MinHash is a function that reduces a text document to a single number. Documents that share many of the same words have numbers that are near each-other.
<p>
How is this done?
<p>
Suppose you build a dictionary of all the words that could possibly occur in your documents, and you number them.
<pre>
0 aardvark
1 abacus
2 abacuses
3 abaft
4 abalone
5 abandon
...
</pre>
The minhash would take this dictionary, and take your document, and assign it the number of the minimum word that occurs. That's it.
<p>
So if your document is "The aardvark abandoned his abacus" then the number assigned would be 0 (because aardvark is the zero'th word in the dictionary). In fact, every document that talks about an aardvark would hash to 0.
<p>
But what if, by chance, there is a document that is similar to our aardvark text but mispells it? Then they would hash to some other number entirely.
<p>
To guard against this, we actually take several random permutations of the dictionary and average the minhash against each of them.
<table>
<tr>
<td>
<pre>
0 abacus
1 abalone
2 abacuses
3 aardvark
4 abaft
5 abandoned
</pre>
</td>
<td>
<pre>
0 abalone
1 abacus
2 abacuses
3 abandoned
4 aardvark
5 abaft
</pre>
</td>
<td>
<pre>
0 abacus
1 abaft
2 abacuses
3 abalone
4 abandoned
5 aardvark
</pre>
</td>
</tr>
</table>
<ul>
<li>Document: "The aardvark abandoned his abacus"
<li>Minhash under first dictionary: 0
<li>Minhash under dictionary 2: 1
<li>Minhash under dictionary 3: 0
<li>Combined minhash: (0 + 1 + 0) / 3 = 0.333333333
</ul>
As you use more and more dictionaries to compute the hash, then documents that share similar sets of words begin to hash to similar values. If you like code, here's some python.
<p>
<pre>
import random
def MinHash(corpus, k = 5):
# Map from words to array of the five values
words = {}
for word in corpus:
words[word] = []
for i in range(k):
shuffled = list(corpus)
random.shuffle(shuffled)
for j in range(len(shuffled)):
words[shuffled[j]].append(j)
def hash(document):
total = 0.
# for each hash function, find the lowest value word in the
# document.
#sum(min(h_k(w) over words in doc)
vals = [-1] * k
for word in document:
if word in words:
m = words[word]
for i in range(k):
if vals[i] == -1 or m[i] < vals[i]:
vals[i] = m[i]
return sum(vals) / k
return hash
</pre>
<h3>Finally</h3>
Using MinHash, you can mark duplicates in three passes through the data, and a sort.
<ol>
<li>In the first pass, build a dictionary of all the words that occur, and use it to create the hash function.
<li>In the second pass, compute the minhash of each document.
<li>Sort the documents by their minhash (if you can afford to do so) or place them into buckets. In either case, documents that are similar will theoretically be close together.
<li>Finally, go through the list (if sorted) or nearby buckets, and compare documents within a certain window using a more refined comparison function, such as Jaccard distance. Anything that is close enough to being the same is a duplicate.
</ol>
<h3>Oh yeah</h3>
I will assume that the most poetic words of all time, in English, are the ones most likely to end a line. After analysis of 2 million song lyrics, with near duplicates removed, they are:
<pre>
at all
no more
don't know
all alone
right now
all night
at night
far away
like that
let go
too late
</pre>
And the number one most poetic phrase in the history of music:
<pre>
oh oh
</pre>
<ul><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=91'>You don't need a project/solution to use the VC++ debugger</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=57'>Coding tips they don't teach you in school</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a></ul>
Let's read a Truetype font file from scratch
tag:smhanovtechblog,2007:id143
2014-11-05T15:45:16-05:00
2014-11-05T15:45:16-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Drag a font file here to reveal its innermost secrets! <a href="https://github.com/creationix/font-awesome/blob/master/FontAwesome.ttf?raw=true">Here's one in case you don't have one handy.</a>
<div id=dropTarget style="border:3px dashed #5f90d0; font-size:30px;width:300px;height:100px;padding:20px;margin:0 auto; text-align: center">Drag TTF file here </div>
<div id=font-container></div>
<script src="TrueType.js" type="text/javascript"></script>
<script>
$("#dropTarget").on("drop", function(e) {
setTimeout(function() {layout.go(); }, 500);
});
</script>
<p>
<a href="https://gist.github.com/smhanov/f009a02c00eb27d99479a1e37c1b3354">Source code</a>
<p>
Here are the steps we will follow:
<ol>
<li>When the file is dragged onto the web page, we want to read it.
<li>We need to be able to interpret the numbers in the file, even though they were made for C programs to read.
<li>We have to find the number of characters in the file and the positions of the glyph outlines in the file
<li>We have to interpret the format of the glyph outlines
<li>Finally, we have to render them to the web page.
</ol>
<h2>Reading Files from Javascript</h2>
<p>
Whoah that sounds dangerous. But javascript can't read any file on your computer; just the ones you happen to drag over the web page, intentionally or accidentally. We do that by handling the dragover and drop events. When the drop event is received, it contains a reference to the file and then our code is allowed to read it. This is done without any interactions with the server.
<p>
We also have to handle the ondragover event and cancel it, because otherwise it won't work.
<pre>
var dropTarget = document.getElementById("dropTarget");
dropTarget.ondragover = function(e) {
e.preventDefault();
};
dropTarget.ondrop = function(e) {
e.preventDefault();
if (!e.dataTransfer || !e.dataTransfer.files) {
alert("Your browser didn't include any files in the drop event");
return;
}
var reader = new FileReader();
reader.readAsArrayBuffer(e.dataTransfer.files[0]);
reader.onload = function(e) {
ShowTtfFile(reader.result);
};
};
</pre>
<p>
You can't do much with the HTML5 File object. To get its data, you have to use the FileReader to read it asynchronously. You can choose to read it as a base64 encoded string or an array buffer. We choose an ArrayBuffer.
<p>
<h2>Interpreting the C structures</h2>
TrueType files were designed when computers had very little memory. They were designed to be mapped into RAM and read in place. C structures were even placed directly in the file. Opening a true type file was just a matter of loading it in. There was no need to do anything else. We will do a similar thing, but we will need a way to easily seek around the file and read numbers in various formats.
<p>
Here's a class that lets you do that.
<pre>
function BinaryReader(arrayBuffer)
{
assert(arrayBuffer instanceof ArrayBuffer);
this.pos = 0;
this.data = new Uint8Array(arrayBuffer);
}
BinaryReader.prototype = {
seek: function(pos) {
assert(pos >=0 && pos <= this.data.length);
var oldPos = this.pos;
this.pos = pos;
return oldPos;
},
tell: function() {
return this.pos;
},
getUint8: function() {
assert(this.pos < this.data.length);
return this.data[this.pos++];
},
getUint16: function() {
return ((this.getUint8() << 8) | this.getUint8()) >>> 0;
},
getUint32: function() {
return this.getInt32() >>> 0;
},
getInt16: function() {
var result = this.getUint16();
if (result & 0x8000) {
result -= (1 << 16);
}
return result;
},
getInt32: function() {
return ((this.getUint8() << 24) |
(this.getUint8() << 16) |
(this.getUint8() << 8) |
(this.getUint8() ));
},
getFword: function() {
return this.getInt16();
},
get2Dot14: function() {
return this.getInt16() / (1 << 14);
},
getFixed: function() {
return this.getInt32() / (1 << 16);
},
getString: function(length) {
var result = "";
for(var i = 0; i < length; i++) {
result += String.fromCharCode(this.getUint8());
}
return result;
},
getDate: function() {
var macTime = this.getUint32() * 0x100000000 + this.getUint32();
var utcTime = macTime * 1000 + Date.UTC(1904, 1, 1);
return new Date(utcTime);
}
};
</pre>
<h3>Fixed point numbers</h3>
Besides unsigned and signed 8, 16, and 32 bit numbers, there are some other types of things that appear in font files. The Fixed type is a way of representing decimals in a certain number of bits. Like fixed-point arithmetic, only we use binary instead of 10s. Suppose we wanted to write (in base 10) the number 1.53 but our decimal point key is broken. We would instead write 153. To convert it back, we divide by 100. Likewise, in binary, it works the same way, except that we divide by a power of two.
<h3>A note on Javascript numbers</h3>
Javascript has a wishy-washy "number" type. It is usually a 32-bit integer. It switches from signed to unsigned whenever it feels like it, and when you least expect it, it will switch to a 64-bit double precision number.
<p>
But you can force it to be signed using the "unsigned shift right" operator (>>>). By shifting it by 0, it converts the internal type to unsigned.
<h2>Finding the treasures</h2>
The TrueType font format is <a href="https://developer.apple.com/fonts/TrueType-Reference-Manual/">described by Apple here</a>. The truetype file is prefixed with something called the "offset" table that tells you where everything else is in the file. We will have to go diving into various tables to find the actual outlines of the fonts.
<p>
The tables also have a checksum to ensure they are right. This is obtained by adding up all the 4-byte integers in them, modulo 2<sup>32</sup>.
Here's the code to read the offsets.
<pre>
function TrueTypeFont(arrayBuffer)
{
this.file = new BinaryReader(arrayBuffer);
this.tables = this.readOffsetTables(this.file);
this.readHeadTable(this.file);
this.length = this.glyphCount();
}
TrueTypeFont.prototype = {
readOffsetTables: function(file) {
var tables = {};
this.scalarType = file.getUint32();
var numTables = file.getUint16();
this.searchRange = file.getUint16();
this.entrySelector = file.getUint16();
this.rangeShift = file.getUint16();
for( var i = 0 ; i < numTables; i++ ) {
var tag = file.getString(4);
tables[tag] = {
checksum: file.getUint32(),
offset: file.getUint32(),
length: file.getUint32()
};
if (tag !== 'head') {
assert(this.calculateTableChecksum(file, tables[tag].offset,
tables[tag].length) === tables[tag].checksum);
}
}
return tables;
},
calculateTableChecksum: function(file, offset, length)
{
var old = file.seek(offset);
var sum = 0;
var nlongs = ((length + 3) / 4) | 0;
while( nlongs-- ) {
sum = (sum + file.getUint32() & 0xffffffff) >>> 0;
}
file.seek(old);
return sum;
},
</pre>
Okay now we know where all the various tables are in the file. But one that we will need later is the "head" table, which contains the dimenions of the font, and importantly, the format of the glyph index.
<pre>
readHeadTable: function(file) {
assert("head" in this.tables);
file.seek(this.tables["head"].offset);
this.version = file.getFixed();
this.fontRevision = file.getFixed();
this.checksumAdjustment = file.getUint32();
this.magicNumber = file.getUint32();
assert(this.magicNumber === 0x5f0f3cf5);
this.flags = file.getUint16();
this.unitsPerEm = file.getUint16();
this.created = file.getDate();
this.modified = file.getDate();
this.xMin = file.getFword();
this.yMin = file.getFword();
this.xMax = file.getFword();
this.yMax = file.getFword();
this.macStyle = file.getUint16();
this.lowestRecPPEM = file.getUint16();
this.fontDirectionHint = file.getInt16();
this.indexToLocFormat = file.getInt16();
this.glyphDataFormat = file.getInt16();
},
</pre>
There are many tables to obtain the characteristics of the font, or the horizontal distance between glyphs, or the minimum recommended height, creation date, etc. But I want to stay focused on the buried treasure -- the glyph outlines.
<p>
The glyph outlines are contained in the "glyf" section. The glyphs are highly compressed and each one is a different length. To find a particular one quickly, we have to first go to the <a href="https://developer.apple.com/fonts/TTRefMan/RM06/Chap6loca.html">"loca" table.</a>
<p>
It is simply an array of 2 byte or four byte values, depending on the "indexToLocFormat" in the header. When this is set to one, the values are four bytes long and give the position of a glyph in the glyf table. Otherwise, they are two bytes long, and give the position of the glyph divided by two in the glyf table. File formats make confusing tradeoffs to be small.
<p>
<pre>
getGlyphOffset: function(index) {
assert("loca" in this.tables);
var table = this.tables["loca"];
var file = this.file;
var offset, old;
if (this.indexToLocFormat === 1) {
old = file.seek(table.offset + index * 4);
offset = file.getUint32();
} else {
old = file.seek(table.offset + index * 2);
offset = file.getUint16() * 2;
}
file.seek(old);
return offset + this.tables["glyf"].offset;
},
</pre>
<p>
Given any glyph index, we can now locate is exact position from the start of the file. Now things get a little complicated.
<p>
Conceptually the glyph can be one of two structures, which share a common header.
(diagram)
<p>
When two shapes are drawn on top of each-other, it is convention that the second will cut out the first one if it has a different winding order. That is, if the points are specified going clockwise instead of counter-clockwise and vice-versa. Fonts use this convention to build up shapes from contours.
For example, the letter O will have two contours -- one for the outer circle, and one for the inner one.
<p>
But there are two kinds of glyphs. The simple type is made of contours, as above. The compound type is made up of other glyphs. To draw the glyph, we have to draw each of the component glyphs and shift them around. This is made to handle characters with accents. Accented versions of the letters can therefore take very little space.
<p>
Let's keep focused on getting the treasure. We will ignore the compound glyphs. We just want to extract those sweet outlines.
<h2>Interpreting the outlines</h2>
<p>
This function will read the glyph header, and then call the right function to read it.
<pre>
readGlyph: function(index) {
var offset = this.getGlyphOffset(index);
var file = this.file;
if (offset >= this.tables["glyf"].offset + this.tables["glyf"].length)
{
return null;
}
assert(offset >= this.tables["glyf"].offset);
assert(offset < this.tables["glyf"].offset + this.tables["glyf"].length);
file.seek(offset);
var glyph = {
numberOfContours: file.getInt16(),
xMin: file.getFword(),
yMin: file.getFword(),
xMax: file.getFword(),
yMax: file.getFword()
};
assert(glyph.numberOfContours >= -1);
if (glyph.numberOfContours === -1) {
this.readCompoundGlyph(file, glyph);
} else {
this.readSimpleGlyph(file, glyph);
}
return glyph;
},
</pre>
The simple glyphs are stored in a compressed format. They can deal with repeated points, and small movements from one point to the next very well. This is done using a series of one-byte flags. Each flag-byte indicates whether the corresponding point is stored in one byte or two bytes, for each of the X and Y coordinates. After the flags come the X coordinates, and finally the Y coordinates. The great thing about this is that if either the X or the Y coordinate doesn't change, only one byte is used to indicate this in the flags.
<p>
When we read a glyph, we will assemble the points together into one array of (x, y) coordinates, plus one of the flags which is very important for rendering.
<pre>
readSimpleGlyph: function(file, glyph) {
var ON_CURVE = 1,
X_IS_BYTE = 2,
Y_IS_BYTE = 4,
REPEAT = 8,
X_DELTA = 16,
Y_DELTA = 32;
glyph.type = "simple";
glyph.contourEnds = [];
var points = glyph.points = [];
for( var i = 0; i < glyph.numberOfContours; i++ ) {
glyph.contourEnds.push(file.getUint16());
}
// skip over intructions
file.seek(file.getUint16() + file.tell());
if (glyph.numberOfContours === 0) {
return;
}
var numPoints = Math.max.apply(null, glyph.contourEnds) + 1;
var flags = [];
for( i = 0; i < numPoints; i++ ) {
var flag = file.getUint8();
flags.push(flag);
points.push({
onCurve: (flag & ON_CURVE) > 0
});
if ( flag & REPEAT ) {
var repeatCount = file.getUint8();
assert(repeatCount > 0);
i += repeatCount;
while( repeatCount-- ) {
flags.push(flag);
points.push({
onCurve: (flag & ON_CURVE) > 0
});
}
}
}
function readCoords(name, byteFlag, deltaFlag, min, max) {
var value = 0;
for( var i = 0; i < numPoints; i++ ) {
var flag = flags[i];
if ( flag & byteFlag ) {
if ( flag & deltaFlag ) {
value += file.getUint8();
} else {
value -= file.getUint8();
}
} else if ( ~flag & deltaFlag ) {
value += file.getInt16();
} else {
// value is unchanged.
}
points[i][name] = value;
}
}
readCoords("x", X_IS_BYTE, X_DELTA, glyph.xMin, glyph.xMax);
readCoords("y", Y_IS_BYTE, Y_DELTA, glyph.yMin, glyph.yMax);
}
</pre>
<h2>Drawing the glyphs in the web page</h2>
Finally we have something to show for all the effort. We want to draw the glyphs. HTML5 has its handy canvas API that will let us draw shapes.
<p>
Here's the function that controls the whole thing. It takes and array buffer
from the drag & drop event, and creates our TrueType object from it. Then it removes any previous glyphs from the screen. For each character, it creates an <canvas> element and scales the font so that it's EM height (literally, the height of the letter 'M') is about 64 pixels high. The font also has to be flipped vertically, because its coordinates assume zero is in the lower left of the screen, but our coordinates are in the top left.
<pre>
function ShowTtfFile(arrayBuffer)
{
var font = new TrueTypeFont(arrayBuffer);
var width = font.xMax - font.xMin;
var height = font.yMax - font.yMin;
var scale = 64 / font.unitsPerEm;
var container = document.getElementById("font-container");
while(container.firstChild) {
container.removeChild(container.firstChild);
}
for( var i = 0; i < font.length; i++ ) {
var canvas = document.createElement("canvas");
canvas.style.border = "1px solid gray";
canvas.width = width * scale;
canvas.height = height * scale;
var ctx = canvas.getContext("2d");
ctx.scale(scale, -scale);
ctx.translate(-font.xMin, -font.yMin - height);
ctx.fillStyle = "#000000";
ctx.beginPath();
if (font.drawGlyph(i, ctx)) {
ctx.fill();
container.appendChild(canvas);
}
}
}
</pre>
All that's left to show you is how they are drawn. In this function we ignore the curves and simply connect each point in the outline. However, in reality, some points are actually control points in a quadratic bezier curve.
<pre>
drawGlyph: function(index, ctx) {
var glyph = this.readGlyph(index);
if ( glyph === null || glyph.type !== "simple" ) {
return false;
}
var p = 0,
c = 0,
first = 1;
while (p < glyph.points.length) {
var point = glyph.points[p];
if ( first === 1 ) {
ctx.moveTo(point.x, point.y);
first = 0;
} else {
ctx.lineTo(point.x, point.y);
}
if ( p === glyph.contourEnds[c] ) {
c += 1;
first = 1;
}
p += 1;
}
return true;
}
</pre>
<h2>The code</h2>
The code that I am using these days is on <a href="https://gist.github.com/smhanov/f009a02c00eb27d99479a1e37c1b3354">GitHub here</a>. It has been enhanced to handle character kerning, and CMaps which map codepoints to glyph indicies.
<ul><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=119'>Throw away the keys: Easy, Minimal Perfect Hashing</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=91'>You don't need a project/solution to use the VC++ debugger</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a><li><a href='?id=65'>Drawing Graphs with Physics</a></ul>
A Quick Measure of Sortedness
tag:smhanovtechblog,2007:id145
2014-09-11T17:45:08-05:00
2014-09-11T17:45:08-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
How do you measure the "sortedness" of a list? There are several ways. In the literature this measure is called the "distance to monotonicity" or the "measure of disorder" depending on who you read. It is still an active area of research when items are presented to the algorithm one at a time. In this article, I consider the simpler case where you can look at all of the items at once.
<p>
The Kendall distance between two lists is the number of swaps it would take to turn one list into another. So, for [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] and [10, 1, 2, 3, 4, 5, 6, 7, 8, 9], it would take nine swaps.
<p>
Edit distance is another method. We could take the 10, and move it after the 9, in one operation. The edit distance is inversely related to the longest increasing subsequence. In the list [1, 2, 3, 5, 4, 6, 7, 9, 8], the longest increasing subsequence is [1, 2, 3, 5, 6, 7, 9], of length seven, and it is three away from being a sorted list. The longest increasing subsequence can be calculated in O(nlogn) time. A drawback of this method is its large granularity. For a list of ten elements, the measure can only take the distinct values 0 through 9.
<p>
Here, I propose another measure for sortedness. The procedure is to sum the difference between the position of each element in the sorted list, x, and where it ends up in the unsorted list, f(x). We divide by the square of the length of the list and multiply by two, because this gives us a nice number between 0 and 1. Subtracting from 1 makes it range from 0, for completely unsorted, to 1, for completely sorted.
<p align=center><img src="sortedness.gif"></p>
<p>
A simple genetic algorithm in python for sorting a list using the above fitness function is presented below.
<pre>
import random
def procreate(A):
A = A[:]
first = random.randint(0, len(A) - 1)
second = random.randint(0, len(A) - 1)
A[first], A[second] = A[second], A[first]
return A
def score(A):
diff = 0.
for index, element in enumerate(A):
diff += abs(index - element)
return 1.0 - diff / len(A) ** 2 * 2
def genetic(root, procreateFn, scoreFn, generations = 1000, children=6):
maxScore = 0.
for i in range(generations):
print("Generation {0}: {1} {2}".format(i, maxScore, root))
maxChild = None
for j in range(children):
child = procreate(root)
score = scoreFn(child)
print(" child score {0:.2f}: {1}".format(score, child))
if maxScore < score:
maxChild = child
maxScore = score
if maxChild:
root = maxChild
return root
A = [a for a in range(10)]
random.shuffle(A)
genetic(A, procreate, score)
</pre>
<p>
Note that under this metric, the completely reversed list does not have a score of 0.
<p>
The <a href="http://webuild.envato.com/blog/using-stats-to-not-break-search/">Spearman's coefficient</a>, mentioned in the comments, might be what you are looking for.<ul><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=98'>Asking users for steps to reproduce bugs, and other dumb ideas</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a></ul>
My thoughts on various programming languages
tag:smhanovtechblog,2007:id142
2014-07-06T20:36:53-05:00
2014-07-06T20:36:53-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center>
<img src="../comics/comic_20040604.png">
</p>
<p>
I hate all the languages. Once, I tried to make my own language, but I couldn't figure out what language to do it in, so I never started.
<p>
Most of the time, you don't have any choice of what language to work in. Whatever language I'm using, I've learned to appreciate both its strengths and weaknesses.
<h2>Java</h2>
People who like Java like typing. I mean: actually hitting keys on the keyboard. You have to keep repeating yourself over and over.
<p>
The whole java system was designed by an insane person whose answer to everything is to use a Design Pattern. If you see design patterns as a way of working around problems in the language, you will see that Java has many.
<p>
On the other hand, the folks at Sun really put the work in to make Java a specification that works on embedded platforms, so we're stuck with it there. I wouldn't really trust Python or C to run my desktop on my phone.
<p>
Also, what's with all those folders? I have to use Eclipse, against my will, because it knows how to jump around all those 1000 character path names. Would it really hurt anybody if I kept the 10 objects in my application in the same folder?
<h2>C</h2>
C is precise. When I write something in C, and it is done, I know that it will work. It's like painting a masterpiece with a single-hair brush. Having to code in that level of detail is a different mindset. When you sit down to write something in C, you have to plan it out before you start. Otherwise it's a lot of work to change it later.
<p>
If you have enough experience, memory leaks are rare. It's second nature -- malloc/free come in pairs. You can't forget one. It would be like forgetting to flush or turn off the lights. You just do it.
<p>
That being said, if you're going to paint a house, you don't want to be using a fine brush. You want huge rollers. If I'm writing a whole application, or a system, I would avoid C if I can.
<p>
It is difficult to make large changes to a C program. When I'm working on an algorithm, and I know that the first cut won't be right, often I will code in python first and then translate it into C by hand when it's done.
<h2>C++</h2>
It's C with a string class. And arrays and lists and heaps of queues to implement whatever you desire. A word for the wise: don't try to make your own templates. It's too hard. Aside from that, C++ makes C better, and you can write some very nice software in C++. The extra features make it scale up to larger systems with only moderate difficulty, as long as everyone follows the same conventions.
<h2>Javascript</h2>
This is the language that nobody loves. But javascript loves you. When you were first learning it, you might have written some very bad code that used an array as a dictionary, with other objects as keys, but it's totally okay, because that code is still running flawlessly and will continue to run as along as browsers run javascript.
<p>
Javascript lacks a linker, so all the code shares the same namespace, but everyone knows that, so everything still works together.
<h2>coffeescript</h2>
Coffeescript is a translator that takes a strange ruby-like language and turns it into Javascript, line by line. It is javascript with all extraneous syntax -- braces, brackets, extra keyswords removed. Only the essential meaning of the code remains.
<p>
Coffeescript is nice. When you have to write tonnes of code, coffeescript will make you at least 25% faster. You can see that many more lines on the screen at once.
<p>
When you code in coffeescript you have to be very aware of what Javascript is going to be generated. That's the problem. You have to know Javascript first. Anyone new coming to your project has to first learn Javascript, and only then learn coffeescript, and then learn your codebase.
<h2>node.js</h2>
I wanted to like it. I think I gave it a good go. It's the callbacks that got me. I just <i>know</i> that someday, for whatever reason, one of those callbacks isn't going to happen and then my app is going to be stalled waiting forever. That's no way to live.
<p>
Also, nearly nothing is built in. But if you have to do X, there are always a dozen modules to choose from that do the same thing. Which do you choose? Which will get support if you have problems?
<h2>Scala</h2>
Scala is a functional, typed language that compiles to JVM code.
<p>
I learned Scala on the job. Yup, a startup was actually using it for their production system, and I joined them fairly late.
<p>
This allowed me to see the ugly side of Scala: Type inference. Types are inferred <i>to the extreme.</i> Everything has a type, but figuring out what that type is means checking different files several levels back. And Scala inherits Java's folder insanity, so it means delving into several levels of folders to find the right file to lookup the type.
<p>
In short, Scala was great -- for the original developers. Newcomers had a long learning curve to learn the existing code.
<h2>Erlang</h2>
Erlang is also one that I wanted to love. I really tried. It is a beautiful functional language that lets you make wonderful little modules that communicate in precise ways, and your system can run for 10 years because it can deal with unexpected problems, restart what's needed, and keep going.
<p>
Unfortunately it's baroque. Development seems to have stopped at about the time that Berkeley invented sockets. Almost nothing needed in the modern era is included. Why is it so much work to make a simple web service?
<h2>Go</h2>
Go is easy to learn, even for newcomers. It uses language concepts from 40 years ago to build a robust, asynchronous system, and lets you code it as if it were synchronous. You can write 1000 threads that work together safely in Go without your brain hurting.
<p>
It still needs some work in library support. When I want to do X, which library should I use -- the one on github from 2011 or the one from 2013 that is half finished? One is linked to from the official pages, but it the official pages don't seem all that up to date. Sigh, I guess I'll have to write my own...
<h2>Python</h2>
There's a library for everything in python, and if you use Linux it's usually clear which is the top one, because it's installable with one command.
<p>
If you have to do some number crunching or scientific computing you will be well-served by choosing Python.
<p>
Strings can be both text and data in python, so you have to learn about text encodings early on.
<h3>Python 3</h2>
Python 3 shares many characteristics with python, though it is a different language. Since it's newer its not supported as much. I want to use it, but there's always that one library I need that only has python 2 support.<ul><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=146'>O(n) Delta Compression With a Suffix Array</a><li><a href='?id=131'>[comic] Appreciation of xkcd comics vs. technical ability</a><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a></ul>
A little VIM hacking
tag:smhanovtechblog,2007:id140
2014-07-06T03:26:18-05:00
2014-07-06T03:26:18-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center>
<img src="vim-hacking.png">
</center><ul><li><a href='?id=119'>Throw away the keys: Easy, Minimal Perfect Hashing</a><li><a href='?id=78'>It's a dirty job... (comic)</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=148'>How to detect if an object has been garbage collected in Javascript</a><li><a href='?id=19'>Installing the Latest Debian on an Ancient Laptop</a><li><a href='?id=90'>Regular Expression Matching can be Ugly and Slow</a></ul>
The strange man reading a novel in the meeting room
tag:smhanovtechblog,2007:id139
2014-07-06T03:07:33-05:00
2014-07-06T03:07:33-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
In late 2011, we were crammed in the table in the break room.
<p>
"I tried to book the meeting room but it's booked all week for the layoffs," said James.
<p>
Times were bleak at BlackBerry. Last Friday 11% of the workforce, over two thousand people, were laid off. On that day I had passed a red faced man being escorted out, flanked by a security goon and a serious looking woman from Organizational Development.
<p>
On the bright side, our team was -- mostly -- still there, and there were plenty of spare monitors lying around. I grabbed a couple and now I had three. There was a lot of room on my desk -- I had grown tired of bringing all my photos and desk toys back and forth every Thursday just in case I was fired. Now I just kept them at home, still packed in bag hanging off a bicycle hook in the garage.
<p>
"But there aren't any layoffs happening now, are there?" I mused. "I mean, maybe if you were on vacation or something last Friday, but they wouldn't need the meeting room all week."
<p>
Chris said, "I checked the room and there's just this guy there. He comes in every morning and just sits there reading a novel."
<p>
When our meeting was finished I stopped by the Marconi room. Inside, sat a bearded man along, reading a novel. His badge had the prominant red escort-required visitors stripe.I hurried off before he could see me.
<p>
Sure enough, each day that week as I walked by this room, he was there, just reading. And each day he would leave by 5.
<p>
A couple of weeks later we were finally able to book the room. As we flung our BlackBerry's and notebooks on the table, Chris was finally able to resolve the mystery.
<p>
"I found out who that guy was."
<p>
The mystery had been solved!
<p>
Chris continued. "He's a counsellor the company brought in to help us deal with the layoffs."
<p>
"How'd you find out?" I asked. I had been reading all of the email announcements and hadn't seen anything about this.
<p>
"I asked him."
<ul><li><a href='?id=73'>How to run a linux based home web server</a><li><a href='?id=58'>Email Etiquette</a><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=7'>Rules for Effective C++</a></ul>
You can cheat so your web site seems faster than it is
tag:smhanovtechblog,2007:id141
2013-12-13T05:47:55-05:00
2013-12-13T05:47:55-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
I couldn't sleep. Took some neo citron for my cold and rewrote the rhymebrain instant algorithm.
<p>
When you start typing "ox" into rhymebrain, there is almost 100% chance that you are going to type "oxygen". Likewise for "or" it is "orange". There is a perceived time savings if rhymebrain prefetches the results for what it thinks you are going to type while you are still typing. On the other hand, if you go ahead and enter "oxen" then it will actually take longer to get the results, since the requests are serialized by the browser. Prefetching can waste time.
<p>
Most people only type in one of 1000 words, which easily fit into some <script defer> javascript.
<p>
Previously, I was precalculating the completion which maximized the probability of the word, using some basterdized half-remembered version of bayes law. But I think there is a better approach, by running simulations on existing data.
<p>
I have lots of data of what people typed into the search box.
<p>
Google-mad-scientist-and-textbook-writer Peter Norvig once wrote a spellchecker in python that works efficiently by brute force. (http://norvig.com/spell-correct.html) I took some inspiration from that and wrote a python program that does, by brute force:
<p>
<pre>
for each possible prefix of all the words,
for each word with that prefix,
assume the word is completed.
calculate time saved if the word is completed * count
for each other word with that prefix,
subtract time wasted * count
</pre>
<p>
...where count is the number of times the word was entered into Rhymebrain's search box by people. This time of year, there is a large savings for completing "Christ" to "christmas" because people enter it so often.
<p>
Now we have a big list of the time savings (or waste) of prefetching each word from each prefix. Sort them by savings and put the top ones into a javascript file: http://rhymebrain.com/prefix.js
<p>
Now when you use rhymebrain and enter one of the 1800 words in the list, and type slowly enough, then the results will instantly pop onto the screen when you press enter.
<p>
Finally getting tired. Almost time to wake up and feed breakfast to kids<ul><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=96'>Bending over: How to sell your software to large companies</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=31'>Free, Raw Stock Data</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a></ul>
Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US
tag:smhanovtechblog,2007:id138
2013-11-13T04:45:08-05:00
2013-11-13T04:45:08-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<div class=idea> <h1>Warning</h1>For qualified advice, ask an organization such as the chamber of commerce. Here is some <a href="http://www.bdo.ca/en/Library/Services/Tax/Documents/Tax-Bulletins/Tax-Consequences-for-Canadians-Doing-Business-in-the-US.pdf">information from BDO for Canadians on selling to the USA.</a> If you can pay for advice, you should set up an appointment with an adviser.
</div>
<p>
After many months, your software sale is complete! You've got a purchase order, sent the invoice, delivered the software. You're already handling some support issues from users at BigCorp. Then BANG! Martha from Procurement emails back, as a favour, just to let you know that BigCorp has not received your W8 form with a valid tax id, and therefore will be withholding 30% of the purchase price of your multi-thousand dollar product for taxes, so that the crazy government in the crazy USA that NEITHER YOU NOR ANYBODY FROM YOUR COMPANY HAS EVER SET FOOT IN can buy more guns.
<p align=center><img src="https://lh6.googleusercontent.com/Q61xK2mufsT9Rig9b1nlXgpRzeEV0oO4ddSD3Ik5dZwMwgnOfyLOkUQwowj5ChJPrWx3Sfqwgxdp8CutuUg7G7fletwnXo4K6jh_fUQSjDnobBOTlk-7rxSu8Q"></p>
<p>
OK, stop and take a deep breath. Refrain from calling your lawyer yet. A quick Googling will tell you that:
<ol>
<li> You don't need a W8-form, Martha is wrong.
<li> Wait that first post was wrong. You absolutely need a W8-form but Martha is incorrect -- you can leave US Tax-ID blank.
<li>Actually the first two posters are wrong you need to get a tax ID and start paying US taxes, and also file paperwork each quarter to collect sales taxes separately in each state that you are selling in.
<li> What, really? That can't be right!
</ol>
<p>
Confident people on Internet forums are a wealth of misinformation about the W8-BEN form. The rules are spelled out for you on the form's instructions. The part that is confusing everybody is: what you can get away with. If your customer's procurement is being ultra strict, it is difficult to argue because they must follow the law and you are just a foreigner.
<p>
The W8-BEN form must be kept on file by US companies that make payments of any type to foreigners. The government never sees this form unless the company is audited, so the rules are loosely enforced. Most companies will accept these explanations when they ask for one:
<ul>
<li> My company has no US presence so I cannot provide this form.
<li> I do not have a US taxpayer ID so I cannot provide this form.
</ul>
<p>
But the <a href="http://www.irs.gov/pub/irs-pdf/iw8ben.pdf">rules that Martha is following</a> are quite clear, no matter how inconvenient it makes your life. You need to provide this form upon request, and it must have a valid US-taxpayer ID for you to claim benefits under a tax treaty, unless the payment is for dividends.
<p>
Because Martha's company is making payments to foreigners, they are a withholding agent, and their finance and legal department are being extra careful to follow the rules in <a href="http://www.irs.gov/Businesses/International-Businesses/U.S.-Withholding-Agent-Frequently-Asked-Question">this document for withholding agents.</a>
<p>
The good news it should only take you about 2 hours time to obtain the documentation to appease Martha. If Martha receives the W8-BEN form, and your country has a tax treaty with the US, then she will probably pay your invoice. Here are the steps.
<h2>Obtain a valid US Taxpayer ID</h2>
<p>
If you are a foreign corporation, the taxpayer identification number you need is called an Employer Identification Number (EIN). It doesn't matter if you have employees, or if you never pay any US taxes due to a treaty.
<ol>
<li><a href="http://www.irs.gov/uac/Form-SS-4,-Application-for-Employer-Identification-Number-(EIN)">Download the SS-4 form</a>, and print it out. Fill it in. Most of it is your company name and address, and date of incorporation, so it will be simple.
<li>Now you can mail or fax it in. If you need the number right away, you will have to call.
<li>Phone the US government using the toll number from the instructions of the SS-4 form. As a foreigner you cannot do it online.
<li>You will be told the wait time is 15 minutes, but you will then wait one or more hours on hold for your opportunity to speak with a tax official. Make sure you have your phone charger.
<li>The tax official will be tired and rude because she has to deal with indignant foreigners all day. Smile as you talk and be polite. Explain that you need an EIN to fill out the W8-BEN form. Do not offer any other information. The rest of the call will be you spelling out your company name and address from the SS-4 form letter by letter. <a href="http://www.osric.com/chris/phonetic.html">(See the NATO Phonetic Alphabet)</a>
<li>She will verbally give you your very own EIN over the phone.
<li>If you are told you need an ITIN then thank her, hang up and try again with a different agent.
</ol>
<h2>Find your applicable tax treaty</h2>
<p>
If you are lucky, then decades ago, your country negotiated a tax treaty with the United States. The instructions to witholding agents contains a table of tax treaties and section numbers that they will refer to. They are listed in <a href="http://www.irs.gov/pub/irs-pdf/p515.pdf">Publication 515</a>
The actual text of the treaties are <a href="http://www.irs.gov/Businesses/International-Businesses/United-States-Income-Tax-Treaties---A-to-Z">helpfully available here</a>. You will need the section number from the table of contents. For my Canadian Company's software sales, I used "Independent Personal Services" because for some reason the 1980 tax treaty failed to forsee the need for a section on "Cloud Platform for Creating UML Diagrams".
<h2>Fill out the W8-BEN form</h2>
<p>
You'd better have a stack of them, because larger companies are asking for original forms with "wet signature". But in the past it has been convenient to put a scanned copy on your web site, in the same place where you accept purchase orders.
<p>
Follow the instructions for the form, filling in your name, address, foreign tax id with your corporation's business number, and enter your new EIN on line 6. On line 9-10, fill in your country and put in the tax treaty section number (eg for Canada: XVII Independent Personal Services)
<h2>Good luck!</h2>
<p>
I hope that you are able to get your invoice paid. Always remember that the people you are dealing with are doing their job, and they have to follow the rules. If you are rude and do not follow instructions, then they will compain about you at the watercooler and possibly misplace your invoice. If you are polite and give them their paperwork, then you will get your money, and they will remember you and future sales with the same company will be a breeze.
<h2>Get tips on improving your software business, right in your inbox</h2>
Enter your email and I will send you tips on selling software right in your inbox. I have been selling software since 1998, and whether it's consulting products, adsense, or software as a service, I have done it all, and I want to tell you what I wish I knew when I started.
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/slim-081711.css" rel="stylesheet" type="text/css">
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; }
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="http://cmo.us7.list-manage.com/subscribe/post?u=594f70a64c065b48e3118ec8d&id=14ffac27b3" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button"></div>
</form>
</div>
<!--End mc_embed_signup--><ul><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=7'>Rules for Effective C++</a><li><a href='?id=82'>The PenIsland Problem: Text-to-speech for domain names</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=60'>Is 2009 the year of Linux malware?</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a></ul>
Asana's shocking pricing practices, and how you can get away with it too
tag:smhanovtechblog,2007:id137
2013-11-06T02:21:20-05:00
2013-11-06T02:21:20-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
If one apple costs $1, how much would five apples cost? How about 500?
<p>
In everyday life, when you buy more of something, you get more <a href="http://rhymebrain.com/en/?mode=alliterate&query=fruit%20dollar">bananas for your buck</a>. The fixed costs decrease. If you sell a lot of apples to one person, you don't have to wrap each one, you don't have to pay fixed transaction fees on each sale, and you don't have to worry about finding someone to buy the other 499 apples. The savings are passed on to the consumer. Often, software is priced this way too.
That's why I love <a href="https://asana.com/pricing">Asana's pricing page</a>. It breaks the rules.
<p align=center><img src="https://lh3.googleusercontent.com/OmteDgN2qMudJmggN7n_LlZFj6acBWj6VqNmtm1c5rsR5kJs7yIcKJrBYqw7nRIPku0FlUqC4kXTIv5uXseScnCp2c20xCv8vNVumMwEWkho_Ihkie3s7qW2EQ"></p>
<p>
Asana prices their product based on its value. It lets teams coordinate about projects and tasks they are working on.
Asana is very clear about the value they give. In fact, the pricing page tells you that the only difference between the paid and free versions is that "premium plans allow you to coordinate with more team members, as well as the features listed in the table above. <i>All other user features are exactly the same</i>."
<p>
It's a mathematical law that as the number of people in a team grows, the number of communication paths grows quadratically. A company with 100 people using it is therefore getting much more value out of it than a company of 15 people, so they pay higher per-seat costs.
<p>
<div class=idea>
<h1>Homework</h1>
What is the one thing that gives your software value? Are you directly charging for that thing, or something else? How can you take advantage of team effects to provide more value when more people use it?
</div>
<h2>Get tips on improving your software business, right in your inbox</h2>
Enter your email and I will send you tips on selling software right in your inbox. I have been selling software since 1998, and whether it's consulting products, adsense, or software as a service, I have done it all, and I want to tell you what I wish I knew when I started.
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/slim-081711.css" rel="stylesheet" type="text/css">
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; }
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="http://cmo.us7.list-manage.com/subscribe/post?u=594f70a64c065b48e3118ec8d&id=14ffac27b3" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button"></div>
</form>
</div>
<!--End mc_embed_signup--><ul><li><a href='?id=123'>Zero load time file formats</a><li><a href='?id=131'>[comic] Appreciation of xkcd comics vs. technical ability</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a><li><a href='?id=140'>A little VIM hacking</a></ul>
5 Ways PowToon Made Me Want to Buy Their Software
tag:smhanovtechblog,2007:id136
2013-10-31T21:28:19-05:00
2013-10-31T21:28:19-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>Powtoon is online software that lets you create animated powerpoint presentations, without the steep learning curve of Adobe Flash. The selling techniques they use are simple and powerful. Even though I saw through their tricks at every step along the way, I am now a customer and proud of it. It is worthwhile to look at what they did, because these are simple things that you can do to improve your software business.
<h2>Help users remember your app with email</h2>
As soon as I signed up with powtoon, the emails started. By the fifth day of the constant barrage I started marking them as spam. But for those five days, my inbox was a constant reminder that Powtoon was there and waiting for me. Since it was on my mind, I mentioned it to three other people in conversations I had. If the emails weren't there, I might be still struggling to remember the name of that cartoon site I signed up with, instead of being a paying user. What was that name again? Powertoon.io? Powrtoon.com?
<div class=idea>
<h1>Homework</h1>
If you offer software as a service, you should have an email campaign that offers helpful tips, as a way of reminding people of where they've been. Emailing every day is excessive, however.
</div>
<h2>Go read the <strike>user manual</strike> $23 ebook we're giving you FREE!</h2>
<p>
Sadly, for users coming from Google Docs or PowerPoint, Powtoon's first-use experience still confusing and it is enormously helpful to read the manual. Many people read the manual without realizing it because it is delivered in the form of an Ebook, with a value attached to it. "You could buy this book from Amazon for $23!" says the copy, "or download it here FREE".
<p>
When I get a free book I usually toss it in the recycling. But if it's worth money, I'll probably read it.
<p align=center><img src="https://lh6.googleusercontent.com/QBqo5kXDakn6Tc6YKyRMXigxy7gn9FpEnaoiX7uQgbsChVW_h6WOXzUSoBVhYKsEvlSuN8KGA9bFKaFa6TxnMkUZDCrEOn6H4ey-mRoyX7Ml_w343Iaq8Qavmg"></p>
<div class=idea>
<h1>Homework</h1>
Sometimes users need to read the manual to get the most out of your product. If reading the manual is a part of your sales funnel, and you should do everything you can to get people to read it. Package it as an E-book to get more people through this stage.
</div>
<h2>Price for value, not competition</h2>
<p>
Powtoon could have looked at Adobe Creative suite and priced their product cheaper. But Adobe is not the competition. Instead, the much larger, richer market of skilled powerpoint folks are looking at Powtoon, and comparing it with professional video design. From that perspective it looks like a bargain.
<p align=center><img src="https://lh4.googleusercontent.com/-BMBb-JXsxZ5qJxvwLCyXv2kD1o0mAGkHLp1kIXBOxxoOBMXSmReRCQx10K4iK_gxM8lEA4SZQqcRZ-FeSn3QxCsGuDkCz-SbU9r2qihrT3HXrM-47QUsyBJOg"></p>
<div class=idea>
<h1>Homework</h1>
Unless your goal is to be acquihired, it is difficult to build a business on $9/month. If your software has no plan that costs at least $100/month, try to think of some feature that would be a must-have for businesses. For big companies, cost doesn't matter, as long as your highest plan has something they need.
</div>
<h2>A clear reason to buy</h2>
<p>
It is easy to make software that hides the premium features away. They are grayed out and pushed to the end of the list. I've done it myself. For users, its like living in a cosy room. If you ignore the locked door behind the couch, you can forget that it is the foyer of a mansion.
<p>
At every step, Powtoon reminds you that you are missing the majority of its capabilities. While choosing a background tune, I had to scroll through hundreds of songs that I could play but couldn't use. Only about five songs were available, but they were strewn through the list. Every other item of clipart is only available in the highest-cost plan, but again, you have to scroll through them. You can make a great video without paying anything. You are free to use any song or image that you can upload. But you leave with the deep sense that the software is crippled.
<div class=idea>
<h1>Homework</h1>
Do users on your free plan say that they don't see any reason to buy? If the premium features are hidden away, it's time to make them more visible.
</div>
<h2>Price segmentation</h2>
<p>
The powtoon Agency plan serves two purposes. Firstly, it lets users who can pay more do so. Some users don't care how much something costs don't want to have to think about it. These users have an option to pay more. But I am not one of them.
<p>
The Agency plan is listed first on the pricing page, and psychologically anchors the value of the rest of the plans. When I first used Powtoon I was stuck by the high cost. I couldn't justify it, so I resolved to use the free plan. But a few days later I saw this:
<p align=center><img src="https://lh4.googleusercontent.com/Ml83SHmgy1sc8ZdRjaei1o8OubwYZvlmqpNYgvoPw9kEa0f2nFZcYUYTF-B9zAu6lY18ml9D32HtlveRNOyq6mRae8rpgBXoDB-iYHEuQwtUmGuoKpzJj-33WA"></p>
<p>
A few days after I signed in, Powtoon sent me this offer. It made me rethink things in a hurry. With this sale, and the "savings" of over $400 buy-now-before-it's-too-late plastered on my monitor every time I sign in, they successfully reframed the middle tier as a bargain.
<div class=idea>
<h1>Homework</h1>
In your pricing page, what techniques do you use to get people from the free tier into the middle tier plan? How can you use the high-cost tier to reframe the value of the other plans?
</div>
<h2>So what did I end up making?</h2>
I whipped together a video for my consulting product, <a href="http://zwibbler.com/buy">Zwibbler</a>. Zwibbler is a drop-in solution that <a href="http://zwibbler.com/buy">lets users draw in your web app.</a>
<h2>Get tips on improving your software business, right in your inbox</h2>
Enter your email and I will send you tips on selling software right in your inbox. I have been selling software since 1998, and whether it's consulting products, adsense, or software as a service, I have done it all, and I want to tell you what I wish I knew when I started.
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/slim-081711.css" rel="stylesheet" type="text/css">
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; }
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="http://cmo.us7.list-manage.com/subscribe/post?u=594f70a64c065b48e3118ec8d&id=14ffac27b3" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button"></div>
</form>
</div>
<!--End mc_embed_signup-->
<ul><li><a href='?id=97'>Creating portable binaries on Linux</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=3'>Cell Phone Secrets</a><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=134'>0, 1, Many, a Zillion</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a></ul>
How I run my business selling software to Americans
tag:smhanovtechblog,2007:id135
2013-04-30T14:09:51-05:00
2013-04-30T14:09:51-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
I first realized I had overpaid when I received my articles of incorporation from the law firm. Was it because they were in a leather bound binder? Was it because it had been shipped overnight from Toronto to Waterloo, a distance of 83 km, such a distance that I could have driven there and picked it up and then returned and paid less than the cost of shipping it? Instead I had to wait two business days for the Fedex truck to drive back to Cambridge since I wasn't home, and then I had to call in to arrange to pick it up at a "conveniently located" Fedex office a week later.
<p>No, I was miffed because I had paid $1500 to incorporate, when I could have done over the web for far less.
Still, it is a very nice binder.
<p>
Since that time I have slowly been finding ways to optimize my business, which consists of selling software to Americans. I sell it all kinds of ways.
<p>
A typical month:
<table border=1>
<tr><td>
WebSequenceDiagrams subscriptions</td><td> $1600</td></tr>
<tr><td>
WebSequencediagrams Server Sales</td><td> $1600</td></tr>
<tr><td>Rhymebrain.com Adsense Revenue</td><td> $1700</td></tr>
<tr><td>Zwibbler.com licensing and consulting</td><td> $2000</td>
<table>
<p>
<span style="font-size:smaller;color:#888">*My <a href="https://plus.google.com/115821829711457452449/posts/Fvp82M27bxu">annual reports</a> are on Google Plus, where nobody reads them.</span>
<p>
According to the Canada Revenue Agency, I'm a profitable small business. The only thing preventing me from spending it all on iPads, Google Glasses and Surface Tablets is the fact that I have to feed my lovely family.
<p align=center><img src="http://alicesmommyblog.com/wp-content/uploads/2013/03/IMG_3180-e1364781736131-200x300.jpg">
<br><span style="font-size:smaller;color:#808080">I keep costs down by feeding them chocolate flavoured <a href="http://www.soylent.me/">soylent</a> in a bucket</span>
</p>
<p>
Here's some tips on running a business in Canada selling software to Americans.
<h2>Incorporation</h2>
<p>
Incorporation is a good choice. While it gives a valuable sense of security (albeit false) against lawsuits, the most useful benefit is income deferral. I started this company while I was working full time. If it were a sole proprietorship, I would have had to pay the top tax rate of 40% on everything I earned. This would have been a huge disincentive to growing my business.
<div class=idea><h1>Evil Tip for starting a company while working somewhere else</h1>
Whenever someone asks if it's legal, point them to the corporate policy and claim that there's a simple form that you fill out, and loudly complain that it takes the legal department eight months to answer any emails. With luck, the person that asked will launch into his own stories about the slow legal department, thus deflecting the conversation to a more useful topic.
</div>
<p>
With all of the profits inside a corporation, I had to pay only the 16% corporate tax on them. But I can keep the profits there until I feel like withdrawing them. It's like having an extra RRSP.
<p>
However, incorporation does have some added responsibilities. First, I have to pay Intuit TurboTax $200 every year to file my taxes. And that software only does about 10% of the work -- I have to maintain a balance sheet and income statement for the year so I can get the numbers to enter into TurboTax. Still, I figure we are about even, because Intuit also bought the server edition of WebsequenceDiagrams.
<div class=idea><h1>Evil Tip for doing your own taxes</h1>
It is easy to make a lot of mistakes the first time. But hiring an accountant costs $2000, while penalties from the government for making a mistake are maybe about $50 tops. I'll re-evaluate this when I'm making sufficiently more profit.
</div>
<p>
After you incorporate, you can't do very much until you get:
<h2>A business bank account</h2>
<p>
Canada has a cartel of five major banks. Stay away from them. I was explaining banking to my 3 year old daughter <a href="https://twitter.com/ToddlerLillian">(her twitter account)</a>:
<div style="font-style:italic">
<p>
Me: Banks are a place where you keep your money.
<p>
Lillian: WHY?
<p>
Me: Because they give you interest... (thinking) but then they take it away and charge you more money.
<p>
Lillian: WHY?
<p>
Me: I guess you put your money in a bank to keep it safe, and every month they take some away.
<p>
Lillian: WHY?
<p>
Me: I don't know. If you keep your money in the bank they will slowly take it away from you.
<p>
Lillian: I WILL KEEP MY MONIES BESIDE MY POTTY.
<p>
Me: Good. Now it's time to watch Dora. Daddy's got to go buy some bitcoin.
</div>
<p>
Instead, I use a <a href="http://www.yncu.com/">local credit union</a>, which has a pay-as-you go account. For $5 a month I can keep all of my profits there and write cheques. They wanted to sell me a business cheque book. What is it with all these leather-bound things? Does my business have to have everything wrapped in cow skin to appear successful? I imagine it might be useful in a narrow range of situations:
<div style="font-style:italic">
<p>
Me in line at the grocery store: Will you take a cheque for these Ruffles<sup>TM</sup> brand potato chips?
<p>
Attractive cashier: Um, noooo. What do you think this is, like 1985? Don't you have a Paypass chip?
<p>
Me: What about... from THIS chequebook? (whips out the corinthian leather-bound Execu-Check 5000 with dual-signature, day-planner, and matching gold pens.)
<p>
Attractive cashier: Oooh, no problem, Mr. Hanov. What are you doing later?
<p>
My wife: He'll be sleeping in the basement. Let's go.
</div>
I managed to get them to give me personal cheques with my business name written on them by asking very nicely. Credit unions are nice that way.
<p>
Unfortunately you have to deal with big banks sometimes. I needed to get:
<h2>A credit card</h2>
<p>
After a lot of research, I selected the Bank of Montreal credit card for businesses, because there is no fee and every December I get some cash back for using it. I filled out the application with my personal information, and since I was working at the time, there was no problem getting it.
<div class=idea><h1>Evil tip for paying for things</h1>
Currency exchange is expensive. As a rule, I pay for US things with US dollars, and Canadian things with Canadian dollars. This was only a problem with Microsoft Office 365, which insisted on charging my Paypal account in CAD. I had to tell Microsoft that I live in Beverly Hills to use my USD Paypal account. Because that's the only zip code I know.
</div>
I've only talked about the Canadian side so far. But many Canadian software companies get all their revenue in US dollars. There is an important trick for dealing with this, which I will get to shortly. But first:
<h2>Paypal</h2>
<p>
Paypal is utterly horrible to use and develop for. For example, to cancel a subscription for a user from last August, I have to page through them all, 25 at a time, waiting 5-10 seconds for each page load, until I get to August. If I didn't know when the subscription started, then I would be there for much longer reading all of the names. For developers, Paypal offers a special sandbox area which hasn't worked in months, and special paypal IPN notifications which are <a href=" http://stackoverflow.com/questions/16069000/paypal-ipn-history-page-bug-resend-messages-not-working">broken for several months</a> out of the year.
<p>
But once it's finally working, Paypal works everywhere. It lets me enter in tax rates for all of the Canadian provinces and territories (Why aren't they there already?). It lets me accept orders from Israel, France, the UK, Germany, Australia, New Zealand, Finland, Norway. When companies that I sell to refuse to use Paypal, it lets me pay $35 to enter the credit card details manually. The fee is outrageous, but it's a steal compared to anywhere else I can use. (Update: Use <a href="http://stripe.com">stripe.com</a>.
<p>
I use Paypal for most of my sales, but I have a small nagging fear that one day, the US Government, (which to us Canadians appears quite insane, so they would do this kind of thing) will take all of my money one day to fight terror by arming day cares. But when I transfer money out of Paypal into Canada, they use over-the-counter exchange rates. <b>If you are using Paypal for currency-conversion you must stop immediately, because there is a better way, which I will explain to you shortly.</b>
But first, to get the money out of Paypal without currency conversion, you need:
<h2>A US dollar bank account</h2>
<p>
My credit union couldn't offer all the services I needed to run my global company. I searched around and I found a reasonable deal with the Royal Bank of Canada USD Checking account. It is regularly $9/month, but with a minimum balance of $2500 the price drops to $2.
<p>
I need the USD bank account so I can accept incoming bank transfers with no currency losses. Outside of North America, it is common to pay for large items by exchanging bank account information, so the buyer can transfer the cash directly into the seller's accounts. North American banks discourage this behaviour by levying huge fees. When I invoice a customer, I include the bank details and in a few weeks I receive the full amount, minus the $15 fee for RBC, and $25 for some mysterious "intermediary bank". Still, a flat $40 charge looks pretty good on amounts greater than $1000 when compared to Paypal's fees for the same.
<div class=idea><h1>Evil tip for well-connected, wealthy European financiers</h1>
An intermediary bank is a good business to get into. Also, drugs.
</div>
<p>
So I have a Canadian Dollar account in a credit union, US dollars in Paypal, and US dollars in RBC. How do I get my money into good old Canadian loonies and toonies? That's where my favourite part comes in.
<h2>XE.com</h2>
<p>
If you are a Canadian company whose revenue comes in USD, you should immediately get an account at a currency broker. Once set up, it is a simple matter to transfer money between a US and Canadian bank account, at crazy-low conversion fees.
<p>
For example, I just went to Paypal and XE and priced out transferring $5000 USD into Canada. Today, the difference is not huge, but the spread has been much higher in the past.
<table>
<tr><td>Paypal</td><td>$4,929.33 CAD</td></tr>
<tr><td>XE.com</td><td>$4,980.00 CAD</td></tr>
</table>
<p>
I do not want 1 to 3% of my revenue disappearing off the top, so I use XE. When I signed up, I registered my US account and the Canadian one with them by copying the numbers off some cheques, and now I can initiate a transfer in seconds.
<h2>Charging tax</h2>
I added this section due to the comments. As a Canadian, if you make more than $30,000 in revenue in a year or recently (See the CRA web site for specific rules) then you have to register for GST/HST. You may register before that time though, and it may be advantageous, because <i>registered businesses don't pay GST / HST</i>. Keep track of all the stuff you buy for your business, tell the CRA the total each January, and you will get it all back in a nice fat cheque. This works out well when you, for instance, buy a $5000 Macbook pro.
<p>
Registering for GST/HST, however, requires you to do certain things.
<ul>
<li>Fill out the GST/HST return each year. It is a simple two-page form. Just report your total sales, the tax you collected, and the tax you paid. That's it!
<li>When you invoice a Canadian customer, then include your HST registration number and the tax charged in the invoice. This varies by province, so make sure you <a href="https://en.wikipedia.org/wiki/Sales_taxes_in_Canada">look it up on Wikipedia</a> before you invoice those freeloading Albertans.
</ul>
You aren't obligated to collect taxes for the governments of other countries. Just invoice the subtotal, and they can work out their own damn taxes. This does lead to a ridiculous HST return, where you've collected $100k in revenue and only $100 in HST. That's OK... software is a global product, and there are hardly any Canadians in this global market.
<p>
Do you charge in US dollars? That's OK too. It may look weird, but you can invoice a Canadian in USD. I do this because all my accounts are set up in USD. Also include HST in USD on the amount, as if it were in CAD. For reporting purposes, I convert the total based on the yearly average USD conversion rate on the CRA web site, but you are free to use other methods.
<p>
In January you will have to pay, or receive, the difference in taxes you collected and taxes you paid for stuff.
<h2>How do you do it?</h2>
Do you have a different way for optimizing your cash flow? Did I miss anything? Please share your tips in the comments.<ul><li><a href='?id=19'>Installing the Latest Debian on an Ancient Laptop</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a><li><a href='?id=128'>Why you should go to the Business of Software Conference Next Year</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a></ul>
0, 1, Many, a Zillion
tag:smhanovtechblog,2007:id134
2013-04-05T17:17:13-05:00
2013-04-05T17:17:13-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<div style="text-align: center;font-size:24pt;font-family:Georgia,times,serif;">
<p>
There are only four numbers in computer programs:
<p>
0, 1, many, "a zillion"
<p>
If you have 2 or more of anything, you are, in general, better off using loops to process <i>many</i> of them.
<p>
But what is "a zillion?"
<p>
Zillion is a made-up number. Your system cannot hold a zillion items in memory. It cannot show a zillion items on the screen.
<hr>
<p>
Doesn't work for "a zillion":
<h2>Select employee name:</h2>
<br>
<select size=4 multiple>
<option>AADLUND, F</option>
<option>AADLAND, G</option>
<option>AADLUND, J</option>
<option>AADLUND, M</option>
<option>AAMOLD, D</option>
<option>AARDE, J. R</option>
<option>AARON, C.</option>
<option>AARON, E</option>
<option>AARON, M</option>
<option>AARON, R</option>
<option>AARON, T</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
<option>1</option>
</select>
<p>
<hr>
Doesn't work for "a zillion":
<pre style="text-align: left;">def handleFiles( filenames: Array[String] ) {
val results = openFiles(filenames).readAll().processAll()
results
}
</pre>
<div style="font-size:10px">
* The program first opens all the files, and then processes them. The OS will run out of file handles.
</div>
<hr>
<p>
Doesn't work for "a zillion":
<p><img src="chrome-browser.png">
<p>
Changing software from handling "many" to "a zillion" is hard if the program is already written.
<p>
Decide when you need to handle a zillion.
</div>
<ul><li><a href='?id=86'>Boring Date (comic)</a><li><a href='?id=123'>Zero load time file formats</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=120'>Succinct Data Structures: Cramming 80,000 words into a Javascript file.</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=26'>A simple command line calculator</a><li><a href='?id=109'>Cross-domain communication the HTML5 way</a></ul>
Give your Commodore 64 new life with an SD card reader
tag:smhanovtechblog,2007:id133
2012-08-01T21:22:38-05:00
2012-08-01T21:22:38-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
This August marks the 30th anniversary of the most successful computer model in history. One company put personal computers into the people's homes, and launched an entire industry overnight. For an entire decade, despite attempts at marketing improvements, the original platform stood the test of time, virtually unchanged. Even today, the Commodore 64 is celebrated by a community of hobbyists.
<p>
In honour of the anniversary month, here are some up to date instructions on how to read old data from the disks. Floppy disks only have a usable lifespan of 10 years, and most of them in my collection are over 25 already. At this point, the chemical binder between the <a href="http://www.clir.org/pubs/reports/pub54/2what_wrong.html">magnetic particles and the plastic substrate is degrading</a>, and the data can literally fall out of the disks. Fortunately, up to date software from the <a href="http://www.trikaliotis.net/opencbm">opencbm project</a> tries very hard to read the data and can often succeed when a disk appears to be unusable when connected to a C64.
<p>
Here's what you need to read your floppy disk into your PC:
<p>
<ol>
<li> A commodore 1541 disk drive, in good condition, and power supply.
<li> The drive cable.
<li> This <a href="http://store.go4retro.com/zoomfloppy/">ZoomFloppy</a> adapter
<li> a USB cable.
</ol>
<p>
In addition, if you want to read disk images from an SD card hooked into the real Commodore 64, you will need the <a href="http://store.go4retro.com/products/uIEC_SD.html">uIEC adapter</a>, which can be substituted for a real floppy drive.
<h2> Transferring floppy disks to the PC</h2>
<p>I want to scan in my copy of <a href="http://archive.kontek.net/jlounge.classicgaming.gamespy.com/">Jumpman</a> so its digital bits are preserved.
<p align=center><img src=c64-4.jpg><br><i>This copy is totally genuine EPYX product. At least that's what they guy at the swap meet told me, behind the K-mart.</i></p>
You have to download the special build of the opencbm software from <a href="http://www.root.org/~nate/c64/xum1541/">here</a>. I am using the windows version. Read the <a href="http://www.root.org/~nate/c64/ZoomFloppy-Manual-2.0.pdf">manual</a> very carefully to install the driver, before you can use the command line tools.
<p>
Then, plug the ZoomFloppy into the Commodore drive using the commodore cable (make sure the drive is off!), and into your PC using a USB cable.
<p align=center><img src=c64-1.jpg><br><i>Confusingly there's some empty ports. If you want, you can hook other things into it for decoration, I guess.</i></p>
Now we turn on the drive, put in the disk and cross our fingers. From a command line, I type:
<pre>
d64copy -r 16 8 "jumpman.d64"
</pre>
<p>
This means:
<ul>
<li>Retry bad blocks up to 16 times before giving up.
<li>Copy from device #8. Remember, on Commodore, the first floppy drive was always device 8.
<li>Copy the contents of the disk to the file called jumpman.d64
</ul>
<p>A minute later, and it is done. Unfortunately, we have a bad sector. Hopefully it is in an unused part of the disk, or in some graphics that won't cause a crash.
<p align=center><img src=c64-5.jpg></p>
<p>At this point, we can run the game in the <a href="http://www.viceteam.org/#download">Vice emulator</a> (On windows <a href="http://www.zimmers.net/anonftp/pub/cbm/crossplatform/emulators/VICE/WinVICE-2.2-x86.zip">this distribution</a> seems to work).
<p align=center><img src=c64-9.png></p>
<h2>Running games from SD cards</h2>
No serious commodore enthusiast is without his or her SD card reader. I happen to have the <a href="http://store.go4retro.com/products/uIEC_SD.html">uIEC</a> which <a href="http://www.go4retro.com/products/zoomfloppy/">Jim Brain</a> soldered together right in front of me at the <a href="http://www.tpug.ca/index.php?option=com_content&view=article&id=75:world-of-commodore-2011&catid=39:world-of-commodore&Itemid=58">World of Commodore 2011</a> in Toronto.
<p align=center><img src="c64-6.jpg"></p>
<p>
It has to plug into the back of the Commodore, presumably for power, and then the drive cable goes into it as well. Then, it is ready to act exactly like a floppy drive.
<p align=center><img src="c64-7.jpg"></p>
<h2>But how do folders work?</h2>
<p>
You can just about fit <b>all the Commodore software in existence</b> onto one card, so how do you access it? You can switch disk images by sending special drive commands to it in BASIC. I've placed the jumpman.d64 file on the SD card. Here is the command to switch to it:
<pre>
OPEN1,8,15:PRINT#1,"CD:JUMPMAN.D64":CLOSE1
</pre>
<p align=center><img src=c64-8.jpg></p>
There are many other commands for the uIEC. It has some buttons on the circuit board that let you swap disks on the fly without typing commands, but these have to be set up by listing them in a special file. All the other commands are <a href="http://sd2iec.de/cgi-bin/gitweb.cgi?p=sd2iec.git;a=blob;f=README;hb=HEAD">described here.</a>
<p>
A major drawback of the uIEC is that while it emulates the standard Commodore drive operation, the commodore 1541 drive was actually another computer that you could load programs on and run. Some programs did this for copy protection or to implement custom fast-loaders. More recently the retro demo-scene uses it for extra storage and computing power. I was disappointed when the <a href="http://www.youtube.com/watch?v=zVPW40ygds4">Second Reality</a> demo didn't work.
<p>
Well, I have Jumpman loaded anyway, so off to an evening of retro fun.
<p align=center><img src="c64-2.jpg"></a>
<h2>The cheater method</h2>
Of course, if you don't want to go to all this trouble, you could obtain just about any software you can think of using pokefinder. Just Bing it.<ul><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=39'>Stock Picking using Python</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a><li><a href='?id=14'>Experiments in making money online</a><li><a href='?id=91'>You don't need a project/solution to use the VC++ debugger</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a></ul>
20 lines of code that will beat A/B testing every time
tag:smhanovtechblog,2007:id132
2012-05-28T21:10:15-05:00
2012-05-28T21:10:15-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<div style="padding:10px;border:2px solid black;background:#ccc;text-decoration:none;display:block" href="http://zwibbler.com">
<a href='http://zwibbler.com'>Zwibbler.com</a> is a drop-in solution that lets users draw on your web site.</div>
<!-- <p style="font-size:small"><a href="http://clipartmag.com/ru-20-lines-a-b-testing">Russian Translation</a></p> -->
<p>
A/B testing is used far too often, for something that performs so badly. It is defective by design: Segment users into two groups. Show the A group the old, tried and true stuff. Show the B group the new whiz-bang design with the bigger buttons and slightly different copy. After a while, take a look at the stats and figure out which group presses the button more often. Sounds good, right? The problem is staring you in the face. It is the same dilemma faced by researchers administering drug studies. During drug trials, you can only give half the patients the life saving treatment. The others get sugar water. If the treatment works, group B lost out. This sacrifice is made to get good data. But it doesn't have to be this way.
<p>
In recent years, hundreds of the brightest minds of modern civilization have been hard at work not curing cancer. Instead, they have been refining techniques for getting you and me to click on banner ads. It has been working. Both <a href="http://research.google.com/search.html#q=bandit">Google</a> and <a href="http://research.microsoft.com/en-us/projects/bandits/">Microsoft</a> are focusing on using more information about visitors to predict what to show them. Strangely, anything better than A/B testing is absent from mainstream tools, including Google Analytics, and Google Website optimizer. I hope to change that by raising awareness about better techniques.
<p>
With a simple 20-line change to how A/B testing works, <b>that you can implement today</b>, you can <i>always</i> do better than A/B testing -- sometimes, two or three times better. This method has several good points:
<ul>
<li>It can reasonably handle more than two options at once.. Eg, A, B, C, D, E, F, G, �
<li>New options can be added or removed at any time.
</ul>
But the most enticing part is that <b>you can set it and forget it</b>. <a href="http://blog.asmartbear.com/value-time.html">If your time is really worth $1000/hour</a>, you really don't have time to go back and check how every change you made is doing and pick options. You don't have time to write rambling blog entries about how you got your site redesigned and changed this and that and it worked or it didn't work. Let the algorithm do its job. This 20 lines of code automatically finds the best choice quickly, and then uses it until it stops being the best choice.
<h2>The Multi-armed bandit problem</h2>
<div float=left align=center>
<img src="http://research.microsoft.com/en-us/projects/bandits/MAB-2.jpg"><br>
Picture from <a href="http://research.microsoft.com/en-us/projects/bandits/">Microsoft Research</a>
</div>
<p>
The multi-armed bandit problem takes its terminology from a casino. You are faced with a wall of slot machines, each with its own lever. You suspect that some slot machines pay out more frequently than others. How can you learn which machine is the best, and get the most coins in the fewest trials?
<p>
Like many techniques in machine learning, the <a href="http://www.cs.mcgill.ca/~vkules/bandits.pdf">simplest strategy</a> is <a href="http://www.cs.nyu.edu/~mohri/pub/bandit.pdf">hard to beat</a>. More complicated techniques are worth considering, but they may eke out only a few hundredths of a percentage point of performance. One strategy that has been shown to perform well time after time in practical problems is the <i>epsilon-greedy</i> method. We always keep track of the number of pulls of the lever and the amount of rewards we have received from that lever. 10% of the time, we choose a lever at random. The other 90% of the time, we choose the lever that has the highest expectation of rewards.
<p>
<pre>
def choose():
if math.random() < 0.1:
# exploration!
# choose a random lever 10% of the time.
else:
# exploitation!
# for each lever,
# calculate the expectation of reward.
# This is the number of trials of the lever divided by the total reward
# given by that lever.
# choose the lever with the greatest expectation of reward.
# increment the number of times the chosen lever has been played.
# store test data in redis, choice in session key, etc..
def reward(choice, amount):
# add the reward to the total for the given lever.
</pre>
<h2>Why does this work?</h2>
<p>
Let's say we are choosing a colour for the "Buy now!" button. The choices are orange, green, or white. We initialize all three choices to 1 win out of 1 try. It doesn't really matter what we initialize them too, because the algorithm will adapt. So when we start out, the internal test data looks like this.
<table border=1>
<tr><th>Orange</th><th>Green</th><th>White</th></tr>
<tr><td>1/1 = 100%</td><td>1/1=100%</td><td>1/1=100%</td></tr>
</table>
<p>
Then a web site visitor comes along and we have to show them a button. We choose the first one with the highest expectation of winning. The algorithm thinks they all work 100% of the time, so it chooses the first one: orange. But, alas, the visitor doesn't click on the button.
<table border=1>
<tr><th>Orange</th><th>Green</th><th>White</th></tr>
<tr><td>1/2 = 50%</td><td>1/1=100%</td><td>1/1=100%</td></tr>
</table>
<p>
Another visitor comes along. We definitely won't show them orange, since we think it only has a 50% chance of working. So we choose Green. They don't click. The same thing happens for several more visitors, and we end up cycling through the choices. In the process, we refine our estimate of the click through rate for each option downwards.
<table border=1>
<tr><th>Orange</th><th>Green</th><th>White</th></tr>
<tr><td>1/4 = 25%</td><td>1/4=25%</td><td>1/4=25%</td></tr>
</table>
<p>
But suddenly, someone clicks on the orange button! Quickly, the browser makes an Ajax call to our reward function <code>$.ajax(url:"/reward?testname=buy-button");</code> and our code updates the results:
<table border=1>
<tr><th>Orange</th><th>Green</th><th>White</th></tr>
<tr><td>2/5 = 40%</td><td>1/4=25%</td><td>1/4=25%</td></tr>
</table>
<p>
When our intrepid web developer sees this, he scratches his head. What the F*? The orange button is the <i>worst</i> choice. Its font is tiny! The green button is obviously the better one. All is lost! The greedy algorithm will always choose it forever now!
<p>
But wait, let's see what happens if Orange is really the suboptimal choice. Since the algorithm now believes it is the best, it will always be shown. That is, until it stops working well. Then the other choices start to look better.
<table border=1>
<tr><th>Orange</th><th>Green</th><th>White</th></tr>
<tr><td>2/9 = 22%</td><td>1/4=25%</td><td>1/4=25%</td></tr>
</table>
<p>
After many more visits, the best choice, if there is one, will have been found, and will be shown 90% of the time. Here are some results based on an actual web site that I have been working on. We also have an estimate of the click through rate for each choice.
<table border=1>
<tr><th>Orange</th><th>Green</th><th>White</th></tr>
<tr><td>114/4071 = 2.8%</td><td>205/6385=3.2%</td><td>59/2264=2.6%</td></tr>
</table>
<h3>Edit: What about the randomization?</h3>
I have not discussed the randomization part. The randomization of 10% of trials forces the algorithm to explore the options. It is a trade-off between trying new things in hopes of something better, and sticking with what it knows will work. There are several variations of the epsilon-greedy strategy. In the epsilon-first strategy, you can explore 100% of the time in the beginning and once you have a good sample, switch to pure-greedy. Alternatively, you can have it decrease the amount of exploration as time passes. The epsilon-greedy strategy that I have described is a good balance between simplicity and performance. Learning about the other algorithms, such as UCB, Boltzmann Exploration, and methods that take context into account, is fascinating, but optional if you just want something that works.
<h2>Wait a minute, why isn't everybody doing this?</h2>
Statistics are hard for most people to understand. People distrust things that they do not understand, and they especially distrust machine learning algorithms, even if they are simple. Mainstream tools don't support this, because then you'd have to educate people about it, and about statistics, and that is hard. Some common objections might be:
<ul>
<li>Showing the different options at different rates will skew the results. (No it won't. You always have an estimate of the click through rate for each choice)
<li>This won't adapt to change. (Your visitors probably don't change. But if you really want to, in the reward function, multiply the old reward value by a forgetting factor)
<li>This won't handle changing several things at once that depend on each-other. (Agreed. Neither will A/B testing.)
<li>I won't know what the click is worth for 30 days so how can I reward it?
</ul>
<h2>More blog entries</h2><ul><li><a href='?id=7'>Rules for Effective C++</a><li><a href='?id=120'>Succinct Data Structures: Cramming 80,000 words into a Javascript file.</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=138'>Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=85'>barcamp (comic)</a></ul>
[comic] Appreciation of xkcd comics vs. technical ability
tag:smhanovtechblog,2007:id131
2012-02-14T08:00:00-05:00
2012-02-14T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=86">Previous Comic</a> | Next Comic </p>
<p align=center>
<img src="http://zwibbler.com/shared/897.png">
</p><ul><li><a href='?id=75'>Pitching to VCs (comic)</a><li><a href='?id=66'>Test Driven Development without Tears</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=58'>Email Etiquette</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=3'>Cell Phone Secrets</a></ul>
VP trees: A data structure for finding stuff fast
tag:smhanovtechblog,2007:id130
2011-12-02T08:00:00-05:00
2011-12-02T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Let's say you have millions of pictures of faces tagged with names. Given a new photo, how do you find the name of person that the photo most resembles?
<p>
<img align=right src="http://stevehanov.ca/wavelet/cup.png" width="150px"> Suppose you have scanned short sections of millions of songs, and for each five second period you have a rough list of the frequencies and beat patterns contained in them. Given a new audio snippet, can you find the song to which it belongs?
<p>
What if you have data from thousands of web site users, including usage frequency, when they signed up, what actions they took, etc. Given a new user's actions, can you find other users like them and predict whether they will upgrade or stop using your product?
<p>
In the cases I mentioned, each record has hundreds or thousands of elements: the pixels in a photo, or patterns in a sound snippet, or web usage data. These records can be regarded as points in high dimensional space. When you look at a points in space, they tend to form clusters, and you can infer a lot by looking at ones nearby.
<p>
In this blog entry, I will half-heartedly describe some data structures for spatial search. Then I will launch into a detailed explanation of VP-Trees (Vantage Point Trees), which are simple, fast, and can easily handle low or high dimensional data.
<h2>Data structures for spatial search</h2>
<p>
When a programmer wants to search for points in space, perhaps the the first data structure that springs to mind is the K-D tree. In this structure, we repeatedly subdivide all of the points along a particular dimension to form a tree structure.
<p align=center><img src="http://upload.wikimedia.org/wikipedia/commons/thumb/b/bf/Kdtree_2d.svg/300px-Kdtree_2d.svg.png"></p>
<p>
With high dimensional data, the benefits of the K-D tree are soon lost. As the number of dimensions increase, the points tend to scatter and it becomes difficult to pick a good splitting dimension. Hundreds of students have gotten their masters degree by coding up K-D trees and comparing them with an alphabet soup of other trees. (In particular, I like <a href="http://www.google.ca/url?sa=t&rct=j&q=ashraf%20masood%20kibriya%20fast%20algorithms%20for%20nearest%20neighbor%20search&source=web&cd=1&ved=0CCUQFjAA&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.148.4652%26rep%3Drep1%26type%3Dpdf&ei=rdXYTonIFanh0QH_9eHfDQ&usg=AFQjCNG93LpA1wldfrSx9RoK2RLEAc3DRA&sig2=Q3-zbUzdiZZgei10A-jRHA&cad=rja">this one.</a>)
<p>
The authors of <a href="http://books.google.ca/books?id=5FIEAwyn9aoC&lpg=PA136&dq=ball%20tree&pg=PA136#v=onepage&q=ball%20tree&f=false">Data Mining: Practical machine Learning Tools and Techniques</a> suggests using <a href="http://www.icsi.berkeley.edu/ftp/pub/techreports/1989/tr-89-063.pdf">Ball Trees</a>. Each node of a Ball tree describes a bounding sphere, using a centre and a radius. To make the search efficient, the nodes should use the minimal sphere that completely contains all of its children, and overlaps the least with other sibling spheres in the tree.
<p align=center><img src="balltree.png"></p>
<p>Ball trees work, but they are difficult to construct. It is hard to figure out the optimal placement of spheres to minimize the overlap. For high dimensional data, the structure can be huge. The nodes must store their centre, and if a point has thousands of coordinates, it occupies a lot of storage. Moreover, you need to be able to calculate these fake sphere centres from the other points. What, exactly, does it mean to calculate a point between two sets of users' web usage history?
<p>
Fortunately, there are methods of building tree structures which do not require manipulation of the individual coordinates. The things that you put in them do not need to resemble points. You only need a way to figure out how far apart they are.
<h2>Entering metric space</h2>
<p>
Image you are blindfolded and placed in a gymnasium filled with other blindfolded people. Even worse: you also lost all sense of direction. When others talk, you can sense how far away they are, but not where they are in the room. Eventually, some basic laws become clear.
<ol><li>If there is no distance between you and the other person, you are standing in the same spot.
<li>When you talk to another person, they perceive you has being the same distance away as you perceive them.
<li>When you talk to person A and person B, the distance to A is always less than the distance to B plus the distance from A to B. In other words, the shortest distance between two people is a straight line. Distance is never negative.
</ol>
<p>
This is a metric space. The great thing about metric spaces is that the things that you put in them do not need to do a lot. All you need is a way of calculating the distances between them. You do not need to be able to add them together or find bounding shapes or find points midway between them. The data structure that I want to talk about is the <a href="http://pnylab.com/pny/papers/vptree/vptree/">Vantage Point Tree</a> (a generalization of the BK-tree that is eloquently reviewed in <a href="http://blog.notdot.net/2007/4/Damn-Cool-Algorithms-Part-1-BK-Trees">Damn cool algorithms</a>.
<p>
Each node of the tree contains one of the input points, and a radius. Under the left child are all points which are closer to the node's point than the radius. The other child contains all of the points which are farther away. The tree requires no other knowledge about the items in it. All you need is a distance function that satisfies the properties of a metric space.
<h2>How searching a VP-Tree works</h2>
<p>
Let us examine one of these nodes in detail, and what happens during a recursive search for the nearest neighbours to a target.
<p align=center><img src="http://zwibbler.com/shared/1795.png"></a>
<p>
Suppose we want to find the two nearest neighbours to the target, marked with the red X. Since we have no points yet, the node's center <i>p</i> is the closest candidate, and we add it to the list of results. (It might be bumped out later). At the same time, we update our variable <i>tau</i> which tracks the distance of the <i>farthest</i> point that we have in our results.
<p>
Then, we have to decide whether to search the left or right child first. We may end up having to search them both, but we would like to avoid that most of the time.
<p align=center><img src="http://zwibbler.com/shared/1796.png"></p>
<p>
Since the target is closer to the node's center than its outer shell, we search the left child first, which contains all of the points closer than the radius. We find the blue point. Since it is farther away than <i>tau</i> we update the tau value.
<p align=center><img src="http://zwibbler.com/shared/1797.png"></p>
<p>Do we need to continue the search? We know that we have considered all the points that are within the distance <i>radius</i> of <i>p</i>. However, it is closer to get to the outer shell than the farthest point that we have found. Therefore there <i>could be</i> closer points just outside of the shell. We do need to descend into the right child to find the green point.
<p>
If, however, we had reached our goal of collecting the <i>n</i> nearest points, and the target point is farther from the the outer shell than the farthest point that we have collected, then we could have stopped looking. This results in significant savings.
<h2>Implementation</h2>
Here is an implementation of the VP Tree in C++. The recursive <code>search()</code> function decides whether to follow the left, right, or both children. To efficiently maintain the list of results, we use a priority queue. (See my article, <a href="http://stevehanov.ca/blog/index.php?id=122">Finding the top k items in a list efficiently</a> for why).
<p>
I tried it out on <a href="cities.txt.gz">a database of all the cities</a> in the world, and the VP tree search was 3978 times faster than a linear search through all the points. You can download the C++ program that uses the VP tree for this purpose <a href="cities.cpp">here.</a>
<p>
It is worth repeating that <b>you must use a distance metric that satisfies the <a href="http://en.wikipedia.org/wiki/Triangle_inequality">triangle inequality</a></b>. I spent a lot of time wondering why my VP tree was not working. It turns out that I had not bothered to find the square root in the distance calculation. This step is important to satisfy the requirements of a metric space, because if the straight line distance to <i>a <= b+c</i>, it does not necessarily follow that <i>a<sup>2</sup> <= b<sup>2</sup> + c<sup>2</sup></i>.
<p>
Here is the output of the program when you search for cities by latitude and longitude.
<pre>
Create took 15484122
Search took 36
ca,waterloo,Waterloo,08,43.4666667,-80.5333333
0.0141501
ca,kitchener,Kitchener,08,43.45,-80.5
0.025264
ca,bridgeport,Bridgeport,08,43.4833333,-80.4833333
0.0396333
ca,elmira,Elmira,08,43.6,-80.55
0.137071
ca,baden,Baden,08,43.4,-80.6666667
0.161756
ca,floradale,Floradale,08,43.6166667,-80.5833333
0.163351
ca,preston,Preston,08,43.4,-80.35
0.181762
ca,ayr,Ayr,08,43.2833333,-80.45
0.195739
---
Linear search took 143212
ca,waterloo,Waterloo,08,43.4666667,-80.5333333
0.0141501
ca,kitchener,Kitchener,08,43.45,-80.5
0.025264
ca,bridgeport,Bridgeport,08,43.4833333,-80.4833333
0.0396333
ca,elmira,Elmira,08,43.6,-80.55
0.137071
ca,baden,Baden,08,43.4,-80.6666667
0.161756
ca,floradale,Floradale,08,43.6166667,-80.5833333
0.163351
ca,preston,Preston,08,43.4,-80.35
0.181762
ca,ayr,Ayr,08,43.2833333,-80.45
0.195739
</pre>
<h3>Construction</h3>
I'm too lazy to implement a delete or insert function. It is most efficient to simply build the tree by repeatedly partitioning the data. We build the tree from the top down from an array of items. For each node, we first choose a point at random, and then partition the list into two sets: The left children contain the points farther away than the median, and the right contains the points that are closer than the median. Then we recursively repeat this until we have run out of points.
<pre>
// A VP-Tree implementation, by Steve Hanov. (steve.hanov@gmail.com)
// Released to the Public Domain
// Based on "Data Structures and Algorithms for Nearest Neighbor Search" by Peter N. Yianilos
#include <stdlib.h>
#include <algorithm>
#include <vector>
#include <stdio.h>
#include <queue>
#include <limits>
template<typename T, double (*distance)( const T&, const T& )>
class VpTree
{
public:
VpTree() : _root(0) {}
~VpTree() {
delete _root;
}
void create( const std::vector<T>& items ) {
delete _root;
_items = items;
_root = buildFromPoints(0, items.size());
}
void search( const T& target, int k, std::vector<T>* results,
std::vector<double>* distances)
{
std::priority_queue<HeapItem> heap;
_tau = std::numeric_limits<double>::max();
search( _root, target, k, heap );
results->clear(); distances->clear();
while( !heap.empty() ) {
results->push_back( _items[heap.top().index] );
distances->push_back( heap.top().dist );
heap.pop();
}
std::reverse( results->begin(), results->end() );
std::reverse( distances->begin(), distances->end() );
}
private:
std::vector<T> _items;
double _tau;
struct Node
{
int index;
double threshold;
Node* left;
Node* right;
Node() :
index(0), threshold(0.), left(0), right(0) {}
~Node() {
delete left;
delete right;
}
}* _root;
struct HeapItem {
HeapItem( int index, double dist) :
index(index), dist(dist) {}
int index;
double dist;
bool operator<( const HeapItem& o ) const {
return dist < o.dist;
}
};
struct DistanceComparator
{
const T& item;
DistanceComparator( const T& item ) : item(item) {}
bool operator()(const T& a, const T& b) {
return distance( item, a ) < distance( item, b );
}
};
Node* buildFromPoints( int lower, int upper )
{
if ( upper == lower ) {
return NULL;
}
Node* node = new Node();
node->index = lower;
if ( upper - lower > 1 ) {
// choose an arbitrary point and move it to the start
int i = (int)((double)rand() / RAND_MAX * (upper - lower - 1) ) + lower;
std::swap( _items[lower], _items[i] );
int median = ( upper + lower ) / 2;
// partitian around the median distance
std::nth_element(
_items.begin() + lower + 1,
_items.begin() + median,
_items.begin() + upper,
DistanceComparator( _items[lower] ));
// what was the median?
node->threshold = distance( _items[lower], _items[median] );
node->index = lower;
node->left = buildFromPoints( lower + 1, median );
node->right = buildFromPoints( median, upper );
}
return node;
}
void search( Node* node, const T& target, int k,
std::priority_queue<HeapItem>& heap )
{
if ( node == NULL ) return;
double dist = distance( _items[node->index], target );
//printf("dist=%g tau=%gn", dist, _tau );
if ( dist < _tau ) {
if ( heap.size() == k ) heap.pop();
heap.push( HeapItem(node->index, dist) );
if ( heap.size() == k ) _tau = heap.top().dist;
}
if ( node->left == NULL && node->right == NULL ) {
return;
}
if ( dist < node->threshold ) {
if ( dist - _tau <= node->threshold ) {
search( node->left, target, k, heap );
}
if ( dist + _tau >= node->threshold ) {
search( node->right, target, k, heap );
}
} else {
if ( dist + _tau >= node->threshold ) {
search( node->right, target, k, heap );
}
if ( dist - _tau <= node->threshold ) {
search( node->left, target, k, heap );
}
}
}
};
</pre>
<ul><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a><li><a href='?id=39'>Stock Picking using Python</a><li><a href='?id=76'>Does Android team with eccentric geeks? (comic)</a><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=96'>Bending over: How to sell your software to large companies</a><li><a href='?id=117'>Why don't web browsers do this?</a><li><a href='?id=22'>Exploring sound with Wavelets</a></ul>
Why you should go to the Business of Software Conference Next Year
tag:smhanovtechblog,2007:id128
2011-10-29T00:19:36-05:00
2011-10-29T00:19:36-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Most people, having already paid $2000.00 of their hard earned money, and then having flown, driven, or otherwise travelled to Boston to attend a conference, and then having paid an additional $250/night plus $33/night parking and "tourism taxes" to the Seaport Hotel -- most people, after all this, are unlikely to say that it was a waste of time and they should have stayed home watching the remaining salvaged episodes of <i>Doctor Who</i> on Netflix.
<p>
In fact, I found it quite useful.
<p>
The talks by Clayton Christenson (author of <u>The Innovators Dilemma</u>), Rory Sutherland (expert on Behavioural economics) and the dozens of entrepreneurs (both serial and parallel) were all very fascinating and useful, and they were all broadcast <i>for free</i>, and they will soon be up for streaming, <i>for free</i>.
<p>
So why go through all of this effort to physically go to the conference?
<p align=center>
<img src=whiskey_priest.jpg><br><b>One of the conference rooms at Business of Software 2011.</b>
</p>
<p>
What the the World Trade Center in Boston lacks in number of bathrooms, it more than makes up for in hallways. It has roughly 1000 miles of hallways in which you can bump into successful business people. And every one of them is trying to meet <i>you</i> and get your take on important, urgent business-related matters like, "Have you seen an empty bathroom?"
<p>Seriously, when not at the conference, and people ask what I do, I have learned to say something like "I do computers". People here understand when I talk about NoSQL databases, SaaS models, and programmer development tools. The amount of time until their eyes glaze over is well over the 60 second mark.
<p>You also get some inside info. People aren't shy talking about their pricing. How much does the super-mega-ultra corporate option cost? The one where instead of a price, it says "Call"? These people will tell you, because they don't get to talk about it much, and they are honestly trying to help.
<p>
I talked to C.E.O.s, and C.T.Os, of 3 to 30 person companies. I talked to VPs, Cloud Engineers, and Intrapreneurs of big companies. For many, this is the first opportunity to talk to an outsider about their businesses. It is like psychotherapy. Often they would come to a sudden realization. "Hey," a micro-ISV would say, "I just have a <i>fear</i> of releasing the next version because it's missing some difficult features. I should just do it anyway!". If you go to this conference, you probably already know what you <i>should</i> do to improve your business. But having Jason Cohen, or some seasoned CEO <i>tell you in person</i> moves it up onto the todo list.
<h3>General trends</h3>
<ul>
<li><b>Disruption</b> - Disruption is big. If you're not disruptive, you might as well be selling mainframes and typewriters. Companies are disrupting each-other at an astounding rate. Sometimes, while one company is busy disrupting an industry, another one will sneak up behind it and try to disrupt it when it is not looking. That is why companies need to be agile and pivot frequently.
<li><b>Metrics</b> - The info-geeks have taken over. Founders are demanding dashboards for their business, updated in <i>real time.</i> But not only for themselves -- every click of the web site, and every cancellation is streamed to <i>every employee</i> to give an accurate picture of the health of the company. A special version containing only the "Customer Happiness Index" and a huge happy face is streamed to the investors.
<li><b>Crowd-sourced employee recognition</b> - At least three companies are working on this. It can be hard for bosses to identify their best contributors to allocate bonuses. The idea is to crowd-source this from their workforce. "So we'll give them a button -- so whenever anybody does something nice, other people will just push it and they get a -- a <i>pony</i> point --- yeah! And then I just have to add them all up to find the best contributors!" If you've worked at a large company for more than a year, you already know what an awesome idea this is. Just rename "pony" to "stab" and invert the score.
<li><b>Skype</b> - Ask anybody, in tiny or large companies. Odds are that they bypass their Enterprise Collabosoft GrouperWare system and secretly use Skype to communicate. Just a minute while I go privately Skype to people about why Microsoft should acquire my startup.
<li><b>Dishonesty</b> - Jason Cohen gave a talk about <a href="http://blog.asmartbear.com/differentiate-yourself-through-honesty.html">how honesty in business</a> can differentiate you. If you are a small company, he says, you should not try to hide it. Companies will be refreshed by your truthfulness, and it sets the correct expectations at the outset. Most of the attendees believe honesty is a great idea. Companies should all be honest! Because Jason Cohen says it pays! But if you are in a uniquely special business, such as storing data securely in the cloud, or selling software as a service, or selling licensed software, or you offer a limited or very diverse product line, or you have competition -- in these very special situations, honesty definitely will never work. At least, that's the going opinion.
</ul>
<p>
I hope you're convinced of the value that Business of Software has to offer, and I hope to see you there next year. I should be finished Doctor Who by then.<ul><li><a href='?id=52'>Automatically remove wordiness from your writing</a><li><a href='?id=139'>The strange man reading a novel in the meeting room</a><li><a href='?id=37'>Spoke.com scam</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=77'>Pitching to VCs #2 (comic)</a></ul>
Four ways of handling asynchronous operations in node.js
tag:smhanovtechblog,2007:id127
2011-09-30T21:30:00-05:00
2011-09-30T21:30:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Javascript was not designed to do asynchronous operations easily. If it were,
then writing asynchronous code would be as easy as writing blocking code.
Instead, developers in node.js need to manage many levels of callbacks.
<p>
Today, we will examine four different methods of performing the same task
asynchronously, in node.js. We will read in all the files in the current
folder,
<ol>
<li>In parallel, using callback functions
<li>Sequentially, using callback functions
<li>In parallel, using promises
<li>Sequentially, using promises
</ol>
<p>
This will help you decide which to use for your particular situation. It is simply a matter of taste. If you want to know which is the best method -- the absolute best way to go -- is probably to switch to <a href="http://golang.org/">Golang</a>.
<h2>Reading the files in parallel using callbacks</h2>
<p>
Our toy problem is to read all of the files in the current folder.
<p>
With our toy problem, we can't do everything in parallel. To read the files in
the folder, we first need to know which files are in the folder. Thus we start
with the readdir() function. We wait for the operation to complete. Then, for
each file, we use the readFile() to get the contents of the file.
<p>
Here are the documentation for the functions, from node.js.
<blockquote>
<p>
<b>fs.readdir(path, [callback])</b>
<p>
Asynchronous readdir(3). Reads the contents of a directory. The callback gets two arguments (err, files) where files is an array of the names of the files in the directory excluding '.' and '..'.
</blockquote>
<blockquote>
<p><b>fs.readFile(filename, [encoding], [callback])</b>
<p>
Asynchronously reads the entire contents of a file.
<p>
The callback is passed two arguments (err, data), where data is the contents of the file.
<p>
If no encoding is specified, then the raw buffer is returned.
</blockquote>
<p>
All of the readFile() operations happen at once, and then we wait for the
results to come in. We simply count how many times a readFile() operation has completed. When all of the files have been read, we
know we are done.
<pre>
// Read all files in the folder in parallel.
var fs = require("fs");
fs.readdir( ".", function( err, files) {
if ( err ) {
console.log("Error reading files: ", err);
} else {
// keep track of how many we have to go.
var remaining = files.length;
var totalBytes = 0;
if ( remaining == 0 ) {
console.log("Done reading files. totalBytes: " +
totalBytes);
}
// for each file,
for ( var i = 0; i < files.length; i++ ) {
// read its contents.
fs.readFile( files[i], function( error, data ) {
if ( error ) {
console.log("Error: ", error);
} else {
totalBytes += data.length
console.log("Successfully read a file.");
}
remaining -= 1;
if ( remaining == 0 ) {
console.log("Done reading files. totalBytes: " +
totalBytes);
}
});
}
}
});
</pre>
<h2>Reading the files sequentially using callbacks</h2>
<p>
It is usually most efficient to to the above. Order the computer to do everything at once, and let the operating system sort it out. But that's not always what you want. Sometimes you need to impose an order and do things sequentially.
<p>
Here is an example of reading each file one at a time. The for loop is gone. It is replaced by a recursive function. The function checks to see if it has reached the last file. If so, it is done. Otherwise, it calls itself to process the next file in the list.
<pre>
// Read all the files in the folder in sequence, using callbacks
var fs = require("fs");
fs.readdir( ".", function( error, files ) {
if ( error ) {
console.log("Error listing file contents.");
} else {
var totalBytes = 0;
// This function repeatedly calls itself until the files are all read.
var readFiles = function(index) {
if ( index == files.length ) {
// we are done.
console.log( "Done reading files. totalBytes = " +
totalBytes );
} else {
fs.readFile( files[index], function( error, data ) {
if ( error ) {
console.log( "Error reading file. ", error );
} else {
totalBytes += data.length;
readFiles(index + 1);
}
});
}
};
readFiles(0);
}
});
</pre>
<h2>Reading the files in parallel using Promises</h2>
<p>
A <i>promise</i> (also known a <i>future</i>, and sometimes a <i><a href="http://www.google.com/codesearch#R_N7SDoEwWc/assets/www/scripts/phonegap.0.9.5.js&q=phonegap%20channel&type=cs">channel</a></i>) is a concept <a href="http://en.wikipedia.org/wiki/Promise_(programming)">from the 1970's</a> that has recently become popular for Javascript programming. Promises are implemented in the node.js module <a href="https://github.com/kriszyp/node-promise">promise</a>. This module is not included unless you add it.
<p>
When you call a function and expect a return value, and the value is not yet available, the function instead returns a <i>promise</i>. The caller can then store the promise for later or schedule a subsequent operation when it completes. Promises can be seen as a more specialized form of node.js's <i>EventEmitter</i>, where the only two events are <i>reject</i> or <i>resolve</i>. Instead of using "on" to listen for the events, we use "then".
<p>
Here is a really simple example of promises being used.
<pre>
var promise = doSomeAsynchronousOperation();
promise.then( function(result) {
// yay! I got the result.
}, function(error) {
// The promise was rejected with this error.
}
function doSomeAsynchronousOperation()
{
var promise = new Promise.Promise();
fs.readFile( "somefile.txt", function( error, data ) {
if ( error ) {
promise.reject( error );
} else {
promise.resolve( data );
}
});
return promise;
}
</pre>
<p>
Promises may be easier to deal with for some people, because functions that return promises are harder to misuse. You could forget whether a callback belongs in the 3rd or 4th parameter, but you can't make that mistake with a return value. Another advantage is that they encapulate the recursive function loop above. You can easily construct a super-promise from an array of promises, what will resolve only when each of its members resolve. That's we do in the code below, with <i>Promise.all()</i>
<p>
<pre>
var fs = require("fs");
var Promise = require("promise");
// Wrap the io functions with ones that return promises.
var readdir_promise = Promise.convertNodeAsyncFunction(fs.readdir);
var readFile_promise = Promise.convertNodeAsyncFunction( fs.readFile );
p = readdir_promise( "." );
p.then( function( files ) {
// Create an array of promises
var promises = [];
for ( var i = 0; i < files.length; i++ ) {
promises.push( readFile_promise( files[i] ) );
}
Promise.all( promises ).then( function(results) {
var totalBytes = 0;
for ( i = 0; i < results.length; i++ ) {
totalBytes += results[i].length;
}
console.log("Done reading files. totalBytes = " + totalBytes);
}, function( error ) {
console.log("Error reading files");
});
}, function( error ) {
console.log( "readdir failed.");
});
</pre>
<h2>Reading the files sequentially using Promises</h2>
<p>
By default, promises don't support sequential operations very well. But we can build on them using a <a href="https://github.com/smhanov/node-promise-sequence">PromiseSequence</a>, which adds the ability to define a series of steps and loops, which are performed sequentially.
<p>
Here is the program again, reading a file. Instead of indenting the code by many levels, we are able to write it in more of a sequential style. Also, the above examples had two places where errors had to be handled. With a promise sequence, the errors for any operation in the sequence are handled in one place.
<p>
We first add the readdir() operation to the sequence, and then add a loop() to the sequence. The loop executes repeatedly until exitLoop() is called. Since there are no further step, the argument to exitLoop resolves the promise and the program ends.
<pre>
// Read all the files in the folder in a sequence, using Promises
var fs = require("fs");
var Promise = require("promise");
var PromiseSequence = require("./PromiseSequence").PromiseSequence;
// Wrap the io functions with ones that return promises.
var readdir_promise = Promise.convertNodeAsyncFunction(fs.readdir);
var readFile_promise = Promise.convertNodeAsyncFunction( fs.readFile );
var seq = new PromiseSequence();
var index = 0;
var totalBytes = 0;
var files = null;
seq.add( function() {
return readdir_promise( "." );
});
seq.loop(
// The "next" function of the loop takes the result of the readdir and
// reads the file. It is executed when the loop is entered, and again after
// each time the body is executed.
function( files_arg ) {
files = files_arg;
if ( index == files.length ) {
seq.exitLoop(totalBytes);
return;
} else {
console.log("Reading file " + files[index]);
return readFile_promise( files[index++] );
}
},
// The "body" function of the loop is called with the result of the "next" function.
// It simply sums the length of the file.
function( contents ) {
totalBytes += contents.length;
}
);
seq.run().then( function(total) {
console.log("Done reading file. Total bytes: " + total);
}, function(error) {
console.log("Error reading files: ", error);
});
</pre>
<h2>Why does my brain hurt?</h2>
I have been programming for over two decades. I know lots of languages, but only Javascript makes my brain hurt when I have to do simple things. If you have a lot of asynchronous operations to perform, and you have the choice, please consider your language selection very carefully. <ul><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=138'>Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US</a><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=4'>UMA and free long distance</a></ul>
Zero load time file formats
tag:smhanovtechblog,2007:id123
2011-06-29T17:27:08-05:00
2011-06-29T17:27:08-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Sometimes you cannot afford to load data files from disk. Maybe you need
results immediately, or the data is simply too large to fit into memory. A
technique that I like to use is an on-disk data structure. Here is a toy
example for instantly accessing lists of related words.
<p>
In this article, I address the problem of the time needed to load data into memory from disk. However, I do not make any optimization for disk caches or blocks. I am not going to talk about B-Trees or cache-oblivious structures.
<h2>No waiting</h2>
Using an on-disk data structure, there is no need to load the whole file into memory
or parse it. Instead of opening a file and reading its contents,
we will use a memory mapped file. We tell the operating system the file name,
and it will lazily load the parts of the file only when we access them. These
parts remain in the disk cache even after our program exits. So if you later
start the program again, it will execute similar queries more quickly. We let
the operating system do the caching for us.
In python, this is done using the <a href="http://docs.python.org/library/mmap.html">mmap</a> module. Mmap makes the file appear as a
very long string.
<h2>Toy example</h2>
Here is an on-disk structure for looking up related words that I prepared.
(<a href="http://www.hanovsolutions.com/thesaurus.dat">Download 11 MB of it</a>). It has three sections: A header, an index, and a word
section. The header contains the number of words. The index contains a list of
<i>pointers</i> to word records. The word section contains the word records.
It is constructed so that we can instantly query for words related to a given
word by jumping around to different parts of the file.
<pre>
--- header
4 bytes: number of words
--- index section. The words are listed in alphabetical order, so you can
--- look one up using binary search.
for each word:
4 byte ptr to word record
--- word section:
for each word:
null terminated text
4 bytes: number of related words
for each link,
ptr to linked word record
</pre>
Here is a short python program for accessing the data file.
<pre>
#!/usr/bin/python
# An on-disk data structure for finding related words
# By Steve Hanov. This code and data file are released to the public domain.
import sys, mmap, struct
class FrozenThesaurus:
def __init__(self, filename):
self.f = file(filename, "rb")
self.mmap = mmap.mmap( self.f.fileno(), 0, access=mmap.ACCESS_READ )
def getDword( self, ptr ):
# return the 32 bit number beginning at the given byte offset in the
# file.
return struct.unpack("<I", self.mmap[ptr:ptr+4])[0]
def getString( self, ptr ):
# return the null terminated string beginning at the given byte offset.
result = []
while self.mmap[ptr] != "\x00":
result.append(self.mmap[ptr])
ptr += 1
return "".join(result)
def getWordCount(self):
# Retrive the number of words in the file.
return self.getDword(0)
def getWord(self, index):
# Retrive a word, given its index. The index must be less then the word
# count.
return self.getString( self.getDword(4 + index * 4) )
def getIndexOf( self, word ):
# perform a binary search through the index for the given word.
high = self.getWordCount()
low = -1
while (high - low > 1):
probe = (high + low) / 2
candidate = self.getWord(probe)
if candidate == word:
return probe
elif candidate < word:
low = probe
else:
high = probe
return None
def getRelatedWords( self, word ):
# Returns the list of related words to the given word.
results = []
index = self.getIndexOf( word )
if index == None: return results
ptr = self.getDword( 4 + index * 4 )
# skip past the word text
while self.mmap[ptr] != '\x00': ptr += 1
ptr += 1
numRelated = self.getDword( ptr )
for i in range(numRelated):
ptr += 4
results.append( self.getString( self.getDword( ptr ) ) )
return results;
data = FrozenThesaurus("thesaurus.dat")
print data.getRelatedWords(sys.argv[1])
</pre>
<h2>When shouldn't you use this?</h2>
<a href="http://www.sqlite.org/">SQLITE</a> uses memory mapped files internally. If you create the proper indicies,
SQLITE will match the performance of any file format that you can come up with
yourself, though it may be larger. If you can store your data in a relational database, you should
not go through the trouble of creating your own on-disk data structure. In
particular, a thesaurus could easily be stored in an SQLITE database.
<ul><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=11'>Cell Phones on Airplanes</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=74'>Blame the extensions (comic)</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=73'>How to run a linux based home web server</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a></ul>
Finding the top K items in a list efficiently
tag:smhanovtechblog,2007:id122
2011-06-04T22:56:39-05:00
2011-06-04T22:56:39-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Algorithms will always matter. Sure, processor speeds are still increasing. But the problems that we want to solve using those processors are increasing in size faster. People who are dealing with social network graphs, or analyzing twitter posts, or searching images, or solving any of the hundreds of problems in vogue would be wasting time without the fastest possible hardware. But they would sitting around forever if they weren't using the right tools.
<p>
That's why I get sad when I see code like this:
<pre>
# find the top 10 results
results = sorted(results, reverse=True)[:10]
</pre>
<p>
Anything involving a sort will usually take O(nlogn) time, which, when dealing with lots of items, will keep you waiting around for several seconds or even minutes. An O(nlogn) algorithm, for large N, simply cannot be run in realtime when users are waiting.
<h2>The Heap</h2>
Finding the top K items can be done in O(nlogk) time, which is much, much faster than O(nlogn), using a heap (<a href="http://en.wikipedia.org/wiki/Heap_(data_structure)">wikipedia</a>). Or, since <a href="http://stevehanov.ca/blog/index.php?id=95">I usually end up rewriting everything in C++ eventually</a>, a <a href="http://www.cplusplus.com/reference/stl/priority_queue/">priority queue</a>.
<p>
The strategy is to go through the list once, and as you go, keep a list of the top k elements that you found so far. To do this efficiently, you have to always know the smallest element in this top-k, so you can possibly replace it with one that is larger. The heap structure makes it easy to maintain this list without wasting any effort. It is like a lazy family member who always does the absolute minimum amount of work. It only does enough of the sort to find the smallest element, and that is why it is fast.
<p>
Here's some code to demonstrate the difference between a linear search, and a heap search to find the top K elements in a large array. The heap search is 4 times faster, despite the test being biased in favour of the linear search. The linear search ends up executing in compiled C inside python itself, while the heap search is completely in interpreted python. If they were both in C, the difference in performance would be more pronounced.
<pre>
#!/usr/bin/python
import heapq
import random
import time
def createArray():
array = range( 10 * 1000 * 1000 )
random.shuffle( array )
return array
def linearSearch( bigArray, k ):
return sorted(bigArray, reverse=True)[:k]
def heapSearch( bigArray, k ):
heap = []
# Note: below is for illustration. It can be replaced by
# heapq.nlargest( bigArray, k )
for item in bigArray:
# If we have not yet found k items, or the current item is larger than
# the smallest item on the heap,
if len(heap) < k or item > heap[0]:
# If the heap is full, remove the smallest element on the heap.
if len(heap) == k: heapq.heappop( heap )
# add the current element as the new smallest.
heapq.heappush( heap, item )
return heap
start = time.time()
bigArray = createArray()
print "Creating array took %g s" % (time.time() - start)
start = time.time()
print linearSearch( bigArray, 10 )
print "Linear search took %g s" % (time.time() - start)
start = time.time()
print heapSearch( bigArray, 10 )
print "Heap search took %g s" % (time.time() - start)
</pre>
<pre>
Creating array took 7.15145 s
[9999999, 9999998, 9999997, 9999996, 9999995, 9999994, 9999993, 9999992, 9999991, 9999990]
Linear search took 10.9981 s
[9999990, 9999992, 9999991, 9999994, 9999993, 9999998, 9999997, 9999996, 9999999, 9999995]
Heap search took 2.66371 s
</pre>
<p>
Also, if you see stuff like this, you should go directly to the wikipedia page on the <a href="http://en.wikipedia.org/wiki/Selection_algorithm">Selection Algorithm</a>
<pre>
# find the median
median = sorted(results)[len(results)/2]
</pre>
<p>
My Heap in Javascript: <a href="https://gist.github.com/smhanov/d7ad133e09c319dbd18f">Github</a>
<ul><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=82'>The PenIsland Problem: Text-to-speech for domain names</a><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a></ul>
An instant rhyming dictionary for any web site
tag:smhanovtechblog,2007:id121
2011-06-04T21:30:00-05:00
2011-06-04T21:30:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Many good web applications, and many bad ones, have an API, and my hobby project, <a href="http://rhymebrain.com">RhymeBrain.com</a>, is no exception. The trouble is: the target users of both the web site and the API don't know the difference between Javascript and Java. They don't even know what "A.P.I." stands for. The most they can do is edit some HTML and paste in some code, so that's what <a href="http://rhymebrain.com/api.html">my API</a> has to be.
<p>
I managed to get it down to that using a technique called JSONP. This technique avoids any problems with cross domain requests, and allows non-coders to use the API using a short, customizable HTML snippet.
<h2>Demonstration</h2>
Paste this in your web site:<p>
<textarea rows=10 cols=85 onclick="this.select()">
<form action="javascript:RhymeBrainSubmit()">
<input type=text id="RhymeBrainInput">
<input type=submit value="Rhyme">
</form>
<script type="text/javascript">
var RhymeBrainMaxResults = 50;
</script>
<script type="text/javascript" src="http://rhymebrain.com/external.js"></script>
Rhyme results are provided by <a href="http://rhymebrain.com">RhymeBrain.com</a>
</textarea>
<p>
... and you get this:
<form action="javascript:RhymeBrainSubmit()">
<input type=text id="RhymeBrainInput">
<input type=submit value="Rhyme">
</form>
<script type="text/javascript">
var RhymeBrainMaxResults = 50;
</script>
<script type="text/javascript" src="http://rhymebrain.com/external.js"></script>
<h2>How it works</h2>
<p>
1. The API user pastes a form and reference to <a href="http://rhymebrain.com/external.js">a script</a> into their web page.
<p>
2. The script inserts some more DIV elements to contain the search results. The DIVs could have been included in the pasted code, but this way is more flexible in case I want to change it later on.
<pre>
var resultDiv;
document.write("<div id='RhymeBrainResultDiv'></div><div style='clear:both'></div>");
resultDiv = document.getElementById("RhymeBrainResultDiv");
</pre>
<p>
When the user clicks the Rhyme button, this function is called.
It creates a special URL that points back to a program on my server. For example, this one: <a href="http://rhymebrain.com/talk?function=getRhymes&maxResults=10&word=javascript&jsonp=RhymeBrainResponse">http://rhymebrain.com/talk?function=getRhymes&maxResults=10&word=javascript&jsonp=RhymeBrainResponse</a>. It adds a brand new script element to the end of the document, and sets it source to point back to a program on the rhymebrain server.
<pre>
function RhymeBrainSubmit()
{
var input = document.getElementById("RhymeBrainInput");
var word = input.value;
$(resultDiv).empty();
resultDiv.appendChild( img );
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "http://rhymebrain.com/talk?function=getRhymes" +
"&word=" + encodeURIComponent(word) +
"&maxResults=" + MaxResults +
"&jsonp=RhymeBrainResponse";
document.body.appendChild(script);
}
</pre>
3. The <a href="http://stevehanov.ca/blog/images/acer.jpg">server</a> does some <a href="http://stevehanov.ca/blog/index.php?id=114">super intensive processing</a> <a href="http://stevehanov.ca/blog/index.php?id=95">involving C</a>, mmap() and 100 MB files, and sends back a response. The response happens to be executable javascript.
<pre>
RhymeBrainResponse([ {"word":"equipped", "freq":22,"score":"300","flags":"c","syllables":"2"},
{"word":"manuscript", "freq":22,"score":"300","flags":"bc","syllables":"3"},
{"word":"script", "freq":21,"score":"300","flags":"bc","syllables":"1"},
{"word":"shipped", "freq":21,"score":"300","flags":"c","syllables":"1"},
{"word":"slipped", "freq":21,"score":"300","flags":"c","syllables":"1"},
{"word":"stripped", "freq":21,"score":"300","flags":"c","syllables":"1"},
{"word":"dipped", "freq":20,"score":"300","flags":"c","syllables":"1"},
{"word":"tipped", "freq":20,"score":"300","flags":"c","syllables":"1"},
{"word":"whipped", "freq":20,"score":"300","flags":"c","syllables":"1"},
{"word":"gripped", "freq":19,"score":"300","flags":"c","syllables":"1"}]);
</pre>
<p>
4. The browser executes the javascript, which calls the function name that was passed as the jsonp parameter to the script URL. It calls a function which displays the results on the web page.
<p>
That's all there is to it. JSONP neatly sidesteps the problem cross domain requests, since script tags can be included from any domain, and it provides a way for non-coders to create mashups.
<ul><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=101'>"Your program is stupid. It doesn't work," my wife told me</a><li><a href='?id=139'>The strange man reading a novel in the meeting room</a><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=31'>Free, Raw Stock Data</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a><li><a href='?id=97'>Creating portable binaries on Linux</a></ul>
Succinct Data Structures: Cramming 80,000 words into a Javascript file.
tag:smhanovtechblog,2007:id120
2011-03-20T22:47:59-05:00
2011-03-20T22:47:59-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Let's continue our short tour of data structures for storing words. Today, we will over-optimize John Resig's <a href="http://ejohn.org/blog/javascript-trie-performance-analysis/">Word Game</a>. Along the way, we shall learn about a little-known branch of computer science, called <i>succinct data structures</i>.
<p>
John wants to load a large dictionary of words into a web application, so his Javascript program can quickly check if a word is in the dictionary. He could transfer the words as long string, separated by spaces. This doesn't take much space once it is gzip-compressed by the web server. However, we also have to consider the amount of memory used in the browser itself. In a mobile application, memory is at a premium. If the user switches tabs, everything not being used is swapped out to <a href="http://en.wikipedia.org/wiki/Flash_memory">flash memory</a>. This results in long pauses when switching back.
<p>
One of the best data structures for searching a dictionary is a trie. The speed of search does not depend on the number of words in the dictionary. It depends only on the number of letters in the word. For example, here is a trie containing the words "hat", "it", "is", and "a". The trie seems to compress the data, since words sharing the same beginnings only show up once.
<p align=center>
<img src="http://zwibbler.com/shared/1024.png">
</p>
<p>
We need to solve two problems. If we transmit the word list to the web browser, it then has to build the trie structure. This takes up a lot of time and memory. To save time, we could pre-encode the trie on the server in JSON format, which is parsed very quickly by the web browser. However, JSON is not a compact format, so some bandwidth is wasted downloading the data to the browser. We could avoid the wasted bandwidth by compressing the trie using <a href="http://lookups.pageforest.com/">a more compact format</a>. The data is then smaller, but the web browser still has to decompress it to use it. In any case, the browser needs to create the trie in memory.
<p>
This leads us to the the second major problem. Despite appearances, <i>tries use a lot of memory</i> to store all of those links between nodes.
<p>
Fortunately, there is a way to store these links in a tiny amount of space.
<p>
<h2>Succinct Data Structures</h2>
Succinct data structures were introduced in Guy Jacobson's 1989 thesis, which you cannot read because it is not available anywhere. Fortunately, this important work has been referenced by many <a href="http://www.cs.cmu.edu/afs/cs/project/aladdin/wwwlocal/compression/00063533.pdf">other papers</a> <a href="http://www.aclweb.org/anthology/P/P09/P09-2086.pdf">since</a> <a href="http://www.springerlink.com/index/pr061mhfqwm6grrw.pdf">then.</a>
<p>A succinct data structure <b>encodes data very efficiently,</b> so that it <b>does not need to be decoded to be used.</b> Everything is accessed in-place, by reading bits at various positions in the data. To achieve optimal encoding, we use bits instead of bytes. All of our structures are encoded as a series of 0's and 1's.
<p>
Two important functions for succinct structures are:
<ul>
<li><b>rank(x)</b> - returns the number of bits set to 1, up to and including position x
<li><b>select(y)</b> - returns the position of the <i>yth</i> 1. This is the inverse of the rank function. For example, if select(8) = 10, then rank(10) = 8.
</ul>
<p>Corresponding functions exist to find the rank/select of 0's instead of 1's. The rank function can be implemented in O(1) time using a lookup table (called a "directory"), which summarizes the number of 1's in certain parts of the string. The select() function is implemented in O(logn) time by performing binary search on the rank() function. It is possible <a href="http://uwspace.uwaterloo.ca/handle/10012/64">to implement <i>select</i> in constant time</a>, but it is complicated and space-hungry.
<center>
<table border=1 cellspacing=0>
<tr>
<th>p</th>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>5</td>
<td>6</td>
<td>7</td>
</tr>
<tr>
<th>Bit</th>
<td>1</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>1</td>
</tr>
<tr>
<th>rank(p)</th>
<td>1</td>
<td>2</td>
<td>2</td>
<td>2</td>
<td>2</td>
<td>2</td>
<td>2</td>
<td>3</td>
</tr>
<tr>
<th>select(p)</th>
<td></td>
<td>0</td>
<td>1</td>
<td>7</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
</table>
</center>
<p>
<h2>A Succinct Trie</h2>
Here's a trie containing the words "hat", "is", "it", and "a".
<p align=center>
<img src="http://zwibbler.com/shared/1024.png">
</p>
First, we add a "super root". This is just an additional node above the root. It's there to make the math work out later. <p>
We then process the nodes in <i>level order</i> -- that is, we go row by row and process the nodes left to right. We encode them to the bit string in that order.
<p>
In the picture below, I've labeled each node in level order for convenience. I've also placed the nodes encoding above it. The encoding is a "1" for each child, plus a 0. So a node with 5 children would be "111110" and a node with no children is "0".
<p align=center>
<img src="http://zwibbler.com/shared/1025.png">
</p>
<p>
Now, we encode the nodes one after another. In the example, the bits would be 10111010110010000. I've separated them out in this table so you can see what's going on, but only the middle row is actually stored.
</p>
<center>
<table cellspacing=0 border=1>
<tr>
<th>Position</th>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>5</td>
<td>6</td>
<td>7</td>
<td>8</td>
<td>9</td>
<td>10</td>
<td>11</td>
<td>12</td>
<td>13</td>
<td>14</td>
<td>15</td>
<td>16</td>
</tr>
<tr>
<th>Bit</th>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<th>Node</th>
<td colspan=2></td>
<td colspan=4>0</td>
<td colspan=2>1</td>
<td colspan=3>2</td>
<td colspan=1>3</td>
<td colspan=2>4</td>
<td colspan=1>5</td>
<td colspan=1>6</td>
<td colspan=1>7</td>
</tr>
</table>
</center>
<p>
We then encode the <i>data</i> for each node after that. To get the data for a given node, just read it directly from that node's index in the data array.
<pre>
hiaatst
</pre>
<h3>Getting the data</h3>
<p>The main thing that we want to do with a trie is follow links from each node to its children. Using our encoding, we can follow a link using a simple formula. If a node is numbered <i>i</i>, then the number of its first child is <i>select<sub>0</sub>(i + 1) - i</i>. The second child is the one after that, and so forth. To obtain the number of children, look up the first child of the <i>i+1</i>th node and subtract, since they are stored consecutively.
<p>
For example: We want the first child of node 2. The 3rd 0 is at position 7. Seven minus two is five. Therefore the first child is numbered 5. Similarly the first child of node 3 is found to be 7 by this formula (no, it doesn't really exist, but it works for the calculation). So node 2 has 7 minus 5 equals 2 children.
<h2>Demo</h2>
Here is a demonstration, hosted on my faster server. (Source code: <a href="http://www.hanovsolutions.com/trie/Bits.js">Bits.js</a>) (<a href="?id=120">It doesn't work in RSS readers -- go to my blog to see it</a>. Paste a list of words in the top text area (or click Load dictionary to load one). Click "Encode" to create the trie and encode it. This step can be very slow, because I did not optimize the encoding process. Once encoding is complete, you can use the Lookup button to check if words are in the dictionary.
<p>
Using this encoding method, a 611K dictionary containing 80000 words is compressed to 216K, or 132K gzipped. The browser does not need to decode it to use it. The whole trie takes as much space as a 216K string.
<p>
<iframe width=700 height=700 src="http://www.hanovsolutions.com/trie/index.html">
Your browser does not support iframes
</iframe>
<h3>Details</h3>
The <i>directory</i> contains the information needed to compute the rank and select functions quickly. The <i>trie</i> is the bitstring representing the trie and the connections between all of its nodes.
<p>
To avoid problems with UTF encoding formats and escaped characters, the bit strings are encoded in BASE-64. All of the bit decoding functions are configured to operated on BASE64 encoded units, so that the input string does not need to be decoded before being used.
<p>
We only handle the letters "a" to "z" in lower case. That way, we can encode each letter in 5 bits.
<p>
You can decrease space usage and performance by increasing the L2 constant, and setting L1 = L2*L2. This controls the number of bits summarized in each section of the rank directory. L2 is the maximum number of bits that have to be scanned to implement rank(). More bits means fewer directory entries, but the select() and rank() functions will take longer to scan the range of bits.
<h3>Caveats</h3>
I described <a href="?id=115">how to create an MA-FSA</a> in a previous article. There is no known way to succinctly encode one. You must store one pointer for each edge. However, as the number of words increases, an MA-FSA (also known as a DAWG) may eventually become more compact than the trie. This is because a trie does not compress common word endings together.
<ul><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=97'>Creating portable binaries on Linux</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=74'>Blame the extensions (comic)</a><li><a href='?id=58'>Email Etiquette</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=7'>Rules for Effective C++</a></ul>
Throw away the keys: Easy, Minimal Perfect Hashing
tag:smhanovtechblog,2007:id119
2011-03-09T18:00:00-05:00
2011-03-09T18:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<div style="border:3px solid #800000; background: #ccc">
<b>CORRECTION</b>: In this article, I incorrectly state that an acyclic finite state automata (aka a DAWG) cannot be used to retrieve values associated with its keys. I have since learned that it can. By storing in each internal node the number of leaf nodes that are reachable from it, we can, upon arriving at the destination key, know its unique index number. Then its value can be looked up in an array. I am now using this technique on <a href="http://rhymebrain.com">rhymebrain.com</a>
<p>
This technique is described in <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.56.5272&rep=rep1&type=pdf">Kowaltowski, T.; CL. Lucchesi (1993), "Applications of finite automata representing large vocabularies", Software-Practice and Experience 1993</a>
<p>
Here's my implementation using an MA-FSA as a map: <a href="https://gist.github.com/smhanov/94230b422c2100ae4218">Github</a>
</div>
<p>
In <a href="http://stevehanov.ca/blog/index.php?id=114">part 1</a> of this series, I described how to find the closest match in a dictionary of words using a Trie. Such searches are useful because users often mistype queries. But tries can take a lot of memory -- so much that they may not even fit in the 2 to 4 GB limit imposed by 32-bit operating systems.
<p align=center><img src="http://zwibbler.com/shared/920.png"></p>
<p>
In <a href="http://stevehanov.ca/blog/index.php?id=115">part 2</a>, I described how to build a MA-FSA (also known as a DAWG). The MA-FSA greatly reduces the number of nodes needed to store the same information as a trie. They are quick to build, and you can safely substitute an MA-FSA for a trie in the fuzzy search algorithm.
<p align=center><img src="http://zwibbler.com/shared/921.png"></p>
<p>
There is a problem. Since the last node in a word is shared with other words, it is not possible to store data in it. We can use the MA-FSA to check if a word (or a close match) is in the dictionary, but we cannot look up any other information about the word!
<p>
If we need extra information about the words, we can use an additional data structure along with the MA-FSA. We can store it in a hash table. Here's an example of a hash table that uses separate chaining. To look up a word, we run it through a hash function, H() which returns a number. We then look at all the items in that "bucket" to find the data. Since there should be only a small number of words in each bucket, the search is very fast.
<p align=center><img src="http://zwibbler.com/shared/997.png"></p>
<p>
Notice that the table needs to store the keys (the words that we want to look up) as well as the data associated with them. It needs them to resolve collisions -- when two words hash to the same bucket. Sometimes these keys take up too much storage space. For example, you might be storing information about all of the URLs on the entire Internet, or parts of the human genome. In our case, we already store the words in the MA-FSA, and it is redundant to duplicate them in the hash table as well. If we could guarantee that there were no collisions, we could throw away the keys of the hash table.
<h3>Minimal perfect hashing</h3>
<p>
Perfect hashing is a technique for building a hash table with no collisions. It is only possible to build one when we know all of the keys in advance. <i>Minimal</i> perfect hashing implies that the resulting table contains one entry for each key, and no empty slots.
<p>
We use two levels of hash functions. The first one, H(key), gets a position in an intermediate array, G. The second function, F(d, key), uses the extra information from G to find the unique position for the key. The scheme will always returns a value, so it works as long as we know <i>for sure</i> that what we are searching for is in the table. Otherwise, it will return bad information. Fortunately, our MA-FSA can tell us whether a value is in the table. If we did not have this information, then we could also store the keys with the values in the value table.
<p>
In the example below, the words "blue" and "cat" both hash to the same position using the H() function. However, the second level hash, F, combined with the <i>d</i>-value, puts them into different slots.
<p align=center><img src="http://zwibbler.com/shared/998.png"></p>
How do we find the intermediate table, G? <b>By trial and error.</b> But don't worry, if we do it carefully, <a href="http://cmph.sourceforge.net/papers/esa09.pdf">according to this paper</a>, it only takes linear time.
<p>
<b>In step 1</b>, we place the keys into buckets according to the first hash function, H.
<p>
<b>In step 2</b>, we process the buckets <i>largest first</i> and try to place all the keys it contains in an empty slot of the value table using F(d=1, key). If that is unsuccessful, we keep trying with successively larger values of d. It <i>sounds like</i> it would take a long time, but in reality it doesn't. Since we try to find the <i>d</i> value for the buckets with the most items early, they are likely to find empty spots. When we get to buckets with just one item, we can simply place them into the next unoccopied spot.
<p>
Here's some python code to demonstrate the technique. In it, we use H = F(0, key) to simplify things.
<pre>
#!/usr/bin/python
# Easy Perfect Minimal Hashing
# By Steve Hanov. Released to the public domain.
#
# Based on:
# Edward A. Fox, Lenwood S. Heath, Qi Fan Chen and Amjad M. Daoud,
# "Practical minimal perfect hash functions for large databases", CACM, 35(1):105-121
# also a good reference:
# Compress, Hash, and Displace algorithm by Djamal Belazzougui,
# Fabiano C. Botelho, and Martin Dietzfelbinger
import sys
DICTIONARY = "/usr/share/dict/words"
TEST_WORDS = sys.argv[1:]
if len(TEST_WORDS) == 0:
TEST_WORDS = ['hello', 'goodbye', 'dog', 'cat']
# Calculates a distinct hash function for a given string. Each value of the
# integer d results in a different hash value.
def hash( d, str ):
if d == 0: d = 0x01000193
# Use the FNV algorithm from http://isthe.com/chongo/tech/comp/fnv/
for c in str:
d = ( (d * 0x01000193) ^ ord(c) ) & 0xffffffff;
return d
# Computes a minimal perfect hash table using the given python dictionary. It
# returns a tuple (G, V). G and V are both arrays. G contains the intermediate
# table of values needed to compute the index of the value in V. V contains the
# values of the dictionary.
def CreateMinimalPerfectHash( dict ):
size = len(dict)
# Step 1: Place all of the keys into buckets
buckets = [ [] for i in range(size) ]
G = [0] * size
values = [None] * size
for key in dict.keys():
buckets[hash(0, key) % size].append( key )
# Step 2: Sort the buckets and process the ones with the most items first.
buckets.sort( key=len, reverse=True )
for b in xrange( size ):
bucket = buckets[b]
if len(bucket) <= 1: break
d = 1
item = 0
slots = []
# Repeatedly try different values of d until we find a hash function
# that places all items in the bucket into free slots
while item < len(bucket):
slot = hash( d, bucket[item] ) % size
if values[slot] != None or slot in slots:
d += 1
item = 0
slots = []
else:
slots.append( slot )
item += 1
G[hash(0, bucket[0]) % size] = d
for i in range(len(bucket)):
values[slots[i]] = dict[bucket[i]]
if ( b % 1000 ) == 0:
print "bucket %d r" % (b),
sys.stdout.flush()
# Only buckets with 1 item remain. Process them more quickly by directly
# placing them into a free slot. Use a negative value of d to indicate
# this.
freelist = []
for i in xrange(size):
if values[i] == None: freelist.append( i )
for b in xrange( b, size ):
bucket = buckets[b]
if len(bucket) == 0: break
slot = freelist.pop()
# We subtract one to ensure it's negative even if the zeroeth slot was
# used.
G[hash(0, bucket[0]) % size] = -slot-1
values[slot] = dict[bucket[0]]
if ( b % 1000 ) == 0:
print "bucket %d r" % (b),
sys.stdout.flush()
return (G, values)
# Look up a value in the hash table, defined by G and V.
def PerfectHashLookup( G, V, key ):
d = G[hash(0,key) % len(G)]
if d < 0: return V[-d-1]
return V[hash(d, key) % len(V)]
print "Reading words"
dict = {}
line = 1
for key in open(DICTIONARY, "rt").readlines():
dict[key.strip()] = line
line += 1
print "Creating perfect hash"
(G, V) = CreateMinimalPerfectHash( dict )
for word in TEST_WORDS:
line = PerfectHashLookup( G, V, word )
print "Word %s occurs on line %d" % (word, line)
</pre>
<h3>Experimental Results</h3>
I prepared separate lists of randomly selected words to test whether the runtime is really linear as claimed.
<table border=1>
<tr><th>Number of items</th>
<th>Time (s)</th>
</tr>
<tr><td>100000</td><td>2.24</td></tr>
<tr><td>200000</td><td>4.48</td></tr>
<tr><td>300000</td><td>6.68</td></tr>
<tr><td>400000</td><td>9.27</td></tr>
<tr><td>500000</td><td>11.71</td></tr>
<tr><td>600000</td><td>13.81</td></tr>
<tr><td>700000</td><td>16.72</td></tr>
<tr><td>800000</td><td>18.78</td></tr>
<tr><td>900000</td><td>21.12</td></tr>
<tr><td>1000000</td><td>24.816</td></tr>
</table>
<p>Here's a pretty chart.
<p align=center>
<img src="http://chart.apis.google.com/chart?chxr=0,0,1000000|1,0,24&chxt=x,y&chs=500x400&cht=s&chco=989866&chds=0,25,0,1000000,0,100&chd=t:2.24,4.48,6.68,9.27,11.71,13.81,16.72,18.78,21.12,24.81|100000,200000,300000,400000,500000,600000,700000,800000,900000,1000000|84,23,69,81,47,94,60,93,64,54&chtt=Number+of+items+vs.+Time+(s)+in+Perfect+Hashing+algorithm" width="500" height="400" alt="Number of items vs. Time (s) in Perfect Hashing algorithm" />
</p>
<h3>CMPH</h3>
<a href="http://cmph.sourceforge.net/">CMPH</a> is an LGPL library that contains a really fast implementation of several perfect hash function algorithms. In addition, it compresses the G array so that it can still be used without decompressing it. It was created by the authors of one of the papers that I cited. <a href="http://cmph.sourceforge.net/papers/thesis.pdf">Botelho's thesis</a> is a great introduction to perfect hashing theory and algorithms.
<h3>gperf</h3>
<a href="http://www.gnu.org/software/gperf/">Gperf</a> is another open source solution. However, it is designed to work with small sets of keys, not large dictionaries of millions of words. It expects you to embed the resulting tables directly in your C source files.
<h3>Dr. Daoud's Page</h3>
Dr. Daoud contacted me below. <a href="http://iswsa.acm.org/mphf/index.html">Minimal Perfect Hashing Resources</a> contains an even better implementation of the algorithm with a more compact representation. He originally invented the algorithm in 1987 and I am honoured that he adapted my python algorithm to make it even better!<ul><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=66'>Test Driven Development without Tears</a><li><a href='?id=144'>Finding Bieber: On removing duplicates from a set of documents</a><li><a href='?id=39'>Stock Picking using Python</a></ul>
Why don't web browsers do this?
tag:smhanovtechblog,2007:id117
2011-02-06T17:10:14-05:00
2011-02-06T17:10:14-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
In the 80's, computers started instantly. They were READY to go when they first turned on.
<p align=center>
<img src="http://upload.wikimedia.org/wikipedia/commons/4/48/C64_startup_animiert.gif">
<p>
Over the next few decades, people wanted to do more things and operating systems got slower to initialize. To solve this, OS and hardware manufacturers created hibernate and standby modes.
<p align=center>
<img src=hibernate.png>
<p>
Now, many people have stopped using native applications and moved to the web. When I load facebook or gmail, it takes dozens of seconds to start up, and minutes over a slower connection. During this time,
<ol>
<li>The source files for the application are loaded from the server,
<li>The source code is compiled and run.
<li>Requests are made to retrieve the application state from the server, and
<li>the DOM is manipulated to present the state to the user.
</ol>
<p align=center>
<img src=loading-gmail.png>
<p>
It would be trivial to snapshot the DOM and application state in Javascript and provide access to these snapshots with a simple API. The API would also allow you to discard an application version that is too old, or convert the state to the newer one. Then, application startup would be instantaneous.
<p>
Or, without any co-operation from standards, browsers can do this RIGHT NOW and snapshot commonly used pages instead of discarding them when users close a tab. When the url is re-entered, from the application perspective it is just as if the machine went into standby and then resumed. The browser could take cookie expiration into account, or to be totally safe, web pages could opt in with a meta tag.
<p>
Just sayin'.
<ul><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a><li><a href='?id=137'>Asana's shocking pricing practices, and how you can get away with it too</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=128'>Why you should go to the Business of Software Conference Next Year</a><li><a href='?id=134'>0, 1, Many, a Zillion</a><li><a href='?id=70'>Finding great ideas for your startup</a></ul>
Fun with Colour Difference
tag:smhanovtechblog,2007:id116
2011-02-04T10:46:55-05:00
2011-02-04T10:46:55-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<a href="http://blog.asmartbear.com/color-wheels.html"></a>
<p>
Are you looking for a nifty way to choose colours that stand out? Are you the type of person who is not satisfied until you have <b>mathematically proven that your choice is optimal?</b>
<p>One way to do it is to treat red, green, and blue colour values as coordinates in a cube. Two colours are different if the distance between their coordinates is large. But the RGB colour space is not perceptually uniform. Because of <a href="http://blog.asmartbear.com/color-wheels.html">the way the human eye works</a>, lots of greens look the same, but we can easily see the difference between subtle shades of yellow. That's why <a href="http://vimeo.com/11688209">George Takei is hawking TVs</a>. It's also why perceptually uniform colour spaces, such as <a href="http://en.wikipedia.org/wiki/Lab_color_space">LAB</a> or <a href="http://en.wikipedia.org/wiki/CIELUV">LUV</a> warp the cube, as if it were made of play-dough, and left out in the sun for a while. The result is that the differences between the coordinates almost correspond to the perceived difference between two colours for most people.
<p>
The colour-space calculations are all on wikipedia, and they are dead simple to implement. For fun, I put them into a simple force system using Javascript (You'll need an HTML5 browser to view. If you're using an RSS reader, you'll have to go to <a href="?id=116">my blog</a> to see it.)
<h3>Explanation</h3>
<p>
Below are all of the CSS colours which have <a href="http://www.w3schools.com/cssref/css_colors.asp">names commonly recognized by most browsers</a>. Every colour name from "AliceBlue" to "Gainsboro" to "YellowGreen" is there. The circles float freely, and are repelled by each other and the four sides of their container.
<p>
When you click on a colour, the background changes to that colour. All of the circles are then attracted to a vertical position based on how different they are from the background. Those near the top are close to the background colour. Those near the bottom are further away from the background. You can change the colour space in which the distance is calculated by clicking on band at the top of the container.
<p>
For HSV, the H parameter is divided by 360 before the distance is calculated, to make its influence fair, since the S and V values range from 0 to 1.
<p align=center>
<canvas id=canvas width=700 height=500>
Sorry, you have to use a browser that supports HTML5 and go to <a href="?id=116">my blog</a> to see it.
</canvas>
<script type="text/javascript" src="../colours/Colours.js"></script>
<script type="text/javascript" src="../colours/driver.js"></script>
<h3>Observations</h3>
In RGB, black is the most different from white. But in LAB, black is in the middle somewhere, and other dark colours are more distant.
<p>
To my eye, both RGB and LAB perform well for finding differences, but HSV results in some odd choices. To choose a contrasting colour, use RGB or LAB and avoid picking anything less than a third of the way down.
<h3>Source code</h3>
<p>The source code is released to the public domain.
<ul>
<li><a href="../colours/Colours.js">Colours.js</a> - A simple library for converting between text, RGB, HSV, and LAB colours.
<li><a href="../colours/driver.js">driver.js</a> - Does the force calculations based on <a href="http://en.wikipedia.org/wiki/Force-based_algorithms_(graph_drawing)">this wikipedia article</a>, and handles mouse clicks.
</ul>
You might be interested in a previous article about <a href="http://stevehanov.ca/blog/index.php?id=62">exploiting colour difference for edge detection</a>.
<h3>Other articles from my blog</h3><ul><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=140'>A little VIM hacking</a><li><a href='?id=55'>How wide should you make your web page?</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=83'>Sign here (comic)</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=4'>UMA and free long distance</a></ul>
Compressing dictionaries with a DAWG
tag:smhanovtechblog,2007:id115
2011-01-24T21:40:57-05:00
2011-01-24T21:40:57-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Last time, I wrote about how to speed up spell checking using a trie (also known as a prefix tree). However, for large dictionaries, a trie can waste a lot of memory. If you're trying to squeeze an application into a mobile device, every kilobyte counts. Consider this trie, with two words in it.
<p align=center>
<img src="http://zwibbler.com/shared/920.png">
<p>
It can be shortened in a way so that any program accessing it would not even notice.
<p align=center>
<img src="http://zwibbler.com/shared/921.png">
<p>
<a href="http://web.cecs.pdx.edu/~bart/cs542-spring2005/papers/appel-scrabble.ps.gz">As early as 1988,</a> Scrabble<sup>TM</sup> programs were using structures like the above to shrink the their dictionaries. Over the years, the structure has been called many things. <a href="http://www.pathcom.com/~vadco/dawg.html">Some web pages</a> call it a DAWG (Direct Acyclic Word Graph). But computer scientists have adopted the name "Minimal Acyclic Finite State Automaton", because some papers were already using the name DAWG for something else.
<p>
The most obvious way to build a MA-FSA, as suggested in many other web pages, is to first build the trie, and look for duplicate branches. I tried this on a list of 7 million words that I had. I wrote the algorithm in C++, but no matter how hard I tried, I kept running out of memory. A trie (or prefix tree) uses a lot of memory compared to a DAWG. It would be much better if one could create the DAWG right away, without first creating a trie. Jan Duciuk describes such a method in his <a href="http://www.aclweb.org/anthology/J00-1002.pdf">paper</a>. The central idea is to check for duplicates after you insert each word, so that the structure never gets huge.
<ol>
<li>Ensure that words are inserted in alphabetical order. That way, when you insert a word, you will then know for sure whether the previous word ended an entire branch. For example, "cat" followed by "catnip" does not result in a branch, because the s just added to the end. But when you follow it with "cats" you know that the "nip" part of the previous word needs checking.
<li>Each time you complete a branch in the trie, check it for duplicate nodes. When a duplicate is found, redirect all incoming edges to the existing one and eliminate the duplicate.
</ol>
<p align=center>
<img src="http://zwibbler.com/shared/930.png">
<p>
<a href="http://www.aclweb.org/anthology/J00-1002.pdf">The paper that I am paraphrasing,</a> by Jan Daciuk and others, also describes a way to insert words out of order. But it is more complicated. In most cases, you can arrange to add your words in alphabetical order.
<h2>What's a duplicate node?</h2>
<p>
Two nodes are considered the same if they are both the final part of a word, or they are both not the final part of a word. They also need to have exactly the same edges pointing to exactly the same other nodes.
<p align=center>
<img src="http://zwibbler.com/shared/931.png">
<p>
We start eliminating duplicates starting from the bottom of the branch, so each elimination can reveal more duplicates. Eventually, the branch of the trie zips together with a prior branch.
<p align=center>
Step 1:<br>
<img src="http://zwibbler.com/shared/935.png"><br>
Several steps later:<br>
<img src="http://zwibbler.com/shared/933.png">
<h2>Why go through so much trouble?</h2>
<p>If you have a large word list, you could run it through gzip and get much better compression. The reason for storing a dictionary this way is to save space and remain easily searchable, without needing to decompress it first. Tries and MA-FSAs can support fuzzy search and prefix queries, so you can do <a href="?id=114">spell checking</a> and auto-completion. They can easily scale up to billions of entries. They have even been used to store large portions of the human genome. If you don't care about memory or speed, just store your words in an SQL database, or spin up 100 machines "in the cloud". I don't mind. More power to you!
<p>
MA-FSAs can be stored in as little as 4 bytes per edge-connector, as described by <a href="http://www.wutka.com/dawg.html">this web page</a>.
<h2>Implementation</h2>
Here's a python implementation. I tried it and it could easily handle seven million words in a couple minutes.
<pre>
#!/usr/bin/python
# By Steve Hanov, 2011. Released to the public domain.
import sys
import time
DICTIONARY = "/usr/share/dict/words"
QUERY = sys.argv[1:]
# This class represents a node in the directed acyclic word graph (DAWG). It
# has a list of edges to other nodes. It has functions for testing whether it
# is equivalent to another node. Nodes are equivalent if they have identical
# edges, and each identical edge leads to identical states. The __hash__ and
# __eq__ functions allow it to be used as a key in a python dictionary.
class DawgNode:
NextId = 0
def __init__(self):
self.id = DawgNode.NextId
DawgNode.NextId += 1
self.final = False
self.edges = {}
def __str__(self):
arr = []
if self.final:
arr.append("1")
else:
arr.append("0")
for (label, node) in self.edges.iteritems():
arr.append( label )
arr.append( str( node.id ) )
return "_".join(arr)
def __hash__(self):
return self.__str__().__hash__()
def __eq__(self, other):
return self.__str__() == other.__str__()
class Dawg:
def __init__(self):
self.previousWord = ""
self.root = DawgNode()
# Here is a list of nodes that have not been checked for duplication.
self.uncheckedNodes = []
# Here is a list of unique nodes that have been checked for
# duplication.
self.minimizedNodes = {}
def insert( self, word ):
if word < self.previousWord:
raise Exception("Error: Words must be inserted in alphabetical " +
"order.")
# find common prefix between word and previous word
commonPrefix = 0
for i in range( min( len( word ), len( self.previousWord ) ) ):
if word[i] != self.previousWord[i]: break
commonPrefix += 1
# Check the uncheckedNodes for redundant nodes, proceeding from last
# one down to the common prefix size. Then truncate the list at that
# point.
self._minimize( commonPrefix )
# add the suffix, starting from the correct node mid-way through the
# graph
if len(self.uncheckedNodes) == 0:
node = self.root
else:
node = self.uncheckedNodes[-1][2]
for letter in word[commonPrefix:]:
nextNode = DawgNode()
node.edges[letter] = nextNode
self.uncheckedNodes.append( (node, letter, nextNode) )
node = nextNode
node.final = True
self.previousWord = word
def finish( self ):
# minimize all uncheckedNodes
self._minimize( 0 );
def _minimize( self, downTo ):
# proceed from the leaf up to a certain point
for i in range( len(self.uncheckedNodes) - 1, downTo - 1, -1 ):
(parent, letter, child) = self.uncheckedNodes[i];
if child in self.minimizedNodes:
# replace the child with the previously encountered one
parent.edges[letter] = self.minimizedNodes[child]
else:
# add the state to the minimized nodes.
self.minimizedNodes[child] = child;
self.uncheckedNodes.pop()
def lookup( self, word ):
node = self.root
for letter in word:
if letter not in node.edges: return False
node = node.edges[letter]
return node.final
def nodeCount( self ):
return len(self.minimizedNodes)
def edgeCount( self ):
count = 0
for node in self.minimizedNodes:
count += len(node.edges)
return count
dawg = Dawg()
WordCount = 0
words = open(DICTIONARY, "rt").read().split()
words.sort()
start = time.time()
for word in words:
WordCount += 1
dawg.insert(word)
if ( WordCount % 100 ) == 0: print "%dr" % WordCount,
dawg.finish()
print "Dawg creation took %g s" % (time.time()-start)
EdgeCount = dawg.edgeCount()
print "Read %d words into %d nodes and %d edges" % ( WordCount,
dawg.nodeCount(), EdgeCount )
print "This could be stored in as little as %d bytes" % (EdgeCount * 4)
for word in QUERY:
if not dawg.lookup( word ):
print "%s not in dictionary." % word
else:
print "%s is in the dictionary." % word
</pre>
<p>Updated code on github: <a href="https://gist.github.com/smhanov/94230b422c2100ae4218">using a DAWG as a map</a></p>
Using this code, a list of 7 million words, taking up 63 MB, was translated into 6 million edges. Although it took more than a gigabyte of memory in Python, such a list <i>could</i> be stored <a href="http://www.wutka.com/dawg.html">in as little as 24 MB</a>. Of course, gzip could do better, but the result would not be quickly searchable.
<h2>Extensions</h2>
<p>
A MA-FSA is great for testing whether words are in a dictionary. But in the form I gave, it's not possible to retrieve values associated with the words. It is possible to include associated values in the automaton. Such structures are called "Minimal Acyclic Finite State Transducers". In fact, the algorithm I above can be easily modified to include a value. However, it causes the number of nodes to blow up, and you are much better off using a minimal perfect hash function in addition to your MA-FSA to store your data. <a href="?id=119">I discuss this in part 3</a>.
<p>
<ul><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=37'>Spoke.com scam</a><li><a href='?id=78'>It's a dirty job... (comic)</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=82'>The PenIsland Problem: Text-to-speech for domain names</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a></ul>
Fast and Easy Levenshtein distance using a Trie
tag:smhanovtechblog,2007:id114
2011-01-14T20:07:53-05:00
2011-01-14T20:07:53-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
If you have a web site with a search function, you will rapidly realize that most mortals are terrible typists. Many searches contain mispelled words, and users will expect these searches to magically work. This magic is often done using levenshtein distance. In this article, I'll compare two ways of finding the closest matching word in a large dictionary. I'll describe how I use it on <a href="http://rhymebrain.com">rhymebrain.com</a> not for corrections, but to search 2.6 million words for rhymes, for every request, with no caching, on my super-powerful <a href="http://stevehanov.ca/blog/index.php?id=95#sock">sock-drawer datacenter</a>:
<p align=center><img src="images/acer.jpg"></p>
<h2>Algorithm #1</h2>
<p>
The levenshtein function take two words and returns how far apart they are. It's an O(N*M) algorithm, where N is the length of one word, and M is the length of the other. If you want to know how it works, go to this <a href="http://en.wikipedia.org/wiki/Levenshtein_distance">wikipedia page.</a>
<p>
But comparing two words at a time isn't useful. Usually you want to find the closest matching words in a whole dictionary, possibly with many thousands of words. Here's a quick python program to do that, using the straightforward, but slow way. It uses the file /usr/share/dict/words. The first argument is the misspelled word, and the second argument is the maximum distance. It will print out all the words with that distance, as well as the time spent actually searching. For example:
<pre>
smhanov@ubuntu1004:~$ ./method1.py goober 1
('goober', 0)
('goobers', 1)
('gooier', 1)
Search took 4.5575 s
</pre>
<p>
Here's the program:
<pre>
#!/usr/bin/python
#By Steve Hanov, 2011. Released to the public domain
import time
import sys
DICTIONARY = "/usr/share/dict/words";
TARGET = sys.argv[1]
MAX_COST = int(sys.argv[2])
# read dictionary file
words = open(DICTIONARY, "rt").read().split();
# for brevity, we omit transposing two characters. Only inserts,
# removals, and substitutions are considered here.
def levenshtein( word1, word2 ):
columns = len(word1) + 1
rows = len(word2) + 1
# build first row
currentRow = [0]
for column in xrange( 1, columns ):
currentRow.append( currentRow[column - 1] + 1 )
for row in xrange( 1, rows ):
previousRow = currentRow
currentRow = [ previousRow[0] + 1 ]
for column in xrange( 1, columns ):
insertCost = currentRow[column - 1] + 1
deleteCost = previousRow[column] + 1
if word1[column - 1] != word2[row - 1]:
replaceCost = previousRow[ column - 1 ] + 1
else:
replaceCost = previousRow[ column - 1 ]
currentRow.append( min( insertCost, deleteCost, replaceCost ) )
return currentRow[-1]
def search( word, maxCost ):
results = []
for word in words:
cost = levenshtein( TARGET, word )
if cost <= maxCost:
results.append( (word, cost) )
return results
start = time.time()
results = search( TARGET, MAX_COST )
end = time.time()
for result in results: print result
print "Search took %g s" % (end - start)
</pre>
<h3>Runtime</h3>
For each word, we have to fill in an N x M table. An upper bound for the runtime is O( <number of words> * <max word length> ^2 )
<h2>Improving it</h2>
Sorry, now you need to know how the algorithm works and I'm not going to explain it. (You really need to read the <a href="http://en.wikipedia.org/wiki/Levenshtein_distance">wikipedia page.</a>) The important things to know are that it fills in a N x M sized table, like this one, and the answer is in the bottom-right square.
<br>
<table border=1 cellspacing=0 align=center>
<tr><th></th><th></th><th>k</th><th>a</th><th>t</th><th>e</th></tr>
<tr><th> </th><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td></tr>
<tr><th>c</th><td>1</td><td>1</td><td>2</td><td>3</td><td>4</td></tr>
<tr><th>a</th><td>2</td><td>2</td><td>1</td><td>2</td><td>3</td></tr>
<tr><th>t</th><td>3</td><td>3</td><td>2</td><td>1</td><td>2</td></tr>
</table>
<br>
But wait, what's it going to do when it moves on to the next word <i>after</i> cat? In my dictionary, that's "cats" so here it is:
<br>
<table border=1 cellspacing=0 align=center>
<tr><th></th><th></th><th>k</th><th>a</th><th>t</th><th>e</th></tr>
<tr><th> </th><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td></tr>
<tr><th>c</th><td>1</td><td>1</td><td>2</td><td>3</td><td>4</td></tr>
<tr><th>a</th><td>2</td><td>2</td><td>1</td><td>2</td><td>3</td></tr>
<tr><th>t</th><td>3</td><td>3</td><td>2</td><td>1</td><td>2</td></tr>
<tr><th>s</th><td>4</td><td>4</td><td>3</td><td>2</td><td>2</td></tr>
</table>
<br>
<p>
<b>Only the last row changes.</b> We can avoid a lot of work if we can process the words in order, so we never need to repeat a row for the same prefix of letters. The <b>trie</b> data structure is perfect for this. A trie is a giant tree, where each node represents a partial or complete word. Here's one with the words <i>cat, cats, catacomb, and catacombs</i> in it (courtesy of <a href="http://zwibbler.com">zwibbler.com</a>). Nodes that represent a word are marked in black.
<p align=center><img src="http://zwibbler.com/shared/898.png"></p>
<b>With a trie, all shared prefixes in the dictionary are collaped into a single path,</b> so we can process them in the best order for building up our levenshtein tables one row at a time. Here's a python program to do that:
<pre>
#!/usr/bin/python
#By Steve Hanov, 2011. Released to the public domain
import time
import sys
DICTIONARY = "/usr/share/dict/words";
TARGET = sys.argv[1]
MAX_COST = int(sys.argv[2])
# Keep some interesting statistics
NodeCount = 0
WordCount = 0
# The Trie data structure keeps a set of words, organized with one node for
# each letter. Each node has a branch for each letter that may follow it in the
# set of words.
class TrieNode:
def __init__(self):
self.word = None
self.children = {}
global NodeCount
NodeCount += 1
def insert( self, word ):
node = self
for letter in word:
if letter not in node.children:
node.children[letter] = TrieNode()
node = node.children[letter]
node.word = word
# read dictionary file into a trie
trie = TrieNode()
for word in open(DICTIONARY, "rt").read().split():
WordCount += 1
trie.insert( word )
print "Read %d words into %d nodes" % (WordCount, NodeCount)
# The search function returns a list of all words that are less than the given
# maximum distance from the target word
def search( word, maxCost ):
# build first row
currentRow = range( len(word) + 1 )
results = []
# recursively search each branch of the trie
for letter in trie.children:
searchRecursive( trie.children[letter], letter, word, currentRow,
results, maxCost )
return results
# This recursive helper is used by the search function above. It assumes that
# the previousRow has been filled in already.
def searchRecursive( node, letter, word, previousRow, results, maxCost ):
columns = len( word ) + 1
currentRow = [ previousRow[0] + 1 ]
# Build one row for the letter, with a column for each letter in the target
# word, plus one for the empty string at column 0
for column in xrange( 1, columns ):
insertCost = currentRow[column - 1] + 1
deleteCost = previousRow[column] + 1
if word[column - 1] != letter:
replaceCost = previousRow[ column - 1 ] + 1
else:
replaceCost = previousRow[ column - 1 ]
currentRow.append( min( insertCost, deleteCost, replaceCost ) )
# if the last entry in the row indicates the optimal cost is less than the
# maximum cost, and there is a word in this trie node, then add it.
if currentRow[-1] <= maxCost and node.word != None:
results.append( (node.word, currentRow[-1] ) )
# if any entries in the row are less than the maximum cost, then
# recursively search each branch of the trie
if min( currentRow ) <= maxCost:
for letter in node.children:
searchRecursive( node.children[letter], letter, word, currentRow,
results, maxCost )
start = time.time()
results = search( TARGET, MAX_COST )
end = time.time()
for result in results: print result
print "Search took %g s" % (end - start)
</pre>
Here are the results:
<pre>
smhanov@ubuntu1004:~$ ./method1.py goober 1
Read 98568 words into 225893 nodes
('goober', 0)
('goobers', 1)
('gooier', 1)
Search took 0.0141618 s
</pre>
The second algorithm is over 300 times faster than the first. Why? Well, we create at most one row of the table for each node in the trie. The upper bound for the runtime is O(<max word length> * <number of nodes in the trie>). For most dictionaries, considerably less than O(<number of words> * <max word length>^2)
<h2>Saving memory</h2>
Building a trie can take a lot of memory. In <a href="?id=115">Part 2</a>, I discuss how to construct a MA-FSA (or DAWG) which contains the same information in a more compact form.
<h2>RhymeBrain</h2>
<a href="http://rhymebrain.com"><img style="margin:1em" align=left src="http://rhymebrain.com/logo.png"></a>In December, I realized that Google had released their <a href="http://ngrams.googlelabs.com/datasets">N-grams data</a>, a list of all of the words in all of the books that they have scanned for their Books search feature. When I imported them all into RhymeBrain, my dictionary size at once increased from 260,000 to 2.6 million, and I was having performance problems.
<p>
I already stored the words in a trie, indexed by pronunciation instead of letters. However, to search it, I was first performing a <i>quick and dirty</i> scan to find words that might possibly rhyme. Then I took that large list and ran each one through the levenshtein function to calculate RhymeRank<sup>TM</sup>. The user is presented with only the top 50 entries of that list.
<p>
After a lot of deep thinking, I realized that the levenshtein function could be evaluated incrementally, as I described above. Of course, I might have realized this sooner if I had read one of the many scholarly papers on the subject, which describe this exact method. But who has time for that? :)
<p>
With the new algorithm, queries take between 19 and 50 ms even for really long words, but the best part is that I don't need to maintain two separate checks (quick and full), and the RhymeRank<sup>TM</sup> algorithm is performed uniformly for each of the 2.6 million words on my 1GHz Acer Aspire One datacenter.
<p><a href="http://stevehanov.ca/blog/index.php?id=81">(Previous articles on RhymeBrain)</a>
<h2>Other references</h2>
In his article <i><a href="http://norvig.com/spell-correct.html">How to write a spelling corrector</a></i>, Peter Norvig approaches the problem using a different way of thinking. He first stores his dictionary in a hash-table for fast lookup. Then he goes through hundreds, or even thousands of combinations of spelling mutations of the target word and checks if each one is in the dictionary. This system is clever, but breaks down quickly if you want to find words with an error greater than 1. Also, it would not work for me, since I needed to modify the cost functions for insert, delete, and substitution.
<p>
In the blog article <a href="http://fiber-space.de/wordpress/?p=1579">Fuzzy String Matching</a>, the author presents a recursive solution using memoization (caching). This is equivalent to flood-filling a diagonal band across the table. It gives a runtime of O(k * <number of nodes in the trie>), where k is the maximum cost. You can modify my algorithm above to only fill in only some entries of the table. I tried it, but it made the examples too complex and actually slowed it down. I blame my python skills.
<p>
<i>Update</i>: I just realized the author has created a <a href="http://fiber-space.de/wordpress/">new solution for dictionary search</a>, also based on tries. I quickly tried it on my machine and dictionary, and got a time of 0.009301, assuming the prefix tree is prebuilt. It's slightly faster for an edit distance of 1! But somethings going on, because it takes 1.5 s for an edit distance of 4, whereas my code takes only 0.44. <i>Phew!</i>
<p>
And of course, you could create a <a href="http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automata">levenshtein automaton</a>, essentially a <i>big honking regular expression</i> that matches all possible mispellings. But to do it efficiently you need to write <i>big honking gobs of code</i>. (The code in the linked article does not do it efficiently, but states that it is possible to do so.) Alternatively, you could enumerate all possible mispellings and <a href="?id=115">insert them into a MA-FSA or DAWG</a> to obtain the regular expression.
<ul>
<li><a href="http://stevehanov.ca/blog/index.php?id=71">Using a netbook as a webserver</a>
<li><a href="http://stevehanov.ca/blog/index.php?id=95">C++: A language for next-generation web apps</a>
</ul><ul><li><a href='?id=76'>Does Android team with eccentric geeks? (comic)</a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a><li><a href='?id=82'>The PenIsland Problem: Text-to-speech for domain names</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a></ul>
The Curious Complexity of Being Turned On
tag:smhanovtechblog,2007:id111
2010-11-29T10:26:01-05:00
2010-11-29T10:26:01-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
The imaginary Larmin Corp is designing the next killer product: A mood ring. Okay it's too big to wear around your finger and is more of a wrist device. But it works with 80% accuracy and it's got its own app store and it is expected to be a big hit at CES. There's a snag: unnamed sources are attributing the delay in the product launch to the "On/Off" problem. Larmin Corp denied all rumours and promptly launched lawsuits against the unnamed sources, their children, and pets, and the everyone at the bar that night.
<p>
Here's how the device works:
<ul>
<li> It is comprised of two parts: The mood detector, and the User Interface (UI)
<li> The user interface runs all the time (It uses your brain waves for energy)
<li> The mood detector can be turned on and off. However, interpreting your brain waves is complex business, so it can take several seconds to switch on or off.
</ul>
<p>
How do you turn on and off this system? One way is to ignore the delays and pretend it takes no time to turn on and off. The UI simply freezes up until the task is done.
<p align=center><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=VUktPk1vb2QgZGV0ZWN0b3I6IFR1cm5PbigpCmFjdGl2YXRlIAAUDQoAIg0tLT5VSToK&s=napkin"><br><a href="http://www.websequencediagrams.com/?lz=VUktPk1vb2QgZGV0ZWN0b3I6IFR1cm5PbigpCmFjdGl2YXRlIAAUDQoAIg0tLT5VSToK&s=napkin">Click to edit</a></p>
<p>
The problem is sometimes mood detector takes a little longer to turn on, and user's think it's crashed and exhibit extreme anger. Some even start banging the entire device on the desk.
<p>
So we don't freeze the UI during the turn-on procedure. But this leads to the following behaviour:
<p align=center><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=ClVJLT5Nb29kIGRldGVjdG9yOiBUdXJuT24oKQphY3RpdmF0ZSAAFA0AGhlmZigpCm5vdGUgb3ZlciAiAEoNIjogRVJST1IK&s=napkin"><br><a href="http://www.websequencediagrams.com/?lz=ClVJLT5Nb29kIGRldGVjdG9yOiBUdXJuT24oKQphY3RpdmF0ZSAAFA0AGhlmZigpCm5vdGUgb3ZlciAiAEoNIjogRVJST1IK&s=napkin">Click to edit</a></p>
<p>
As users get impatient waiting for it to turn on, they keep restarting the procedure. But if you try to turn off the detector while it is turning on, it crashes. The UI team first decides to handle this by adding another layer above the mood detector. If you send a command to it, and the mood detector is busy, it stores it in a queue for later. As soon as the mood detector completes, the layer replays the next queued action.
<p align=center><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=cGFydGljaXBhbnQgVUkKAAMMIkNvbW1hbmQgUXVldWUiIGFzAAQGClVJLT4ADwU6IFR1cm5PbigpICMxCgAiBS0-TW9vZCBkZXRlY3RvcgAYCgphY3RpdmF0ZSAAFA0Kbm90ZSBsZWZ0IG9mIFVJOiBUaGUgdXNlciBnZXRzIGltcGF0aWVudABlEWZmKCkgIzIAeBYzABgXNACBJxY1CgCBKQ0tAIFXCWRvbmUKZGUAgSUXAIFXGwCBFggAgUA2YW5ncnkuCg&s=napkin"><br><a href="http://www.websequencediagrams.com/?lz=cGFydGljaXBhbnQgVUkKAAMMIkNvbW1hbmQgUXVldWUiIGFzAAQGClVJLT4ADwU6IFR1cm5PbigpICMxCgAiBS0-TW9vZCBkZXRlY3RvcgAYCgphY3RpdmF0ZSAAFA0Kbm90ZSBsZWZ0IG9mIFVJOiBUaGUgdXNlciBnZXRzIGltcGF0aWVudABlEWZmKCkgIzIAeBYzABgXNACBJxY1CgCBKQ0tAIFXCWRvbmUKZGUAgSUXAIFXGwCBFggAgUA2YW5ncnkuCg&s=napkin">Click to edit</a></p>
<p>
The problem is the user gets impatient and starts repeatedly hitting the button, and the device eventually gets so many commands queued up that it just sits there, repeatedly turning on and off until the user slams it against the wall and the battery falls out. Also, if mood detector ever turns on while the user is angry, it screws up the detector's calibration. (In version 1, users are instructed to be in a neutral mood when activating the ring).
<p>
So the design architects bring out the big guns and propose a "OnOffManager". Instead of using a queue of commands, the OnOffManager remembers the last requested state and uses it.
<p align=center><img src="onoff3.png"></p>
<p>
This works pretty well, except that during the design phase the graphics designer gets fed up with the whole debate and and simply grays out the button with an ajax spinny thing, so that any further clicks are ignored during turn on. The OnOffManager code is left in, because it took six months to design, but it is never exercised. Everyone lives happily ever after.
<p>
Wait, scratch that. Shortly before release, someone writes a location aware app which periodically turns on the mood detector and sends its status to Facebook. Another group is working on the highly secretive "mood gestures" app, which turns off the mood detector if the user thinks a certain sequence of moods. It's not long before somebody complains about their mood ring randomly turning on and off all the time. After analysis, we see the following sequence:
<p>
<p align=center><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=cGFydGljaXBhbnQgVUkgYXMgdWkKAAkMIkFwcCAxIiBhcyBhcHAxAAoSMgAUCDIAKg5Pbi9PZmYgTWFuYWdlcgA8BW0ABgYAUg5Nb29kIGRldGVjdG8AIQYABQgKdWktPgAtBzogVHVybiBvbgoAPgctPgAsCAAVBk9uKCkKYWN0aXZhdGUAOQpub3RlIHJpZ2h0IG9mAF4JOiBPbgoAbAgtAFoLZG9uZQpkZQA2EmFwcDEAehNhcHAyAIETEWZmAIEMGWZmAHguZmYAgQ0aCg&s=napkin"></p>
<p>It's an easy fix. The On/Off manager is modified to keep a count of every app that wants it on. The mood detector is only turned on when the counter goes from 0 to 1, and off when the counter goes from 1 to 0. All other states are ignored.
<p align=center><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=cGFydGljaXBhbnQgVUkgYXMgdWkKAAkMIkFwcCAxIiBhcyBhcHAxAAoSMgAUCDIAKg5Pbi9PZmYgTWFuYWdlcgA8BW0ABgYAUg5Nb29kIGRldGVjdG8AIQYABQgKdWktPgAtBzogVHVybiBvbgpub3RlIG92ZXIARwg6IGNvdW50ZXI9MQoAWwctPgBJCAAyBk9uKCkKYWN0aXZhdGUAVgoARgVyaWdodCBvZgB7CTogT24KAIEJCC0Adwtkb25lCmRlADYSYXBwMQB8LjIKYXBwMgCBTRFmZgCBQR4K&s=napkin"></p>
Everything is great, until the charismatic C.E.O of Larmin Corp, George Jalopsky is giving a keynote speech. During the speech, his mood ring turns on without him realizing it. The video goes viral. Jalopsky flies back in a huff and holds a meeting of all of software development. <em>"You must fix this problem," he cries, waving his wrists around, projecting bright crimson onto the walls and the faces of the engineers. "When I turn it off, I want it to stay <i>off!</i>"</em>
<p>
All development is halted and design committees are formed. Soon, no meeting room at Larmin is available because they are all full of developers talking about the problem. Curious discussions like this one are overheard: "If I turn you on, but George turns you off, are you on or are you off?" This is followed by snickers.
<p>
And then someone proposes a solution: The mood ring will have a "soft off" function. You can turn it off, but it will still be allowed to turn on again by third party apps, unless you turn it <i>really off</i>.
Provisional patents are quickly filed for the software and the design of the power button itself. The software folks toss around the ideas of what <i>off</i>, and <i>really off</i> mean, and whether it makes sense for the ring to be <i>really turned on</i> *snicker*.
<p>
Eventually, they come up with the generalized solution. The On/Off manager shall have <i>two</i> counters. One is for a special class of apps that are designated as "System Apps". There's only one for now -- the on off button. But there could be a plurality in the future. The other counter works as before for third-party apps. If the system counter is 0, the mood detector is off and any other commands are ignored. However, if the system counter is non-zero, then the value of the app counter is used to determine if the mood detector should be on. This is illustrated in the following sequence diagram.
<p align=center><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=cGFydGljaXBhbnQgIlN5c3RlbSBBcHAgMSIgYXMgc3lzMQoAFg0AEgphcHAACRMyABQIMgAqDk9uL09mZiBNYW5hZ2VyAFgFbQAGBgBSDk1vb2QgZGV0ZWN0bwAhBgAFCAphcHAxLT4ALwc6IFR1cm4gb24Kbm90ZSBvdmVyAEkIOiBzeXM9MFxuYXBwPTEKYXBwMgAGMTIAIRZmZgBOIXN5cwB7KzEAgR4IAIIGBy0-AIF0CACBWwZPbigpCmFjdGl2YXRlAIIBCgCBbwVyaWdodCBvZgCCJgk6IE9uCgCCNAgtAIIgC2RvbmUKZGUANhIKCg&s=napkin"><br><a href="http://www.websequencediagrams.com/?lz=cGFydGljaXBhbnQgIlN5c3RlbSBBcHAgMSIgYXMgc3lzMQoAFg0AEgphcHAACRMyABQIMgAqDk9uL09mZiBNYW5hZ2VyAFgFbQAGBgBSDk1vb2QgZGV0ZWN0bwAhBgAFCAphcHAxLT4ALwc6IFR1cm4gb24Kbm90ZSBvdmVyAEkIOiBzeXM9MFxuYXBwPTEKYXBwMgAGMTIAIRZmZgBOIXN5cwB7KzEAgR4IAIIGBy0-AIF0CACBWwZPbigpCmFjdGl2YXRlAIIBCgCBbwVyaWdodCBvZgCCJgk6IE9uCgCCNAgtAIIgC2RvbmUKZGUANhIKCg&s=napkin">Click to edit</a></p>
<p>
And that, folks, is how something like turning the system on and off can grow in complexity very quickly. Soon, Larmin Corp will add low power modes and the special "BlueMood" peripheral, which transmits the moods to other users, but due to brain wave interference patterns, it only works with the mood detector is off even though the user has buttons for both independently.<p>
Come back next time, to read about how the moods are sent from the detector to the display, in "States of confusion".
<ul><li><a href='?id=98'>Asking users for steps to reproduce bugs, and other dumb ideas</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=117'>Why don't web browsers do this?</a></ul>
Cross-domain communication the HTML5 way
tag:smhanovtechblog,2007:id109
2010-11-25T12:30:12-05:00
2010-11-25T12:30:12-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Making a web application mashable -- useable in another web page -- has some challenges in the area of cross-domain communications. Here is how I solved those problems for Zwibbler.com. (See the <a href="http://zwibbler.com/api.html">API demo here</a>)
<p>
Zwibbler consists of a large javascript program and a little HTML. The javascript part uses Ajax methods to send POST requests back to the zwibbler.com server, to render some items without the limitations of the CANVAS tag. In particular, this allows it to support PDF output as well as SVG and PNG.
<p>
If you want to include Zwibbler.com on another web site, the zwibbler application still needs to communicate with zwibbler.com in order to perform these tasks. However, browsers will not allow this due to security restrictions. Javascript code can only communicate with the server that the main web page came from.
<p>
HTML allows you to embed one web page inside another, in the <iframe> element. They remain essentially separated. The container web site is only allowed to talk to its web server, and the iframe is only allowed to talk to its originating server. Furthermore, because they have different origins, <em>the browser disallows any contact between the two frames</em>. That includes function calls, and variable accesses.
<p align=center><img src="http://zwibbler.com/shared/774.png"></p>
But what if you want to get some data in between the two separate windows? For example, a zwibbler document might be a megabyte long when converted to a string. I want the containing web page to be able to get a copy of that string when it wants, so it can save it. Also, it should be able to access the saved PDF, PNG, or SVG image that the user produces.
HTML5 provides a restricted way to communicate between different frames of the same window, called window.postMessage(). The postMessage function takes two parameters:
<ul>
<li>A string to pass
<li>The target's origin, or "*" to allow any origin.
</ul>
<p>
For example, to pass a message from the container web page to the iframe, we use:
<pre>iframe.contentWindow.postMessage("hello there", "http://zwibbler.com");
</pre>
The receiver of the message must have previously registered for an HTML event named "message". This event arrives via the same mechanism as mouse clicks.
<pre>
window.addEventListener("message", function( event ) {
if ( event.data === "hello there" ) {
// event.origin contains the host of the sending window.
alert("Why, hello to you too, " + event.origin);
}
}, false );
</pre>
<h2>Problem 1: Two way communication</h2>
This method of communication is one way, but for a method call, we have to allow two way communication. We add a simple wrapper on top, called a Messenger, to allow two way communication. Each time you call a method in the iframe, you pass a reply function that is called with the results of that method call. We use JSON for the parameter marshalling.
<p>
The Messenger object must also keep track of how to direct the replies it receives. It assigns each request a unique <i>ticket</i>, and stores them in a table along with the reply function. When a reply with a matching ticket is recieved, the corresponding function is called:
<pre>
Messenger.prototype = {
init: function( targetFrame, targetDomain) {
// The DOM node of the target iframe.
this.targetFrame = targetFrame;
// The domain, including http:// of the target iframe.
this.targetDomain = targetDomain;
// A map from ticket number strings to functions awaiting replies.
this.replies = {};
this.nextTicket = 0;
var self = this;
window.addEventListener("message", function(e) {
self.receive(e);
}, false );
},
send: function( functionName, args, replyFn ) {
var ticket = "ticket_" + (this.nextTicket++);
var text = JSON.stringify( {
"function": functionName,
"args": args,
"ticket": ticket
});
if ( replyFn ) {
this.replies[ticket] = replyFn;
}
this.targetFrame.postMessage( text, this.targetDomain );
},
</pre>
The receive function first checks the origin of the message. If it is not the one that we expected, then we ignore the message. Maybe it's from another iframe, such as an ad or a game that happens to be on the same page. It then checks to see if it has a ticket number. If so, it decodes the arguments and calls the associated reply function.
<pre>
receive: function( e ) {
if ( e.origin !== this.targetDomain ) {
// not for us: ignore.
return;
}
var json;
try {
json = JSON.parse( e.data );
} catch(except) {
alert( "Syntax error in response from " + e.origin + ": " + e.data );
return;
}
if ( !(json["ticket"] in this.replies ) ) {
// no reply ticket.
return;
}
var replyFn = this.replies[json["ticket"]];
delete this.replies[json["ticket"]];
var args = [];
if ( "args" in json ) {
args = json["args"];
}
replyFn.apply( undefined, args );
},
</pre>
<h2>Problem 2: Delayed loading</h2>
There is one other complexity to handle. When we load the iframe, it takes some time to initialize before it is ready to receive events. If you send a message before it has registered to receive it, I'm not sure what happens, but it didn't work when I tried it.
<p>
So we have to add a bit of logic to the above code. When the iframe completes initializing, it sends a message consisting of the text "ready" to its parent window. If the Messenger is asked to send a message, and it has not yet received the "ready" message, then instead of sending it, it adds it to a queue for later. When it finally receives the ready message, it loops through the queue and finally sends all of the waiting messages to the iframe.
<p>
The complete code is contained in <a href="http://zwibbler.com/component.js">component.js</a>
<ul><li><a href='?id=57'>Coding tips they don't teach you in school</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=98'>Asking users for steps to reproduce bugs, and other dumb ideas</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a></ul>
Five essential steps to prepare for your next programming interview
tag:smhanovtechblog,2007:id106
2010-09-27T18:00:00-05:00
2010-09-27T18:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>There are at least two kinds of programming interviews. One type is where you are asked for details about your prior work experience. The other one is where they put you in a room, give you a problem, and stare at you while you fumble around with markers on a whiteboard for 45 minutes. The first focuses on what you have done in the past. The second focuses on what you can do in the room <i>right now</i> without looking anything up. You should be prepared for either.
<h2>Step 1: Get your stories straight</h2>
You will spend a large chunk of time in a job interview talking about things that you have done in the past. If haven't figured out a half dozen stories that best represent your skills, then you need to do that now. Here is a list of questions from a standard list. Many of them are stupid, but trust me -- they force you to think about yourself. Even if you aren't asked a question identical to one on this list, <i>you will use your prepared answers during an interview.</i> The point of this exercise is to build a repertoire of examples from your work life that you can use to answer questions.
<p>
<ol>
<li>Tell me about yourself
<li>What are your short-term goals? What about in 2 and 5 years from now?
<li>What is your own vision/mission statement?
<li>What do you think you will be looking for in the job following this position?
<li>Why do you feel you will be successful in this work?
<li>What other types of work are you looking for in addition to this role?
<li>What supervisory or leadership roles have you had?
<li>What experience have you had working on a team?
<li>What have been your most satisfying/disappointing experiences?
<li>What are your strengths/weaknesses?
<li>What kinds of problems do you handle the best?
<li>How do you reduce stress and try to achieve balance in your life?
<li>How did you handle a request to do something contrary to your moral code or business ethics?
<li>What was the result the last time you tried to sell your idea to others?
<li>Why did you apply to our organization and what do you know about us?
<li>What do you think are advantages/disadvantages of joining our organization?
<li>What is the most important thing you are looking for in an employer?
<li>What were some of the common characteristics of your past supervisors?
<li>What characteristics do you think a person would need to have to work effectively in our company with its policies of staying ahead of the competition?
<li>What courses did you like best/least? Why?
<li>What did you learn or gain from your part-time/summer/co-op/internship experiences?
<li>What are your plans for further studies?
<li>Why are your grades low?
<li>How do you spend your spare time?
<li>If I asked your friends to describe you, what do you think they would say?
<li>What frustrates you the most?
<li>When were you last angry at work and what was the outcome?
<li>What things could you do to increase your overall effectiveness?
<li>What was the toughest decision you had to make in the last year? Why was it difficult?
<li>Why haven't you found a job yet?
<li>You don't seem to have any experience in ___ (e.g., sales, fundraising, bookkeeping), do you?
<li>Why should I hire you?
</ol>
<i><a style="font-size: xx-small" href="http://www.cdm.uwaterloo.ca/step4_4_1.asp">Source: The University of Waterloo Career Development Manual</a></i>
<p>
The problem is that they require deep thought and introspection to answer, so it�s important to do that thinking in advance. Take an hour and think about the answers to these questions (you can use the same answer for more than one). For questions where you need to tell a story, your answer should follow this format:
<ol>
<li>20 seconds: Describe the situation. "The code was crashing and the whole team had to stop and figure out why."
<li>30 seconds: Describe what you did "I thought of doing a memory dump, and I noticed that the AbstractMemberCreationFactory had a lot of instances but it was supposed to be a singleton."
<li>20 seconds: Describe the results. "I fixed the memory leak with one line of code and we shipped the product on time. Later on, I added a test to make sure this wouldn't happen again."
</ol>
<p>
<em>Before each interview, go through the entire list and practice your answers out loud.</em> Doing this will give you an edge over the other candidates, because it will make you more comfortable during the interview. When asked a question, other candidates will be staring at the ceiling saying "ummm", trying to remember everything that happened to them in the the past five years. Meanwhile, you'll smile, look the interviewer in the eye, and launch into your story.
<h2>Step 2: Build confidence by solving the most common programming exercises beforehand</h2>
Pianists have to learn a specific set of short pieces before they advance to the next level. These tunes will never be a hit at parties, but they exercise particular things, such as the <a href="http://www.sheetmusic2print.com/Turk/Angehende-Klavierspieler/Dedicated-Most-Humbly.aspx">right hand little finger</a>, or syncopation. Likewise, certain problems keep coming up in programming interviews, although you will probably never, ever use them in your code. You will probably be asked one of the these time worn classics.
<ul>
<li>Reverse a singly linked list (in one pass through the list)
<li>Reverse a string (in one pass). Reverse the order of words in a paragraph (in two passes)
<li>Draw a circle of arbitrary size by printing out "*" characters. (hint: calculating whether to go "one down, two over" is the wrong approach)
<li>Convert an integer to a string. Convert a string to an integer. (Manually, of course, by looping through each digit somehow.)
<li>Write a function to return the number of 1's in the binary representation of an integer.
<li>Write a function that will display all possible arrangements of letters in a string. Example: abc acb bac bca cab cba
</ul>
Always start with the easiest solution that works, without considering the runtime. Then, try to make it faster. It's better to have something that works than spend all your time trying to optimize and end up with a page full of scribbles.
<div style="margin-top: 1em; background: #c0c0c0; border: 2px solid black; padding: 0.5em">
<h3>Don't cheat yourself by looking up the answers</h3>
The first time I tried to reverse a singly-linked list, it was between classes at school. I wasn't rushing, and it took me over half an hour to go from the slow and obvious solution to the fast one. But when I verified that my answer was correct, I was thrilled! I knew that I could tackle this question without looking up the answer. During interviews, when I was given a problem that I hadn't seen before, that experience gave me the confidence I needed to avoid blanking and keep trying.
</div>
<h2>Step 3: Practice your problem-solving</h2>
<p>
Some interviewers believe that being able to solve brain-teasers equates to good programming ability. In case you get one of these, you should develop a passing interest in puzzles and techniques for solving them. A visit to your local library will result in a dozen books, filled with puzzles to practice. Pick some interesting problems to tackle, and resist looking up the answers until you have spent at least a half hour on each one. Before long you'll have no difficulty helping foxes & ducks cross rivers, or figuring out to escape locked rooms by burning various lengths of rope.
<h2>Step 4: Show genuine enthusiasm</h2>
<p>
A powerful technique is to show real enthusiasm. As human beings, we can't help responding in kind and becoming excited to work with you. On the other hand, we also have evolved the ability to see through fake smiles, so it's vital that you be genuinely yourself.
<p>
The best interviewers will try to get you to talk about something that you are passionate about, even if it doesn't directly relate to the job. Most interviewers, however, will not. You will have to think about something that you've done that excites you, and look for opportunities to talk about it. Do this early in the interview. After the first 10 minutes it is probably too late, since the interviewers will have already ranked you.
<p>
Picture yourself coming in to work at this new job on the first day, turning on the new quad-core development workstation, meeting some interesting new friends, and learning about life at the company. There's got to be something exciting about that. Otherwise, why are you applying?
<h2>Step 5: Sleep</h2>
The "Tip of the tongue" phenomenon -- the inability to recall names, words, and facts -- increases dramatically if you have a sleep debt. Don't be caught struggling to remember an important detail during an interview. Instead, get a good night's sleep (<a href="http://www.cdc.gov/sleep/how_much_sleep.htm">7-9 hours</a>).
<h3>Further Reading</h3>
<ul>
<li><a href="?id=56">How a programmer reads your resume</a>
<li><a href="?id=105">Finding awesome developers in programming interviews</a>
<li><a href="?id=67">Game theory, salary negotiation, and Programmers</a>
<li><a href="?id=96">Bending over: How to sell your software to large companies</a>
<li><a href="http://careers.stackoverflow.com/stevehanov">The Completely Honest Resume</a>
</ul>
<ul><li><a href='?id=77'>Pitching to VCs #2 (comic)</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=83'>Sign here (comic)</a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=13'>Draw waveforms and hear them</a></ul>
Finding awesome developers in programming interviews
tag:smhanovtechblog,2007:id105
2010-09-13T18:00:00-05:00
2010-09-13T18:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
In a job interview, I once asked a very experienced embedded software developer to write a program that reverses a string and prints it on the screen. He struggled with this basic task. This man was awesome. Give him a bucket of spare parts, and he could build a robot and program it to navigate around the room. He had worked on satellites that are now in <i>actual orbit.</i> He could have coded circles around me. But the one thing that he had never, ever needed to do was: display something on the screen.
<p>
Some people have a knack for asking the right questions to spot awesome developers in a job interview. <em>Other interviewers dread it, come in with their tail between their legs, ask a few questions from the Internet and just go along with the group decision.</em> But interviewing is an essential skill for most developers. A bad hire has terrible long term consequences, because eventually a sub-par employee may bring others into the organization. On the other hand, unfairly excluding an awesome candidate also hurts.
<p>
A programming interview includes at least three parts. In part I, we prove any assumptions we have after reading the resume. In part II, we get a sense for how much <b>true experience</b> the candidate has. Finally, we test this experience using a few spot checks and a coding question.
<h2>Part I: Testing assumptions from the resume</h2>
Once I was intervewing a candidate along with a fellow co-worker. When it was done, I thought the candidate had done okay, but not brilliantly. My co-worker, on the other hand, seemed angry. "He lied about technology X. He obviously has not worked with it. Definately a no-hire." Technology X was not even important to us. <em><i>"If he lied about that,"</i> my co-worker went on, <i>"I don't trust anything else on the resume."</i></em>
<p>
Candidates <i>should</i> use the resume to portray themselves in a positive light. (See <a href="http://careers.stackoverflow.com/stevehanov">The Completely Honest Resume</a>). However, there is a line where this positive portrayal becomes misrepresentation. In the example above, I wasn't as concerned as my colleague, because I already assume that <em>everything on the resume is false until proven otherwise</em>. If the resume says, "expert in technology X", then I will assume that the candidate merely knows the <i>name</i> of technology X. If the resume says, "Worked in a group that created a multi-threaded stock trading platform," then I will assume that the candidate merely chose the colours for the background. I used to be less strict until I met the guy with 10 years of experience who couldn't write code. If someone says that they wrote the text formatter in OpenOffice, or has a Ph.D, it is easy to make assumptions about their skills. <b>Assume nothing. All must be tested.</b>
<p>
For each relevant item on the resume, I first try to get a sense of what the candidate actually did. Then, I get him or her to prove it by talking about it.
<ul>
<li>Created a real-time operating system as a course project. <br>
<i>How large a group did you work in? A group of 15? Oh, okay then, what specific part did you work on? The message queue? Great! Can you describe what happens when a high priority task sends a message to a low priority task?
</i>
<li>Developed from scratch an audio transfer protocol for wireless security systems.<br>
<i>How large was your team? Just you? Wow, how did you test it? Why didn’t you use RTP?</i>
<li>Fixed bugs in the XYZEngine.<br>
<i>Can you describe a bug that you found particularly challenging, and how you fixed it?</i>
</ul>
<h2>Part II: Finding <b>true experience</b></h2>
<p>
Having more experience is a good indication of awesomeness. Experienced developers have made mistakes. They know when, and when not to apply design patterns. They have a sixth sense about what part of requirements will probably change, and what part will probably stay the same. They know when to be lazy and when to be pedantic. It is <i>true</i> experience which makes the gap between awesome and mediocre programmers so wide.
<p align=center><img src="http://zwibbler.com/shared/416.png"></p>
<p>
<em>But not all experience is the same.</em> It is certainly possible for someone to gain solid skills in a couple of years, simply by working on lots of different things, writing and rewriting countless lines of code, and making many mistakes. On the other hand, it is also possible for someone to spend a decade writing one-line changes to a single project, without learning anything new.
<p align=center><img src="http://zwibbler.com/shared/417.png"></p>
<h3>Finding hidden time</h3>
There are lots of great developers who started coding when they were in their second year of university. By the time they get out of school, they will have had a few years of experience. On the other hand, <em>some awesome developers started learning their art at an early age.</em> I know several people who wrote some non-trivial programs in their teens or earlier. This information is nowhere to be found on their resume, and must be coaxed out during an interview.
<ul>
<li>Why did you get into the software development field?
<li>What's the first programming language that you ever learned?
</ul>
<p>
<h3>Density of Experience</h3>
<p>
Many awesome programmers do all of their coding at work. These are great, well, rounded individuals that you should definitely hire. However, doing personal programming projects outside of work or class is a pretty good indicator of awesomeness. A candidate with personal programming time simply has more flight time under his or her belt, and will be better for it. No personal projects? These other indicators will also count for some points:
<ul>
<li>Working on smaller teams or groups.
<li>Working on a wide variety of projects
<li>Detailed knowledge of several layers of abstraction on a large project
<li>Being the main contributor in a group project
</ul>
<h2>Part III: Verifying experience</h2>
<p>
After gaining a sense of the candidate's true level of experience, it is important to verify that experience testing their programming abilities. A few minutes of time is completely inadequate for a true test, but that's all that's available. We can get an idea of the breadth and depth of knowledge of the candidate by asking questions about different areas of software development. Of course your perception of the candidate's skills will be biased by your own experiences. You cannot judge the correctness of answers in topics that are unfamiliar to you. That's why there are several interviewers.
<p>
The specific topics depend on the job requirements. Nevertheless, some example areas are:
<ul>
<li>data structures and algorithms
<li>multithreading
<li>bit manipulation
<li>memory allocation
<li>objects and inheritance, design patterns
<li>recursion
<li>compilation and how computers run programs
</ul>
<p>
Each area that I choose has a selection of basic questions (“What’s a semaphore?”). These are so basic that if the candidate has done any work at all in the area he or she would be able to answer. Each area also has some more detailed follow-up questions. The way in which a candidate answers can prove or disprove awesomeness. For example something is amiss if you ask a seasoned embedded programmer to convert 0x4c to binary, and they start by writing down 4 x 16 + 12.
<h3>The Coding Question</h3>
<p>
Usually, after all of the above, I have a very good idea whether the candidate will pass or fail, but the coding question removes all doubt. It is so important, that even phone interviews are not exempt. To be useful, a coding question requires careful thought and planning before the interview. Asked the wrong way, the response will be useless.
<p>
First, one must choose a question based on what the candidate has had experience with. You may have a clever problem that becomes easy if you think of converting everything to intersecting 3D planes. Save it for the lunch hour with your colleagues. If the job does not involved 3D graphics, candidates would be unfairly excluded.
<p>
The question must be precisely worded. "Write a function to shuffle a deck of cards" is woefully ambiguous. Provide the function header and avoid misunderstandings, which are all too common. If you are not careful, the candidate will answer a harder or easier problem than the one you asked. The harder one is nice, unless it causes him or her to freeze up. The easier one provides no information. To prevent a huge waste of time, ask for a verbal outline of the solution after a few minutes, to check if the candidate is on the right track.
<h3>The influence of the order of questions</h3>
<p>
The order in which you ask questions can profoundly influence the thought processes of the candidate. For example, I used to ask question about hash tables when I thought the candidate knew about them. Later on in the interview, I would ask a coding question that had nothing to do with hash tables. Candidates would invariably decide to use a hash table in their implementation, with the keys being unique, consecutive integers starting at 0. If I avoided talking about hash tables, the candidates would instead choose to use an array.
<p>
Candidates are also strongly influenced by you in their choice of language. For example, if you say the job primarily involves Java, every candidate will swear that, by golly, Java is his best and favourite language to work in. He will choose to use it for all coding questions, realizing too late that he can't remember how to declare variables in the language he is "best" at.
<h2>Avoid language bias</h2>
It's terribly easy to be biased toward a specific programming language that you use at your company. By fixating on a particular tool, you throw away a lot of awesome developers. Do not try to determine if the candidate awesome at programming in C or Java or whatever. <em>Instead, you should be trying to find out if the candidate awesome at programming in the language that he or she knows best.</em>
<p>
<h2>Going further</h2>
<p>
The guidelines above <i>do not address everything</i>. They focus on experience, and <em>they might miss awesome developers that have little experience, but a lot of innate ability.</em> In particular, interviewers may want to test problem solving ability using puzzles that don't require any coding.
<p>
The interviewing technique that I have described here is based on proving a hypothesis, probability, and gut instinct. The hypothesis is that the candidate is an awesome developer. What traits does an awesome developer have? You cannot directly measure those traits, so you have to instead ask: What is the probability that the candidate has those traits given that he or she can answer a particular question quickly? It is not possible to assess a candidate within an interview with 100% success, but by asking thoughtful questions, you can come close.
<h2>Further Reading</h2>
<ul>
<li><a href="?id=56">How a programmer reads your resume</a>
<li><a href="?id=67">Game Theory, Salary Negotiation, and Programmers</a>
</ul>
<ul><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=8'>A Rhyming Engine</a><li><a href='?id=101'>"Your program is stupid. It doesn't work," my wife told me</a><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a><li><a href='?id=78'>It's a dirty job... (comic)</a></ul>
Compress your JSON with automatic type extraction
tag:smhanovtechblog,2007:id104
2010-08-16T21:00:47-05:00
2010-08-16T21:00:47-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<a href="http://www.json.org">JSON</a> is horribly inefficient data format for data exchange between a web server and a browser. For one, it converts everything to text. The value 3.141592653589793 takes <a href="http://www.hunlock.com/blogs/The_Complete_Javascript_Number_Reference">only 8 bytes of memory</a>, but JSON.stringify() expands it to 17. A second problem is its excessive use of quotes, which add two bytes to every string. Thirdly, it has no standard format for using a schema. When multiple objects are serialized in the same message, the key names for each property must be repeated, even though they are the same for each object.
<p>
JSON <i>used to</i> have an advantage because it could be directly parsed by a javascript engine, but even that advantage is gone because of security and interoperability concerns. About the only thing JSON going for it is that it is <a href="http://en.wikipedia.org/wiki/JSON#XML">usually more compact</a> than the alternative, XML, and it is well supported by many web programming languages.
<p>
Compression of JSON data is useful when large data structures must be transmitted from the web browser to the server. <b>In that direction, it is not possible to use gzip compression, because it is not possible for the browser to know in advance whether the server supports gzip.</b> The browser must be conservative, because the server may have changed abilities between requests.
<p>
Today, let's tackle the most pressing problem: the need to constantly repeat key names over and over. I will present a Javascript library for compressing JSON strings by automatically deriving a schema from multiple objects. The library can be used as a drop in replacement for the methods JSON.stringify() and JSON.parse(), except that it lacks support for a reviver function. In combination with <a href="http://mjtemplate.org/examples/rison.html">Rison</a>, the savings could be significant.
<p align="center">
<a href="cjson.js">Download it here</a>
<p>
Suppose you have to transmit several thousand points and rectangles. JSON might encode them like this (without the comments):
<pre>
[
{ // This is a point
"x": 100,
"y": 100
},
{ // This is a rectangle
"x": 100,
"y": 100,
"width": 200,
"height": 150
},
{}, // an empty object
... // thousands more
]
</pre>
<p>
A lot of the space is taken up by repeating the key names "x", "y", "width", and "height". They only need to be stored once for each object type:
<p>
<pre>
{
"templates": [ ["x", "y"], ["x", "y", "width", "height"] ],
"values": [
{ "type": 1, "values": [ 100, 100 ] }, { "type": 2, "values": [100, 100, 200, 150 ] }, {} ]
}
</pre>
<p>
Each object in the original input is transformed. Instead of listing the keys, the "type" field refers to a list of keys in the schema array. (The type is 1-based, instead of zero based, and I will explain why later). But we are still repeating "x", and "y". The rectangle shared these properties with the point type, and there is no need to repeat them in the schema:
<p>
<pre>
{
"templates": [ [0, "x", "y"], [1, "width", "height"] ],
"values": [
{ "type": 1, "values": [ 100, 100 ] }, { "type": 2, "values": [100, 100, 200, 150 ] }, {} ]
}
</pre>
We prefix each key list in the schema with a number. This number is the one-based index of a prior schema which is prepended to it to form the combined list. Zero means the empty object, which is why we use one-based indicies.
<p>
But we can still go a little further. Instead of having a separate "type" field in each object, we stick the type as the first element of the values array.
<pre>
{
"templates": [ [0, "x", "y"], [1, "width", "height"] ],
"values": [
{ "values": [ 1, 100, 100 ] }, { "values": [2, 100, 100, 200, 150 ] }, {} ]
}
</pre>
<p>
Finally, since we are trying to save space, we rename our properties, and stick in a format code so we can detect that compresed json is used.
<p>
<pre>
{
"f": "cjson",
"t": [ [0, "x", "y"], [1, "width", "height"] ],
"v": [ { "": [1, 100, 100 ] }, { "": [2, 100, 100, 200, 150 ] }, {} ]
}
</pre>
<h2>Automatic type extraction</h2>
<p>
The hard part is finding the objects which share sets of keys. It sounds a lot like the <a href="http://en.wikipedia.org/wiki/Set_cover_problem">Set Cover problem</a>, and if so, an optimal solution is NP-complete. Instead, we will approximate the solution using a tree structure. While we are building the value array, when we encounter an object, we add all of its keys to the tree in the order that we encounter them.
<p align=center>
<a href="http://zwibbler.com"><img src="http://zwibbler.com/shared/369.png"></a>
<p>
At the end of the process, we can traverse the nodes of the tree and create the templates. Nodes which represent the end of a key list (shown in gray) must have entry in the key list. Although not illustrated here, nodes with multiple children are also points where the the child object types inherit from a common parent, so they also get an entry.
<p>
The astute reader will realize that the final schema depends on the order that we inserted the keys into the tree. For example, if, when we encountered the rectangle, we inserted the keys "width" and "height" before "x", and "y", the algorithm would not find any common entries.
<p>
It is possible to gain more efficient packing by using a greedy algorithm. In the greedy algorithm, before we begin, an initial pass through all the objects would be made to build a list of unique object types. Then when it comes time to insert keys into the tree, they are first sorted so that the ones which occur in the most unique types are inserted first. However, this method adds a lot of extra processing and I feel the gains would not be worthwhile.
<p>
<h2>Real world savings</h2>
Here is an actual document from my web site, <a href="http://zwibbler.com">Zwibbler.com</a>. Click on "Transform" to see how CJSON compresses it vs. JSON.
<p>
<script type="text/javascript" src="cjson.js"></script>
<input type="button" value="Transform" onclick="transform()"><br>
<textarea id="input" rows=10 cols=50>
{"type":"Node","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":30,"dy":415},"children":[],"fillStyle":"#0089ff","text":"Created with Zwibbler.com","fontName":"Arial","fontSize":12},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":467,"dy":23},"children":[],"fillStyle":"#000000","text":"is processed","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":419,"dy":1},"children":[],"fillStyle":"#000000","text":"After the empty object","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":248,"dy":24},"children":[],"fillStyle":"#000000","text":"is processed","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":243,"dy":1},"children":[],"fillStyle":"#000000","text":"After a Rect","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":30,"dy":28},"children":[],"fillStyle":"#000000","text":"is processed","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":25,"dy":5},"children":[],"fillStyle":"#000000","text":"After a Point","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":33,"dy":358},"children":[],"fillStyle":"#000000","text":"object.","fontName":"FG Virgil","fontSize":15},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":32,"dy":341},"children":[],"fillStyle":"#000000","text":"the path to a completed","fontName":"FG Virgil","fontSize":15},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":32,"dy":323},"children":[],"fillStyle":"#000000","text":"Gray nodes represent","fontName":"FG Virgil","fontSize":15},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":504,"dy":375},"children":[],"fillStyle":"#000000","text":"height","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":487,"dy":295},"children":[],"fillStyle":"#000000","text":"width","fontName":"FG Virgil","fontSize":20},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":296,"dy":-5},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":246,"startY":273,"closed":false,"segments":[{"type":2,"x":231,"y":296}],"shadow":false}},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":-1.0052223548588493,"m12":0,"m21":0,"m22":0.9694181823431851,"dx":616.8428551273748,"dy":154.213227992421},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":103,"startY":192,"closed":false,"segments":[{"type":2,"x":89,"y":225}],"shadow":false}},{"type":"PathNode","matrix":{"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948628471,"dy":371.5686591257294},"children":[],"strokeStyle":"#000000","fillStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"PathNode","matrix":{"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.1381032855618,"dy":133.20628077515605},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":126.25,"startY":127.50445838342671,"closed":true,"segments":[{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127.50445838342671,"r":[-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.053223272785544395]},{"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.6827937204955,"r":[-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},{"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":[0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.1330692209303379]},{"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.50445838342671,"r":[0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}],"shadow":false},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":[],"fillStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":[],"fillStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":103,"startY":192,"closed":false,"segments":[{"type":2,"x":89,"y":225}],"shadow":false}},{"type":"PathNode","matrix":{"m11":0.4841400176012311,"m12":0,"m21":0,"m22":0.48095238095238096,"dx":526.2705783334583,"dy":222.70238095238096},"children":[],"strokeStyle":"#000000","fillStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":76,"startY":111,"closed":false,"segments":[{"type":2,"x":92,"y":143}],"shadow":false}},{"type":"PathNode","matrix":{"m11":1.3368391918073623,"m12":0,"m21":0,"m22":1.3058369941348298,"dx":397.4634652643233,"dy":-29.230987805439646},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":126.25,"startY":127.50445838342671,"closed":true,"segments":[{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127.50445838342671,"r":[-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.053223272785544395]},{"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.6827937204955,"r":[-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},{"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":[0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.1330692209303379]},{"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.50445838342671,"r":[0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}],"shadow":false},{"type":"PathNode","matrix":{"m11":0.4841400176012311,"m12":0,"m21":0,"m22":0.48095238095238096,"dx":505.2705783334583,"dy":61.70238095238095},"children":[],"strokeStyle":"#000000","fillStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":290,"dy":373},"children":[],"fillStyle":"#000000","text":"height","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":273,"dy":293},"children":[],"fillStyle":"#000000","text":"width","fontName":"FG Virgil","fontSize":20},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":82,"dy":-7},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":246,"startY":273,"closed":false,"segments":[{"type":2,"x":231,"y":296}],"shadow":false}},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":-1.0052223548588493,"m12":0,"m21":0,"m22":0.9694181823431851,"dx":402.8428551273748,"dy":152.213227992421},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":103,"startY":192,"closed":false,"segments":[{"type":2,"x":89,"y":225}],"shadow":false}},{"type":"PathNode","matrix":{"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":351.52019486284706,"dy":369.5686591257294},"children":[],"strokeStyle":"#000000","fillStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"PathNode","matrix":{"m11":-1.4512941300747246,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":483.6192163567064,"dy":131.20628077515607},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":126.25,"startY":127.50445838342671,"closed":true,"segments":[{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127.50445838342671,"r":[-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.053223272785544395]},{"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.6827937204955,"r":[-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},{"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":[0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.1330692209303379]},{"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.50445838342671,"r":[0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}],"shadow":false},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":329,"dy":223},"children":[],"fillStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":345,"dy":142},"children":[],"fillStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":250,"dy":-5},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":103,"startY":192,"closed":false,"segments":[{"type":2,"x":89,"y":225}],"shadow":false}},{"type":"PathNode","matrix":{"m11":0.4841400176012311,"m12":0,"m21":0,"m22":0.48095238095238096,"dx":312.2705783334583,"dy":220.70238095238096},"children":[],"strokeStyle":"#000000","fillStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":250,"dy":-5},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":76,"startY":111,"closed":false,"segments":[{"type":2,"x":92,"y":143}],"shadow":false}},{"type":"PathNode","matrix":{"m11":1.3368391918073623,"m12":0,"m21":0,"m22":1.3058369941348298,"dx":183.46346526432328,"dy":-31.230987805439646},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":126.25,"startY":127.50445838342671,"closed":true,"segments":[{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127.50445838342671,"r":[-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.053223272785544395]},{"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.6827937204955,"r":[-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},{"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":[0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.1330692209303379]},{"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.50445838342671,"r":[0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}],"shadow":false},{"type":"PathNode","matrix":{"m11":0.4841400176012311,"m12":0,"m21":0,"m22":0.48095238095238096,"dx":291.2705783334583,"dy":59.70238095238095},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":96,"dy":228},"children":[],"fillStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},{"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":112,"dy":147},"children":[],"fillStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":17,"dy":0},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":103,"startY":192,"closed":false,"segments":[{"type":2,"x":89,"y":225}],"shadow":false}},{"type":"PathNode","matrix":{"m11":0.4841400176012311,"m12":0,"m21":0,"m22":0.48095238095238096,"dx":79.27057833345825,"dy":225.70238095238096},"children":[],"strokeStyle":"#000000","fillStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false},{"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":[],"arrowSize":10,"path":{"type":"PathNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":17,"dy":0},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":76,"startY":111,"closed":false,"segments":[{"type":2,"x":92,"y":143}],"shadow":false}},{"type":"PathNode","matrix":{"m11":1.3368391918073623,"m12":0,"m21":0,"m22":1.3058369941348298,"dx":-49.536534735676724,"dy":-26.230987805439646},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":126.25,"startY":127.50445838342671,"closed":true,"segments":[{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127.50445838342671,"r":[-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.053223272785544395]},{"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.6827937204955,"r":[-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},{"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":[0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.1330692209303379]},{"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.50445838342671,"r":[0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}],"shadow":false},{"type":"PathNode","matrix":{"m11":0.4841400176012311,"m12":0,"m21":0,"m22":0.48095238095238096,"dx":58.270578333458246,"dy":64.70238095238095},"children":[],"strokeStyle":"#000000","fillStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":[-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.07100312784314156]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":[0.24857700895518064,0.030472169630229473,0.49844827968627214,0.13260168116539717]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":[0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]},{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":[0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],"shadow":false}]}
</textarea>
<textarea id="output" rows=10 cols=50>
</textarea>
<pre id="results">
</pre>
<script type="text/javascript">
function transform()
{
var pre = document.getElementById("results");
var text = "";
try {
var json = JSON.parse( document.getElementById("input").value );
var cjson = CJSON.stringify( json );
document.getElementById("output").value = cjson;
text = "Compressed from " + document.getElementById("input").value.length + " to " + cjson.length + " bytes (" + (cjson.length / document.getElementById("input").value.length) + "X)";
} catch( e ) {
text = "" + e;
}
while( pre.childNodes.length ) {
pre.removeChild( pre.firstChild );
}
pre.appendChild( document.createTextNode(text));
}
</script>
<pre id="results">
</pre>
<h2>Download</h2>
<a href="cjson.js">Download the CJSON code here.</a>
<h3>Further reading</h3>
<ul>
<li><a href="http://stevehanov.ca/blog/index.php?id=93">Zwibbler: A simple drawing program using Javascript and Canvas</a>
<li><a href="http://stevehanov.ca/cs744_type_inference_project.pdf">Type Inference in a Dynamically Typed Language</a>: an implementation of the Cartesian Product algorithm
</ul>
<ul><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=55'>How wide should you make your web page?</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=31'>Free, Raw Stock Data</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a></ul>
"Your program is stupid. It doesn't work," my wife told me
tag:smhanovtechblog,2007:id101
2010-06-19T10:31:12-05:00
2010-06-19T10:31:12-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
The <a href="http://en.wikipedia.org/wiki/HSL_and_HSV">HSV</a> colour wheel, based on <a href="http://en.wikipedia.org/wiki/Barycentric_coordinates_(mathematics)">barycentric coordinates</a>, is my favourite colour selection device. It discourages picking <font color="#00ffff">unnatural</font> <font color="#ff0000">looking</font> <font color="#00ff00">saturated</font> <font color="#0000ff">colours</font>. Instead, it gives the realistic designer colours more space in the triangle. That's why I chose it for <a href="http://zwibbler.com">Zwibbler.com</a>, my online Javascript sketching application.
<p align=center><img src="wheel1.png"></p>
I was working on the drawing tool one evening and <a href="http://alicemommyblog.blogspot.com/">my dear wife</a> happened to start using it to draw stuff. I watched her and asked her to change the colour of what she had drawn to blue.
<p>
Naturally, she clicked on the blue outer portion of the colour wheel.
<p align=center><img src="wheel2.png"></p>
<i>"This is stupid. It doesn't work,"</i> she complained. I love that I can always count on her for honest feedback!
<p>
The colour wheel works, of course. It works exactly the same way as Inkscape and other graphics design software. Clicking on the ring sets the <i>hue</i>. But when the saturation is zero, the hue component doesn't matter, because the absence of colour is always gray.
<p>
On <a href="http://zwibbler.com">Zwibbler</a>, <em>this always happens, because the default colours are black and white.</em>
<p>
With one line of code, I made the colour wheel behave the way she expected, and eliminated a negative experience for first time users.
<p>
<b>(Update in response to criticism.)</b> The fix I made matches expectations of new users, as well as experienced designers. If the current colour has saturation level 0 (i.e. it's white/black/gray), and you click on the outer ring, then obviously your intention is to eventually increase the saturation. Otherwise what you are doing has no effect. The program sets the saturation component to 1.0 <i>only in that specific case</i>. Otherwise, it leaves your current position in the triangle alone, allowing you to rotate the hue value.
<h3>You might also want to read:</h3>
<ul>
<li><a href="?id=93">Zwibbler: A simple drawing program using Javascript and Canvas</a>
<li><a href="?id=62">Exploiting perceptual colour difference</a>
</ul>
<ul><li><a href='?id=76'>Does Android team with eccentric geeks? (comic)</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=8'>A Rhyming Engine</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=37'>Spoke.com scam</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a></ul>
The simple and obvious way to walk through a graph
tag:smhanovtechblog,2007:id99
2010-06-03T08:00:00-05:00
2010-06-03T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
At some point in your programming career you may have to go through <a href="?id=65">a graph of items</a> and process them all exactly once. If you keep following neighbours, the path might loop back on itself, so you need to keep track of which ones have been processed already.
<pre>
function process_node( node )
{
if ( node.processed ) {
return;
}
node.processed = true;
var i;
for( i = 0; i < node.neighbours.length; i++ ) {
process_node( node.neighbours[i] );
}
// code to process the node goes here. It is executed only
// once per node.
}
</pre>
<p>
The code works, but it only works once! The next time you try it, all the nodes will already be marked as processed, so nothing will happen.
<p>
Here's a neat trick that elegantly solves the problem. Instead of using a boolean flag in each node, use a generation count.
<pre>
var CurrentGeneration = 0;
function process_node( node )
{<font color="#C00000">
if ( node.generation == CurrentGeneration ) {
return;
}
node.generation = CurrentGeneration;
</font>
var i;
for( i = 0; i < node.neighbours.length; i++ ) {
process_node( node.neighbours[i] );
}
// code to process the node goes here. It is executed only
// once per node.
}
function process_all_nodes( root )
{
CurrentGeneration += 1;
process_node( root );
}
</pre>
<p>
It's simple and obvious right? So why didn't I think of it in eight years?
<ul><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=96'>Bending over: How to sell your software to large companies</a><li><a href='?id=57'>Coding tips they don't teach you in school</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=50'>Why Perforce is more scalable than Git</a><li><a href='?id=117'>Why don't web browsers do this?</a></ul>
Asking users for steps to reproduce bugs, and other dumb ideas
tag:smhanovtechblog,2007:id98
2010-05-27T21:00:00-05:00
2010-05-27T21:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>A common misconception about software development:
<ol>
<li>When a bug occurs, users will it into a tracking system with detailed information on how to reproduce it.
<li>A developer walks through the given steps to reproduce the issue, finds the problem, and submits a fix
</ol>
<p>
That's based on several bad assumptions. Most users will not bother to enter bugs into your system. It is an unselfish act of altruism to enter a bug report. The user knows that it could be <i>months</i>, or even <i>years</i> before the bug is fixed, but she needs to be finished with your app by 5pm today.
<p>
The second problem is that <i>many bugs are not reproducible</i>. Maybe the bug depends on something the user was doing, the fact that she always clicks on the okay button instead of pressing enter, or because she installed a printer driver that replaced part of the OS with a modified, out of date library. Maybe your application is the first thing she uses in the morning while the hard drive is still chugging and 99 other programs are trying to update themselves at the same time.
<p>
Even worse: sometimes the problem just goes away on its own.
<p>
It's tempting to <a href="http://www.openoffice.org/issues/show_bug.cgi?id=88318">dismiss bug reports that you cannot reproduce</a>. As a developer, you have enough work to do. The <i>least</i> they can do is tell you how to reproduce the problem then you'll have a chance at fixing it. <a href="http://linuxhaters.blogspot.com/2008/08/one-bug-report-to-rule-them-all.html">Thousands of open bugs are closed</a> or left to languish for years because they cannot be reproduced.
<p>
Refusing to fix a serious problem until you have reproducible steps is a cop-out excuse for lazy developers, who would rather get back to working on the physics engine for that secret <a href="http://www.eeggs.com/items/29841.html">flight simulator easter egg</a>. That excuse might work for non-commercial software, but in the commercial software business, it will lose customers and get you fired.
<h2>How to fix non-reproducible bugs</h2>
<h3>Use a logging system</h3>
<p>
The best way to fix non-reproducible issues is to have adequate logging in the first place. Whenever the user does something, selects a menu, clicks "cancel", or <i>inhales</i>, record that action somewhere. Keep the file small, so the old parts scroll away. Take care to scrub anything that would violate privacy. Then, when a problem occurs, you can ask the user to attach the log file to the problem report.
<p>
You can use a well designed log <a href="http://stevehanov.ca/blog/index.php?id=66#good">as input to a test framework</a>. You can then automatically reproduce the issue as many times as you need to test the fix, and ultimately make it part of your regression test suite.
<h3>Otherwise, fix by inspection</h3>
<p>
In 1981, Mark Weiser <a href="http://portal.acm.org/citation.cfm?id=800078.802557">studied how experts debug software</a>. The very best programmers create a mental slice of the program, so they only have to think about a few functions at a time. Weiser defined a program slice as the minimal program that still reproduces the problem. He developed an automatic method for finding the program slice to make debugging easier.
<p>
Unfortunately, thirty years later we are still doing things manually, and debugging requires lots of creative detective work. Use source control to ensure that you are looking at the same version of the software that your customer is using. Work backwards from the error message, keeping careful notes of the reverse call graph. If the <tt>X</tt> variable was set, what were the possible values of <tt>Y</tt>? Could this <tt>else</tt> clause have run? It's grueling work, and it takes <i>days</i>, but at the end of it, you will have a list of potential paths through the system that caused the error. And now you can methodically fix each one of them, without ever having reproduced the problem.
<p>
Sometimes, though, you will find that the problem logically could not have happened. In that case, you can either add more logging, or detect and recover, or do both.
<h3>Otherwise, detect and recover</h3>
<p>
Okay, so you had adequate logging. You've mentally traced through the source code for days. You've written additional tests for different theories and failed to reproduce the problem. The only way it could have happened is if a hole in the universe opened up and changed the laws of physics for a moment, or <a href="http://www.newscientist.com/blog/technology/2008/03/do-we-need-cosmic-ray-alerts-for.html">cosmic rays rewrote some register values</a>. You will just have to detect and recover.
<p>
If slow memory leak causing the application to crash after three weeks, make sure your application restarts itself every few hours. If your complicated data structure somehow gets into an inconsistent state, write a function to go through and fix it <i>after every single change</i>. You get the idea.
<p>
Detecting and recovering from a mysterious bug is sometimes the only way to turn a major show-stopper into a minor annoyance. It is not the final solution, but it is a way to give you more time to find the real cause.
<h3>Get to it</h3>
The next time you get a serious bug that you can't reproduce, don't close it. Fix it, as if your job depended on it.
<ul><li><a href='?id=91'>You don't need a project/solution to use the VC++ debugger</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=55'>How wide should you make your web page?</a><li><a href='?id=144'>Finding Bieber: On removing duplicates from a set of documents</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a></ul>
Creating portable binaries on Linux
tag:smhanovtechblog,2007:id97
2010-04-29T18:55:31-05:00
2010-04-29T18:55:31-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Distributing applications on Linux is hard. Sure, with modern package management, <i>installing</i> software is easy. But if you are <i>distributing</i> an application, you probably need one Windows version, plus umpteen different versions for Linux. In this article, we'll create a dummy application that targets the following operating systems, which are commonly used in business environments:
<ul>
<li>Windows Server 2003
<li>Windows XP
<li>Red Hat Enterprise Linux 3
<li>Red Hat Enterprise Linux 4
<li>Ubuntu 6.06.2
<li>Ubuntu 8.04
<li>Ubuntu 9.10
</ul>
<p>
As evidence that the problem is hard, try <a href="http://www.mozilla.com/firefox">downloading Firefox</a>. It fails to start on many of the above platforms, due to missing libraries.
<h2>The sample application</h2>
The sample application is called plookup (download <a href="plookup.cpp">source</a> and <a href="plookup.zip">all binaries</a>). It runs from the command line and takes a hostname, looks it up and prints out the IP address. Ignoring the security flaws, it has several monkey wrenches thrown in that make it hard to port to different versions of linux:
<ul>
<li>It uses C++, which causes headaches when dynamically linking
<li>It uses socket functions, which cause migraines when statically linking
</ul>
<h2>Will distributing source code solve the problem?</h2>
<p>
In theory, distributing source code seems to be an easy way to get around the problem, assuming your end user 1) has administrative access, 2) can install a compiler 2) knows how to run a configure script, 3) has the technical knowledge to interpret the output of a configure script and download the appropriate dependencies, 4) has technical expertise to resolve conflicts in library versions.
<p>
In other words, it's a completely unreasonable solution for software written for normal human beings.
<h2>Building on Windows</h2>
<p>
With the appropriate build flags, you can produce software on Windows 7 that will run on all versions of windows since Windows 95. <a href="http://blogs.msdn.com/oldnewthing/archive/2007/04/11/2079137.aspx">By defining WINVER and some other macros,</a> you'll be warned at compile time if you're using a feature that will break your program on earlier versions.
<pre>
CL /EHsc /Feplookup.exe /DWINVER=0x0400 /D_WIN32 /D_WIN32_WINDOWS=0 /D_WIN32_IE=0x0400 wsock32.lib *.cpp
</pre>
We're done with Windows. On to Linux!
<h2>Static linking on Linux FAIL</h2>
<p>
Static linking has gained an undeserved reputation for being portable. But see what happens when you try to create a statically linked version of plookup:
<pre style="background:black;color:white;font:Lucida Console;white-space:pre-line">
steve@ubuntu:~/plookup$ g++ -static -static-libgcc -o plookup main.cpp
/tmp/ccMhUffR.o: In function `hostlookup(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)':
main.cpp:(.text+0x15): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
</pre>
<p>
It defeats the purpose. The warning isn't kidding, either. Your app might run fine for months, then all it takes is one update and it will crash. Even if you didn't use any socket functions, <a href="http://www.trilithium.com/johan/2005/06/static-libstdc/">you might have other problems</a>.
<p>
Note: If you statically link using the GNU compiler, then according to the L-GPL you have to also distribute your object files so the end-user can possibly re-link them to another version of the C libraries that they could have modified. Because your customers <i>love</i> hacking on <tt>strcat</tt> in their spare time.
<h2>Dynamic Linking</h2>
<p>
I built and tested the sample application on many different platforms. This table summarizes the results of the experiment.
<table border=1>
<tr>
<th></th>
<th>Build on Red Hat EL 3</th>
<th>Build on Red Hat EL 4</th>
<th>Build on Ubuntu 6.06.2 g++3.3</th>
<th>Build on Ubuntu 6.06.2 g++4.0</th>
<th>Build on Ubuntu 8.04</th>
<th>Build on Ubuntu 9.10</th>
</tr>
<tr>
<th>Runs on Red Hat EL 3?</th>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>No</td>
</tr>
<tr>
<th>Runs on Red Hat EL 4?</th>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<th>Runs on Ubuntu 6.06.2?</th>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<th>Runs on Ubuntu 8.04?</th>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<th>Runs on Ubuntu 9.10?</th>
<td>No</td>
<td>Yes</td>
<td>No</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
</table>
<h3>Analysis</h3>
<p>
There are two classes of systems. Those with libstdc++5.0 (Red Hat EL 3), and those with libstdc++6.0 (All others). If you build on a system with one version, your application will only run on systems which have that library.
<p>
Ubuntu 6.06.2 is a special case. It does not have libstdc++5.0 by default, but you can add it by installing and building with g++3.3. That makes Ubuntu 6.06.2 a great build environment for portable binaries.
<h2>Linux Standards Base</h2>
<p>
Linux Standards Base (LSB) has an <a href="http://ldn.linuxfoundation.org/lsb/check-your-app">excellent utility</a> that will predict which versions of Linux your application won't run on. But I had some problems using the rest of <a href="http://ldn.linuxfoundation.org/lsb">their toolchain</a>:
<ol>
<li>It's not clear what you have to download to use their toolchain.
<li>It's not clear how to use their toolchain (Hint: It's a wrapper around gcc)
<li>Their toolchain doesn't work with recent versions of gcc
<li>Once I finally found a distribution that would work with the lsb tools, it did not produce a portable application.
</ol>
<p>
I spent six hours on LSB one weekend and gave up.
<h2>Summary</h2>
To distribute binaries on Linux,
<ul>
<li>Dynamically link the standard libraries
<li>Produce one binary using g++3.3 for older systems
<li>Produce another binary using g++4.0 for newer systems
</ul>
Hopefully in time the situation will improve. According to the <a href="http://www.cs.huji.ac.il/~etsman/Docs/gcc-3.4-base/libstdc++/html/faq/index.html">FAQ on libstdc++ 3</a>:
<blockquote>
The GNU C/C++/FORTRAN/<pick-a-language> compiler (gcc, g++, etc) is widely considered to be one of the leading compilers in the world. Its development has recently been taken over by the GCC team. <b>All of the rapid development and near-legendary portability that are the hallmarks of an open-source project are being applied to libstdc++. </b>
</blockquote>
<h3>If you liked this you'll love:</h3>
<ul>
<li><a href="http://stevehanov.ca/blog/index.php?id=54">Xfce vs. Windows 3.1 Control Panel: head to head!</a>
<li><a href="http://stevehanov.ca/blog/index.php?id=95">C++: A Language for Next Generation Web Apps</a>
<li><a href="http://stevehanov.ca/blog/index.php?id=68">When programmers design web sites (comic)</a>
</ul>
<ul><li><a href='?id=78'>It's a dirty job... (comic)</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=65'>Drawing Graphs with Physics</a></ul>
Bending over: How to sell your software to large companies
tag:smhanovtechblog,2007:id96
2010-04-16T18:00:00-05:00
2010-04-16T18:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<i>This post also appears, with my permission, on <a href="http://blog.asmartbear.com/selling-to-large-companies.html">A Smart Bear</a>, with additional editorial comments by Jason Cohen, founder of Smart Bear Software.</i>
<p>
<p>
For a micro-ISV, selling to businesses can be more lucrative than selling to consumers. Instead of making a few dollars per sale and hoping for thousands of sales, you sell to only a few customers, and charge much higher rates. But the rates are high for a reason. It takes more time and money to sell to businesses.
</p>
<img src="images/chair.jpg" align=right>
<h2>Legal Issues</h2>
<p>Consumers rarely read software license agreements. Most corporate customers don't read them either, but some have legal departments that must approve any agreement that the company makes, no matter how small. Your EULA will be examined with the same fervor as a billion dollar acquisition.
</p>
<p>The license agreement's primary purpose, then, is to get past the customer's legal team quickly, because they stand between you and a sale. It helps if it is fair and well balanced at the start. That way, if they add crazy one-sided terms, you can negotiate without sounding unreasonable.
</p>
<p>Some terms that you may be asked for:
</p>
<ul>
<li><b>"If you go out of business, we get all of your source code."</b> The request is common. The customer sees it as an insurance policy in case a smaller supplier disappears, and the assurance it provides may be so important that they are unwilling to drop it. Source code escrow services will hold on to your source code for a fee (hint: get the buyer to pay). Opt for a more informal arrangement if they don't specifically ask for this service.</li>
<li><b>"If someone sues us over your product, you have to pay our legal costs."</b> Indemnification is also a standard clause that is difficult to get removed. If you can't stomache any risk of personal bankruptcy, incorporating your micro-ISV is a must.</li>
<li><b>Support details.</b> Are you going to be providing free technical support for this product in perpetuity? I hope not.</li>
<li><b>"What happens if the product is defective?"</b> It's only fair to offer a full refund if the customer is not satisfied.</li>
</ul>
<p>A good software license agreement that you can re-use in a variety of situations can cost anywhere from $1000 to $5000. It pays to shop around.
</p>
<h2>The procurement process</h2>
<h3>Quotations</h3>
<p>A quotation looks just like an invoice, except that it has an expiry date. Sixty days ought to be long enough for the client to make a decision, even if the whole department goes on consecutive vacations.
</p>
<h3>Evaluation Version</h3>
The purchasing process can take a long time, so you might be asked to provide an evaluation version while the details of the sale are worked out. It's a great idea, because after the buyer incorporates your product into their processes, they aren't going walk away from the deal easily. However, it is unclear whether the customer acknowledges any of your license terms during the evaluation period. The product should have a time limited expiry and other technical measures to ensure compliance.
<h3>Purchase orders</h3>
<p>You and your buyer have patiently waited for five months for the company's legal team review your license. Now, the signed copies have been faxed (yes, faxed!) back and forth. At last, they'll click on that Paypal button on your order page...
</p>
<p>Think again. Once a large business has agreed to buy your product, you are expected to send it to them <i>for free</i>. They do not have to pay you a dime until they feel like it. Instead, they will send a purchase order.
</p>
<p>The good news is that purchase orders are a legally binding promise to pay you, after all of the terms have been fulfilled. Here is a diagram to illustrate the procedure:
</p>
<p align=center>
<a href="http://www.websequencediagrams.com/?lz=QnVzaW5lc3MtPlN1cHBsaWVyOiBQdXJjaGFzZSBPcmRlclxuKEl0ZW1zLCBQcmljZSwgYW5kIFBPIE51bWJlcikKbm90ZSBvdmVyIABECApUaGUABAkgaXMgbm93IGxlZ2FsbHkgYm91bmQgdG8gCnBheSB0aGUgcwByByBhY2NvcmRpbmcgdG8AFQV0ZXJtcwpvbgAiBQCBCA4gKHVzdQBMBTMwIGRheXMuKQplbmQgbm90ZQoAgUMILT4AgVcIOiBzaGlwIHByb2R1Y3RzAA0VSW52b2ljZQCBPjI6IFdhaXRzIGV4YWN0bHkgMjkAgQ0FCgCCSBRIZXJlJ3MgYSBjaGVjay4KCg&s=napkin"><img src="http://www.websequencediagrams.com/cgi-bin/cdraw?lz=QnVzaW5lc3MtPlN1cHBsaWVyOiBQdXJjaGFzZSBPcmRlclxuKEl0ZW1zLCBQcmljZSwgYW5kIFBPIE51bWJlcikKbm90ZSBvdmVyIABECApUaGUABAkgaXMgbm93IGxlZ2FsbHkgYm91bmQgdG8KcGF5IHRoZSBzAHEHIGFjY29yZGluZyB0bwAVBXRlcm1zCm9uACIFAIEHDiAodXN1AEsFMzAgZGF5cy4pCmVuZCBub3RlCgCBQggtPgCBVgg6IHNoaXAgcHJvZHVjdHMADRVJbnZvaWNlAIE9MjogV2FpdHMgZXhhY3RseSAyOQCBDQUKAIJHFEhlcmUncyBhIGNoZWNrLgoK&s=napkin"><br>
Click to Edit on WebSequenceDiagrams.com</a>
</p>
<p>If you are lucky, they will use PDF files for the purchase order and invoice. But you will probably have to send and receive some more faxes.
</p>
<h3>"Please read this 100 page document about our invoicing process.."</h3>
<p>Sometimes, after everything is agreed, you'll be asked to perform some kind of insanely complex invoicing procedure. The instructions are laced with stern, upper case warnings that if the invoice doesn't follow the proper format, lacks item category labels (found in document B), or is submitted during the wrong hours, it will be ignored.
</p>
<p>If you have priced your product appropriately it will be worth it to spend a few hours to learn their codes and procedures. If the price is too low, you can try your luck and (politely) ask if there are any other options. (<i>Do not</i> mention why!)
</p>
<p>
I have successfully gotten a couple of companies to use a reseller instead of dealing with me directly, but it is only because they had an existing relationship with the reseller. See below.
</p>
<h3>"We will release the funds after you provide your US social security number.."</h3>
US customers will sometimes ask for a Taxpayer Identification Number (TIN). If you are not a US taxpayer, you don't need one. Once you point this out, they may be okay with it, or they will ask you to fill out a US form <a href="http://en.wikipedia.org/wiki/IRS_tax_forms#W-8BEN">W-8BEN</a> and send it to them. The form is scary because it states that your "income" will be subject to a 40% withholding tax. Don't worry: the purchase price is not "an amount subject to withholding", and sellers do not need to start doing US taxes (if they aren't already). Some US businesses feel that they must keep this form on file for all suppliers, and it's easier to comply than argue. <a href="http://taxes.about.com/od/taxplanning/qt/form_W8BEN.htm">Here's some more information</a>.
<h3>"We only pay using Bankers' Scrolls made from papyrus"</h3>
<p>Many companies have a policy against using Paypal. It's best to use an old fashioned check if you can. You can suggest, but never insist on a method of payment. Money is money! Some international customers only use bank transfers. If so, call your bank for the information that you need to provide them, and expect about $30 of the payment to go to fees.
<p>
I absolutely <i>love</i> cheques. I walk into the bank and pay $0.60 to deposit a $5000 cheque, whereas Stripe or Paypal would have cost me $125 for the same. More and more, though, US companies are using electronic transfers. RBC randomly charges me $15 for these, but not all the time.
<h2>Resellers</h2>
<p>Imagine you are asked to buy some software from, say Adobe. </p>
<ol>
<li>You go to their web site,</li>
<li>try to find the link to buy, </li>
<li>figure out how to pay, </li>
<li>get to the checkout page, </li>
<li>then stop and search Google for "adobe coupon codes", </li>
<li>go back to step 1</li>
<li>keep refreshing your email for the link, </li>
<li>download the software.</li>
<li>Keep of record of the receipt somewhere.</li>
</ol>
<p>
Now imagine you have to do this for 1000 different items, at 1000 different web sites. It gets to be a very large job. Some companies have outsourced their procurement and license management to resellers.
</p>
<p>A reseller is simply an intermediary who pays you and provides the software to their client. It's also their job to ask for a discount, but there is no need to provide one. They have been told to acquire your product, and have already been paid a fee as a percentage of your price.
</p>
<p>
As a consequence, the reseller will be very quick to renew your software and pay the maintenance fee each year. This often occurs even if the original buyer is no longer at the company.
</p>
<p>
You don't need to be listed in the reseller's catalogue, and you don't need to have a relationship with the reseller. Just be prepared for emails asking, "Do you work with resellers?" and respond yes, because they have a client who definitely wants buy your software.
</p>
<p>
Follow instructions, invoice quickly, and you will learn to love resellers. The only annoying thing is on mornings when you get six of them asking for quotes on the same product, and you know a big company is just trying to shop around, causing more work for you.
</p>
<h2>Keep smiling</h2>
<p>
Selling to big companies can be frustrating. Throughout the process, it is important to stay professional and pleasant. Sometimes, it may appear that your customer is trying to screw you. Even if they are, is your job to be jovial, point it out, and assume that it is a simple oversight. It makes no business sense to <a href="http://www.kalzumeus.com/2007/02/16/how-to-deal-with-abusive-customers/">throw money away</a> because of a rude email.
</p>
<h2>Further reading</h2>
<p>
Here are some of my favourite blogs on the software biz:
</p>
<ul>
<li><a href="http://blog.asmartbear.com/">A Smart Bear</a> (Creator of Code Collaborator)</li>
<li><a href="http://successfulsoftware.net/">Successful Software</a></li>
<li><a href="http://www.kalzumeus.com/">MicroISV on a shoestring</a></li>
</ul>
<p><ul><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=13'>Draw waveforms and hear them</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a><li><a href='?id=83'>Sign here (comic)</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=123'>Zero load time file formats</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a></ul>
Regular Expression Matching can be Ugly and Slow
tag:smhanovtechblog,2007:id90
2010-03-13T11:00:00-05:00
2010-03-13T11:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
If you open the first few pages of O'Reilly's <a href="http://books.google.ca/books?id=gJrmszNHQV4C&lpg=PP1&dq=beautiful%20code&pg=PA1#v=onepage&q=&f=false">Beautiful Code</a>, you will find a well written chapter by Brian Kernighan (Personal motto: "No, I didn't invent C. Who told you that?"). The non-C inventing professor describes how a limited form of regular expressions can be implemented elegantly in only a few lines of C code.
<pre>
/* match: search for re anywhere in text */
int match(char *re, char *text)
{
if (re[0] == '^')
return matchhere(re+1, text);
do { /* must look at empty string */
if (matchhere(re, text))
return 1;
} while (*text++ != '');
return 0;
}
/* matchhere: search for re at beginning of text */
int matchhere(char *re, char *text)
{
if (re[0] == '')
return 1;
if (re[1] == '*')
return matchstar(re[0], re+2, text);
if (re[0] == '$' && re[1] == '')
return *text == '';
if (*text!='' && (re[0]=='.' || re[0]==*text))
return matchhere(re+1, text+1);
return 0;
}
/* matchstar: search for c*re at beginning of text */
int matchstar(int c, char *re, char *text)
{
do { /* a * matches zero or more instances */
if (matchhere(re, text))
return 1;
} while (*text!='' && (*text++==c || c=='.'));
return 0;
}
</pre>
<p>
However, if you try to actually use it for something you will realize that it is really just a glorified filename globber, which is a much simpler problem than regular expressions. In particular, it doesn't support alternation (|), subexpressions, character ranges, or capturing.
<p>
I wondered what it would take to implement the whole shebang � capturing regular expressions as concisely as possible, in C.
<pre>
int _capslot = 0;
int match_char(char* re, char str)
{
return *re == '' ?
re[1] == 'd' ? isdigit(str) :
re[1] == 's' ? isspace(str) :
re[1] == 'w' ? isalnum(str) || str == '_' :
re[1] == 'n' ? str == 'n' :
re[1] == 'r' ? str == 'r' :
re[1] == 't' ? str == 't' :
re[1] == str
: (*re == '.' || *re == str);
}
// Performs a capturing match-here operation. Returns the number of characters
// matched.
//
// re: Indirect pointer to regular expression
// str: Indirect pointer to string to match.
// skip: must be 0. (Used internally for recursion)
// capture: Returned array of captured subexpressions. To determine the size
// needed, count the number of unescaped '(' in the expression. Subexpressions
// may be nested.
int parse_expr(char** re, char** str, int skip, char* capture[])
{
int ok = 0;
char* backup_str = *str;
for( ;; ) {
int ok2 = 1, skip2 = skip;
while( **re != 0 && **re != '|' && **re != ')' ) {
int ok3 = 1, cnt=0, backup_capslot = _capslot;
char* backup_re = *re;
for( ;; *re = backup_re, _capslot = backup_capslot ) {
char* backup_str = *str;
if ( **re == 0 ) {
} else if ( **re == '(' ) {
char* start = *str;
int slot = _capslot++;
(*re)++;
ok3 = !!parse_expr(re,str,skip2, capture);
if ( ok3 && capture ) {
free( capture[slot] );
capture[slot] = (char*)malloc( *str-start+1 );
memcpy( capture[slot], start, *str-start );
capture[slot][*str-start] = 0;
}
if ( **re == ')' ) (*re)++;
} else if ( **re == '[' ) {
int inv = 0;
(*re)++;
ok3 = 0;
while( **re && **re != ']' ) {
if ( **re == '^' ) {
inv = 1;
} else if ( **re != '' && (*re)[1] == '-' ) {
ok3 |= **str >= (*re)[0] && **str <= (*re)[2];
*re += 2;
} else if ( match_char( *re, **str ) ) {
ok3 = 1;
}
(*re)++;
}
ok3 ^= inv;
ok3 && (*str)++;
if ( **re == ']' ) (*re)++;
} else if ( skip2 ) {
*re += (**re == '') + 1;
} else if ( match_char(*re, **str ) ) {
*re += (**re == '') + 1;
(*str)++;
} else {
*re += (**re == '') + 1;
ok3 = 0;
}
if ( !ok3 ) *str = backup_str;
cnt += ok3;
if ( **re != '+' && **re != '*' || skip2 || !ok3 ) break;
}
if ( **re == '?' ) {
(*re)++;
ok3 = 1;
} else if ( **re == '+' ) {
(*re)++;
ok3 = cnt >= 1;
} else if ( **re == '*' ) {
(*re)++;
ok3 = cnt >= 0;
}
ok2 &= ok3;
skip2 |= !ok2;
}
ok |= ok2;
if ( **re != '|' ) break;
(*re)++;
ok && (skip = 1) || (*str = backup_str);
}
if ( !ok ) *str = backup_str;
return *str - backup_str;
}
int main(void)
{
char* text = "September 22, 1966";
char* expr = "([A-Za-z]+) (d+), (d+)";
char* fields[3];
parse_expr( &expr, &text, 0, fields );
printf("Month: %s, Day: %s, Year: %sn", fields[0], fields[1], fields[2]);
return 0;
}
</pre>
<p>
This is horrible on so many levels. First of all, it is impossible to understand. Secondly, it uses backtracking, which means if the expression did not match a certain way it keeps trying until all possibilities are exhausted. Thirdly, I forgot to put in "$" and "^".
<h2>There is a better way</h2>
A more efficient way to match regular expressions has been known for decades, and is described nicely in <a href="http://swtch.com/~rsc/regexp/regexp1.html">Regular Expression Matching can be Simple And Fast</a>. I won't repeat it here (yet).
<p>
However, I will point out some code that is beautiful.
<p>
In 1993, <a href="http://www.snert.com/">Anthony Howe's</a> concise implementation of <tt>grep</tt> won the <a href="http://www.ioccc.org/years.html#1993_ant">Obfuscated C Coding competition</a>. It lazily constructs a finite state machine for the expression as it goes, forming only the parts needed to match or reject the input text.
<p>
Howe kindly released an unobfuscated version of his entry. If you are interested in the innards of regular expressions, after you read <a href="http://swtch.com/~rsc/regexp/regexp1.html">Regular Expression Matching can be Simple And Fast</a>, then take a look at the elegant representation of NFAs in Howe's comments. He describes how the state machine is stored using a flat array. No structures, nodes, or pointers are needed.
<p>
If you're used to making an object for everything, then spending 15 minutes to understand this is guaranteed to expand your mind.
<p>
The full source code is <a href="https://github.com/SirWumpus/ioccc-ag">now on github.</a> (Look for agag.c).
<pre>
/*
* Non-deterministic Finite-state Automaton
*
* An NFA is a directed graph whose nodes are called states. Each
* state is either a labeled state denoting a literal value, or a
* NIL state that has no label. Each state has at most two edges
* leaving it. There is only one start state and one final state,
* which may be the same state.
*
* Various state-structures can be represented by an array. These
* structures have only one start point being the left most index,
* and one end point being the right most index.
*
* a ...[a]...
*
* ab ...[a][b]...
*
* ^a [bol][a]...
*
* a$ ...[a][eol]
*
* a.b ...[a][wild][b]...
* _
* v
* a* ...[][a][]...
* ____^
*
* a? ...[][a]...
* ___^
* _______________
* / v
* a|b ...[][a][/][stop][pass][b]...
* _____________^
*
* [a0-9] ...[class][a][0][1][2][3][4][5][6][7][8][9][stop]...
*
* [^a0-9] ...[nclass][a][0][1][2][3][4][5][6][7][8][9][stop]...
*
* _____________ _______________
* / v/ v
* a|b|c ...[][a][/][stop][][b][/][stop][pass][c]...
* _____________^_____________^
*
* _ _______________
* v / v
* a*(b|c) ...[][a][][][b][/][stop][pass][c]...
* ____^ _____________^
*/
#define BOL ('n')
#define EOL ('n')
#define STOP ('n')
#define WILD (0)
#define CLASS (-1)
#define NCLASS (-2)
#define PASS (-3)
#define JUMP(n) (-4-(n))
#define CHAR(c) (c)
#define ISJUMP(x) ((x) <= PASS)
#define ISCHAR(x) (0 <= (x))
</pre>
<ul><li><a href='?id=73'>How to run a linux based home web server</a><li><a href='?id=142'>My thoughts on various programming languages</a><li><a href='?id=140'>A little VIM hacking</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=97'>Creating portable binaries on Linux</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a></ul>
C++: A language for next generation web apps
tag:smhanovtechblog,2007:id95
2010-01-26T18:00:00-05:00
2010-01-26T18:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<!-- <div style="font-size: smaller;color:gray;">This article is translated to <a href="http://science.webhostinggeeks.com/c-jezik">Serbo-Croatian</a> language by Anja Skrba from Webhostinggeeks.com</div>
<div style="font-size: smaller;color:gray;">This article is translated to <a href="https://www.espertoautoricambi.it/science/2017/07/14/c-keel-jaergmise-polvkonna-veebirakendustega/">Estonian</a> language by Karolin Lohmus</div>
<div style="font-size: smaller;color:gray;">This article is translated to <a href="https://www.homeyou.com/~edu/c-idioma-para-aplicativos">Portugese</a> language by Artur Weber</div>
-->
<p>
On Monday, I was pleased to be an uninvited speaker at <a href="http://devwaterloo.pbworks.com/">Waterloo Devhouse</a>, hosted in <a href="http://postrank.com">Postrank's</a> magnificent office. After making some surreptitious alterations to their agile development wall, I gave a tongue-in-cheek talk on how C++ can fit in to a web application.
<p>
There were <a href="http://devwaterloo.pbworks.com">other cool presentations</a> there too. Check it oot! Er, out!
<p align=center><img border=2 src="images/c_presentation_01.png"/></p>
<p>
During this presentation, I hope to convince you that the C++ programming language is ideal for developing your next web application.
<p align=center><img border=2 src="images/c_presentation_02.png"/></p>
<p>
You might be asking, <i>Steve, Why C++? Why would I subject myself to this horrible language where you have to manage your own memory?</i>
<p align=center><img border=2 src="images/c_presentation_03.png"/></p>
<p>
One reason is efficiency. Not only is C++ inherently fast. It also forces you to think differently. It discourages the use of overly complicated data structures that are built in to other languages. In C++, nesting more than one or two structures results in something that is simply too awkward to use. Instead, the you are forced to seriously consider using the simplest possible representation.
<p align=center><img border=2 src="images/c_presentation_04.png"/></p>
<p>
In addition, just about any library that you'd want to use has already been written, and has a free implementation available on the internet. <a href="http://www.json.org/">Json</a>, and <a href="www.steveshipway.org/software/utils/querystring.c">CGI</a> decoders are freely available. Also, as often overlooked, you have to handle <a href="http://en.wikipedia.org/wiki/UTF-8#Description">UTF8 to Wide-character conversion</a>, but this is easily achieved in a 10 line function.
<p>
The only thing you do have to be careful with is SQL database access. MySql is GPL'd, so you can't even link to its client library in a closed source app. SQLite seems to be free, except if you do business in Germany <a href="http://www.hwaci.com/cgi-bin/license-step1">it is $1000</a> because they have a different definition of <a href="http://stackoverflow.com/questions/219742/open-source-why-not-release-into-public-domain">public domain</a>.
<p align=center><img border=2 src="images/c_presentation_05.png"/></p>
<p>
Here is the first strategy that you might use to incorporate C++ into your web app. In this diagram, there are two things between the browser and your application -- the web server, and the php script. It is not as efficient as it could be, and there are also security implications with calling command line programs from php.
<p align=center><img border=2 src="images/c_presentation_06.png"/></p>
<p>
In this model, you write your C++ program as a CGI script directly, using one of the freely available query string parsing libraries. It is efficient and clean. When your browser requests information, the webserver starts your program, which spits out the response in json format. This response is then relayed back to the browser.
<a name="sock"></a>
<p align=center><img border=2 src="images/c_presentation_07.png"/></p>
<p>
I use the above strategy for <a href="http://rhymebrain.com">RhymeBrain</a>. It includes advanced statistical algorithms that let it sound out any word that you put in. For example, you can enter the word "postrank" and see that it rhymes with such gems as "blank, prank, drank, tanked, and stank".
</p>
<p align=center><img border=2 src="images/c_presentation_08.png"/></p>
<p>
Because it's written in C++, I can run it on my <a href="http://stevehanov.ca/blog/index.php?id=71">super-powerful datacenter</a>. It sits atop my sock drawer at RhymeBrain headquarters.
</p>
<p>
This powerful 1GHz beast can load the entire 2.6 million word database and fire back the response in about 50 milliseconds. It does this from a cold start, for each request.
</p>
<p align=center><img border=2 src="images/c_presentation_09.png"/></p>
<p>
But there is a third strategy: If you write your own webserver, you can cut out the middleman and serve the request directly. Your javascript code makes a request, and all your web server has to do is call a function to send back the results.
</p>
<p align=center><img border=2 src="images/c_presentation_10.png"/></p>
<p>
Writing a webserver isn't that hard. Here is the complete implementation of the <a href="http://hibachi.snert.org:8008/">Hibachi</a> web server. It supports virtual hosts, and perl and php scripting, among other things. It was written by former Waterloo-ite <a href="http://www.snert.com/about.html">Anthony Howe</a>, and won the <a href="http://www.ioccc.org/2004/hibachi.hint">2004 International Obfuscated C Coding Competition</a>.
</p>
<p align=center><img border=2 src="images/c_presentation_11.png"/></p>
<p>
Inspired by Hibachi, I wrote my own webserver and built <a href="http://www.websequencediagrams.com">WebSequenceDiagrams</a> (which runs in a real data center..). Doing it this way reveals a new business model. It is possible to package up the entire web application into a single installer that runs on Windows and Linux. Since it's all integrated, there is no need for customers to fuss around with Apache or the numerous other moving parts that could break if I shipped separate components. (You can run it in your organization for <a href="http://www.websequencediagrams.com/order.html">as little as $99</a>).
<p>
The only disadvantage is that the final executable is really small -- around 700K. It may be a little smaller than some customers are expecting. <i>During the presentation, it was suggested that I ship it as an appliance, in a <b>huge box</b>, to compensate</i>.
</p>
<h3>Conclusion</h3>
I hope that I've convinced you of the benefits of C++ in your next web application:
<ul>
<li>Reduced hardware costs
<li>Readily available libraries for web tasks
<li>Portability
<li>Extreme flexibility in deployment
</ul>
<p>
</p><ul><li><a href='?id=144'>Finding Bieber: On removing duplicates from a set of documents</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=11'>Cell Phones on Airplanes</a><li><a href='?id=83'>Sign here (comic)</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=71'>Using the Acer Aspire One as a web server</a><li><a href='?id=115'>Compressing dictionaries with a DAWG</a></ul>
qb.js: An implementation of QBASIC in Javascript
tag:smhanovtechblog,2007:id92
2010-01-08T21:30:00-05:00
2010-01-08T21:30:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
If your browser supports the proposed CANVAS tag, you will see a screen below containing a BASIC program. <b>This only implements enough of the language to run NIBBLES.BAS</b>
<p>
<script type="text/javascript" src="http://stevehanov.ca/qb.js/qb.js"></script>
<p>
Many programmers seldom think about how their compiler or scripting language is implemented. To them, it is a tool, and the less it gets in the way, the better. However, for a craftsman, knowing how your tools work can help you do a better job. It helps to think on several levels at once. Take this code, for example:
<p>
<pre>
function createDummyString( length )
{
var result = "";
for ( var i = 0; i < length; i++ ) {
result = result + " ";
}
return result;
}
</pre>
<p>
<img src="http://unrealitymag.com/wp-content/uploads/2009/02/neotitle.jpg" align=right>In some languages, strings are immutable, so <tt>result</tt> could be repeatedly created, copied, and destroyed millions of times in this one function call.
<p>
Those who are familiar with compilers have the ability to think on another level. For them, visible about 10 cm behind the computer screen is a whole other level of code, which contains how the programming language might be implemented. Beyond that, further in the distance, lies assembly language, with its precarious branch prediction, happy cache hits, and misrable misses.
<p>
Here is an implementation of QBASIC in Javascript. In this next series of blog entries, we will explore its inner workings, covering all parts of the compilation process.
<h3>What works</h3>
<p>Only text mode is supported. The most common commands (enough to run nibbles) are implemented. These include:
<ul>
<li>Subs and functions
<li>Arrays
<li>User types
<li>Shared variables
<li>Loops
<li>Input from screen
</ul>
<h3>What doesn't work</h3>
<ul>
<li>Graphics modes are not supported
<li>No statements are allowed on the same line as IF/THEN
<li>Line numbers are not supported
<li>Only the built-in functions used by NIBBLES.BAS are implemented
<li>All subroutines and functions must be declared using DECLARE
</ul>
<p>
This is <i>far</i> from being done. In the comments, <b>AC0KG points out that P=1-1 doesn't work. </b>
<p>
In short, it would need another 50 or 100 hours of work and there is no reason to do this.
<p>
The parser is slow because we are using a simple Earley parser. <a href="../qb.js/EarleyParser.js">(EarleyParser.js)</a>. Update in 2014: If I ever do this again I would use a "Packrat" parser.
<p>
<h3>License</h3>
License is GPL v3.
<h2>Overview of the system</h2>
<p>
The compiler takes your BASIC program and converts it into a list of user data types, data from DATA statements, and bytecode statements. Here's pretty picture of the previous sentence:
<p align=center><img src="images/comp-stages.png"/></p>
<p>
If you just look at the Javascript source it will be confusing to figure out what goes where. So here is a map of all the important "classes" of the system:
<p align=center><img src="images/comp-modules.png"/></p>
<h3>Console</h3>
<a href="../qb.js/console.js">(Source)</a> A canvas that represents the screen and captures keyboard input
<h3>Virtual machine</h3>
<a href="../qb.js/virtualmachine.js">(Source)</a> The virtual machine executes bytecode. The bytecode instructions may manipulate the stack, jump to a new address, or use the console functions. They may also execute system functions (such as LEFT$) or system subroutines (such as CLS).
<h3>Types</h3>
<a href="../qb.js/types.js">(Source)</a> Each type (single, double, integer, long, array, user) has functions to create an initial value, or to copy a value into a variable. Upon a copy, for instance, the integer type performs rounding and truncates the value to 16 bits.
<h3>Codegenerator</h3>
<a href="../qb.js/CodeGenerator.js">(Source)</a> The code generator visits each node of the abstract syntax tree abd generates bytecode for it. It uses the type information added by the TypeChecker as an aid.
<h3>TypeChecker</h3>
<a href="../qb.js/TypeChecker.js">(Source)</a> The TypeChecker's job is to catch any errors before we try compiling. Without it, you could write a program that tries to multiply an array by the string "George". Then the virtual machine would say "WTF?!" and crash. It fills in the type for any expression in the syntax tree.
<h3>GlrParser</h3>
<a href="../qb.js/GlrParser.js">(Source)</a> The GlrParser is an implementation of Tomita's GLR parser. It uses a RuleSet to parse the program into an abstract syntax tree.
<p>
Unfortunately, there is a problem with my implementation of the GLR parser. I think I know how to solve it, but at the moment, the system uses the very slow, but concise Earley parser, in <a href="../qb.js/EarleyParser.js">EarleyParser.js</a> EarleyParser.js.
<h3>Tokenizer</h3>
<a href="../qb.js/Tokenizer.js">(Source)</a> If you try to use javascript's built in Regex object for splitting text into tokens, you will soon pull your hair out and run screaming through the halls. Instead, the tokenizer implements a simple Thompson NFA with lazy evaluation. In some cases, this technique is faster than Javascript's own RegEx functions!
<h3>Ruleset</h3>
<a href="../qb.js/RuleSet.js">(Source)</a> The ruleset contains grammar rules. In addition, it can remove redundant rules, and compute the FIRST and FOLLOW sets.
<h3>Ruleparser</h3>
<a href="../qb.js/RuleParser.js">(Source)</a> The RuleParser is implemented on top of the RuleSet. It uses the parser to parse rules, and transforms them to add goodies like comma separated lists, kleene star, and alternation operators. But it's main purpose is to allow me to say, "My parser uses itself to parse its own rules!"
<h3>QBasic</h3>
<a href="../qb.js/qbasic.js">(Source)</a> Finally, qbasic.js contains all of the grammar rules and Abstract Syntax Tree nodes for BASIC programs.
<hr/>
<p>
<h2>Virtual machine</h2>
<p>
The most straightforward way of creating a basic compiler in javascript is to directly translate the basic into javascript functions. But this approach will not work for two reasons. First, there is "goto" which, although it is a reserved word, is not yet in Javascript. (Obviously, ECMAScript community finds "with", prototype inheritance, and the rules of the 'this' keyword to be far less confusing than allowing "goto"). It is possible to <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.42.1485&rep=rep1&type=pdf">automatically move statements around to eliminate GOTO</a>, but you don't want to go there.
<p>
The other problem is that browsers tend to freeze until javascript programs finish running. To avoid freezing the browser until the program ends, we break the program into small chunks, and execute a few of those chunks every so often using a javascript timer. This gives the appearance of a running program and doesn't freeze the browser.
<p>
Bytecode solves both of those problems. By breaking the program into bytecode instructions, we can implement goto by just changing which instruction we are going to execute next. We can also suspend execution any time to allow the user to interact with the browser.
<p>
The virtual machine executes programs, which consist of types, an array of instructions, and a set of variable names which are shared. It has some data:
<p>
<ul>
<li><em>pc, the program counter</em> (the index of the next instruction to execute)
<li><em>a stack of frames</em>. A frame maps variable names to their values. Each time a function is called, the frame is added to the stack. Each time it returns, the frame is removed, thus destroying all local variables.
<li><em>an execution stack</em>. The instructions can manipulate this stack. Two types of things can be pushed onto the stack: either a value (which is.a javascript string, number, or null), or a reference to a value (which can be the ScalarVariable or ArrayVariable objects).
</ul>
<p>
<h2>Executing Instructions</h2>
<p>
Javascript excels at looking up things in objects and mapping strings to functions. This makes the virtual machine instruction lookup very efficient.
<p>
All of the information about each instruction is stored in a single object, called "Instructions". That means we can run dispatch instructions very simply, leaving the heavy work of figuring out where the code is to the javascript runtime:
<pre>
while( this.pc < this.instructions.length ) {
var next = this.instructions[this.pc++];
next.instr.execute( this, instr.arg );
}
</pre>
<p>
Each instruction can manipulate the stack, set variables, or change the <tt>pc</tt> to jump to another location.
<h2>Example</h2>
Here's some basic code:
<p>
<pre>
A = 1
B = 2
PRINT A + B
</pre>
<p>
And here's the bytecode produced to run the above statements:
<p>
<pre>
' L1 A = 1
[0] pushconst 1
[1] pushref A
[2] assign
' L2 B = 2
[3] pushconst 2
[4] pushref B
[5] assign
' L3 PRINT A + B
[6] pushvalue A
[7] pushvalue B
[8] +
[9] syscall print
[10] pushconst 'n'
[11] syscall print
[12] ret
[13] end
</pre>
<p>
PUSHCONST 1 pushes a number 1 onto the stack. PUSHREF A is a bit complicated, though. It pushes a reference to variable A onto the stack. Since there was no prior value of A, it has to create one with the default type of SINGLE and adds the mapping from the name "A" to that variable. After all that housekeepint, it does the push. The state of the virtual machine looks like this:
<p align=center><img src="images/vm-state1.png"></p>
<p>
The ASSIGN instruction expects a reference and a constant on the stack. It removes them, and assigns the reference to the variable. Here's the actual javascript implementation of the instruction:
<p>
<pre>
ASSIGN: {
name: "assign",
numArgs: 0,
execute: function( vm, arg )
{
// Copy the value into the variable reference.
// Stack: left hand side: variable reference
// right hand side: value to assign.
var lhs = vm.stack.pop();
var rhs = vm.stack.pop();
lhs.value = lhs.type.copy( rhs );
}
</pre>
<p>
Let's look at instructions 6 to 8. We've seen PUSHREF, and PUSHCONST, now what's PUSHVALUE? This instructions the variable name and pushes its current value on the stack. After instruction 7, the state of the virtual machine is this:
<p align=center><img src="images/vm-state2.png" /></p>
<p>
All of the instructions are pretty simple to implement. Here's how the "+" instruction works. Thanks to Javascript, it works for both strings and numbers.
<p>
<pre>
"+": {
name: "+",
numArgs: 0,
execute: function( vm, arg )
{
var rhs = vm.stack.pop();
var lhs = vm.stack.pop();
vm.stack.push( lhs + rhs );
}
},
</pre>
<p>
Finally, we call the "print" system function, which just pops the result off the stack and displays it on the screen.
<p>
Here are the other instructions:
<h3>ASSIGN</h3>
Pop two things off the stack, and assign the value to the reference.
<h3>MEMBER_VALUE</h3>
Pop the reference to the user defined structure, and push the value of the named member.
<h3>MEMBER_DEREF</h3>
Like MEMBER_VALUE, but pushes a reference to the named mamber.
<h3>ARRAY_DEREF</h3>
Pop the array indices, and push a reference to the given location of the array.
<h3>PUSHCONST</h3>
Push the literal value onto the stack.
<h3>RET</h3>
destroy the current variable map, and restore the previous one, jumping to the previous value of PC.
<h3>GOSUB</h3>
Like call, but instead of using a new variable map, copy the current one.
<h3>CALL</h3>
Create a new, empty variable map, store PC in it, and jump to the given location.
<h3>JMP</h3>
Immediately jump to the given location.
<h3>BNZ</h3>
pop the top of the stack and jump to the given location if it is non-zero
<h3>MOD, /, *, +, -, AND, OR, NOT, <>, >=, <=, <, >, =</h3>
Pop one or two arguments off the top of the stack, perform the given operation, and push the result onto the stack.
<h3>BZ</h3>
Pop the top of the stack and jump to the given location if it is zero.
<h3>END</h3>
Halt the VM.
<h3>NEW</h3>
Create a new unnamed instance of the variable of the specified type, and push it onto the stack.
<h3>PUSHTYPE</h3>
Push the named type onto the stack
<h3>PUSHVALUE</h3>
Push the value of the given variable onto the stack.
<h3>PUSHREF</h3>
Push a reference to the given variable onto the stack.
<h3>POP</h3>
Pop the stack and discard
<h3>POPVAL</h3>
Set the given variable's value to be the value at the top of the stack.
<h3>RESTORE</h3>
Set data index to the given value
<h3>COPYTOP</h3>
duplicate the top of the stack.
<h3>FORLOOP</h3>
Using counter, step, and end value on the stack, determine if the for loop should continue. If not, jump to the given location.
<h3>SYSCALL</h3>
Call the given system function or subroutine (eg, LOCATE or CLS or LEFT$)
<h2>Console</h2>
<p>The console performs these functions:
<ul>
<li>Printing to the screen
<li>Converting QBASIC's colour codes to HTML colours
<li>Keeping track of cursor position
<li>Blinking the cursor
<li>Allowing line oriented user input
<li>Keeping a keyboard buffer for INKEY$, and converting DOM keycodes to qbasic keyboard codes.
</ul>
<p>
The console uses an HTML Canvas object for display. It has <a href="../qb.js/charmap.png">an image of every character in the IBM character set</a>. When you want to print something, it first draws a solid rectangle of the background colour at that position. Then it copies the character's image, leaving alone any transparent pixels.
<p>
In input mode, anything the user types is displayed on the screen and copied to a buffer. When you hit enter, input mode ends, and a completion function is called to restart the virtual machine and process the input.
<p>
While not in input mode, and you hit a key, it is converted into a QBASIC character code and added to the keyboard buffer. This buffer is used for the INKEY$ function.
<h2>Until next time</h2>
<p>
We've covered the runtime system of our virtual computer. We've left out how to handle arrays and function calls. For those features, you'll have to look up how the ARRAY_DEREF and CALL instructions are implemented, in <a href="../qb.js/virtualmachine.js">virtualmachine.js</a>.
<p>
<h2>Further reading...</h2>
<ul>
<li><a href="http://stevehanov.ca/blog/?id=79">How QBASIC almost got me killed</a>
<li><a href="http://stevehanov.ca/blog/?id=67">Game theory, Salary negotiation, and Programmers</a>
<li><a href="http://stevehanov.ca/blog/?id=56">How a programmer reads your resume</a>
<li><a href="http://stevehanov.ca/blog/?id=68">When programmers design web sites...</a>
</ul><ul><li><a href='?id=60'>Is 2009 the year of Linux malware?</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=82'>The PenIsland Problem: Text-to-speech for domain names</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=65'>Drawing Graphs with Physics</a></ul>
Zwibbler: A simple drawing program using Javascript and Canvas
tag:smhanovtechblog,2007:id93
2009-12-28T16:17:47-05:00
2009-12-28T16:17:47-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><img src="images/stick-figures.png"></p>
<p align=center><a href="http://zwibbler.com"><img border=0 src="images/try_it_now.png"></a><br>
Tested under Firefox 3.5.6 and Google Chrome 3.0.195.38</p>
<h1>Introduction</h1>
<p>
This project extends the technique I created for <a href="http://stevehanov.ca/blog/index.php?id=33">imprecise line-drawing</a> to create an entire vector graphics application, similar to <a href="http://www.inkscape.org/screenshots/index.php?lang=en">Inkscape</a>. It is written almost entirely in Javascript, except for a server-side program that renders text.
<p>
See <a href="#implementation">below</a> if you are interested in the implementation and coding parts.
<p>
<h2>Features</h2>
<ul>
<li>Download and share your drawings in PNG, PDF, or SVG formats.
<li>Box, circle, lines, and curve primitive shapes
<li>Shadows when supported by browser
<li>Text in several hand-drawn fonts rendered on the server
<li>Rotate & scale shapes individually or in groups
<li>Select colours using an HSV colour wheel.
<li>Unlimited levels of Undo/Redo
</ul>
<h1>Usage</h1>
Users who are familiar with vector graphics programs such as CorelDRAW! or Inkscape will have no problem using it, because the some of the most common keyboard shortcuts work the same way. Otherwise, it will take some getting used to.
<h2>List of keyboard shortcuts</h2>
<p>The keyboard is the only way to do some things.
<table>
<tr><td>C</td><td>Start drawing a new curve</td></tr>
<tr><td>L</td><td>Start drawing a new line</td></tr>
<tr><td>Ctrl+D</td><td>Duplicate selection</td></tr>
<tr><td>Page Up</td><td>Move selected shapes toward you</td></tr>
<tr><td>Page Down</td><td>Move selected shapes away from you</td></tr>
<tr><td>Home</td><td>Bring selected shapes to front</td></tr>
<tr><td>End</td><td>Send selecting shapes to back</td></tr>
<tr><td>Ctrl+G</td><td>Group selected shapes</td></tr>
<tr><td>Ctrl+Shift+G</td><td>Un-group selected shapes</td></tr>
<tr><td>+</td><td>Zoom in</td></tr>
<tr><td>-</td><td>Zoom out</td></tr>
<tr><td>Shift +</td><td>Restore normal zooming</td></tr>
<tr><td>Arrow Keys</td><td>Move around while zoomed-in</td></tr>
</table>
<h2>Saving and loading</h2>
Drawings can be stored on the website by clicking on "Save". If you do not have an account, the drawings will be deleted in a few hours, or if you close your web browser. As soon as you create an account, the drawings will be transferred to long term storage.
<p>
If you don't want to create an account, the "Save" option will also allow you to download your drawing as an image.
<p>
<h2>Drawing simple shapes</h2>
<p>
Click on the rectangle or circle tool in the toolbar, and the desired shapes will immediately appear in the upper left of the drawing area. You can then drag them to where you want to go.
<p>
No two shapes are alike. If you create two circles, they will be slightly different. However, if you duplicate a shape using Ctrl+D, the duplicate will be exactly the same.
<h2>Drawing lines and curves</h2>
<p>To draw a line, click the line tool in the toolbar, then click anywhere in the drawing window to place the first point. You can then click again to place the second point, and so on. To end the line, double click. If you end the line or curve close enough to where you started it, then it will create a closed shape.
<p><i>Hint:</i> The line tool can be activated by pressing "L", and the curve tool can be activated by pressing "C".
<p>Lines have a sloppiness property that can be set after drawing it. To set the property, select on the line you have drawn. The options pertaining to the line will appear to the left of the drawing window.
<p>Curves do not have sloppiness, but smoothness. By increasing smoothness, the curve is modified to make rounder corners.
<p align=center><img src="images/lines_curves.png"></p>
<h2>Drawing text</h2>
<p>
Text is placed the same way as circles or rectangles. Click on the text tool, and some default text is placed in the upper left of the image. You can change what the text says by clicking on the text, and then modifying the "text" property that appears to the right of the canvas.
<p align=center>Example: Editing text using the properties area<br>
<img src="images/text-properties.png" border="2"/></p>
<p>
You can move, scale, or rotate text. However, in this version of the drawing software, scaling will result in loss of quality. Instead, change the font size in the properties area.
<h2>Selecting things</h2>
<p>
While not in line or curve mode, you can select things by dragging a box around them using the mouse. The box must fully enclose the shapes to select them. You can also add a shape to an existing selection by pressing the shift key while clicking it.
<h3>Overlapping shapes</h3>
You can move a shape to the front using the Home key on the keyboard. Send it to the back using the End key.
<h3>Point edit mode</h3>
For some shapes, you can move the corners around by entering "Edit" mode. You can enter edit move by clicking on an already-selected object. In edit mode, the corners that you can move are highlighted using a blue box.
<p align=center>Example: Point edit mode<br>
<img src="images/point-edit.png"/></p>
<h3>Delete</h3>
Delete all objects in the selection by hitting the Delete key on the keyboard.
<h3>Grouping</h3>
<p>
To avoid having to repeatedly select complex groups of shapes, you can "group" the current selection using Ctrl+G. Now, when you click on any one of the group members, all will be selected. You can break apart any selected groups using Ctrl+Shift+G.
<h3>Copy/Paste</h3>
You can copy pieces of your drawings between documents and browser windows. While there are shapes selected, click on "Copy" and they will be transferred to the zwibbler clipboard, stored in your browser. You can then open another document later or in another browser window and paste the shapes there.
<h3>Duplicate</h3>
While there are shapes selected, you can duplicate them by pressing Ctrl+D. Copies of the objects will appear over the existing ones. You will have to move them aside to see the effect.
<h3>Zooming in</h3>
Increase magnification by pressing +. Decrease by pressing -. Restore to original magnification by pressing Shift+'+'. While zoomed in, you can move around using the arrow keys on the keyboard.
<h2>Bugs</h2>
The last 10% of work to make a polished program takes 90% of the time. I know about these bugs:
<ul>
<li>Scaling text reduces its quality.
<li>There's no way to set the background colour of a drawing.
</ul>
<a name="implementation"></a>
<h1>Implementation Notes</h1>
<p>
I should write something about the implementation. Hmm, the <a href="http://www.hanovsolutions.com/webdraw/UndoStack.js">undo stack</a> is pretty standard stuff but may be new to some people. But the thing I was happiest to find was the concept of abstracted Mouse Behaviours.
<p>
Design patterns for mouse behaviour are slim pickings.
When a novice programmer designs a mouse-intensive application, he or she will tend to make a very complex function that deals with all possible cases when the mouse is clicked or moved. It can quickly grow to be unmaintainable.
<p>
To deal with this complexity, we separate the possible states of the system into various MouseBehaviour objects, which implement onMouseDown(x,y), onMouseUp(x,y), and onMouseMove(x,y). (This is a variation on the State design pattern). The default mouse behaviour is the selection mode. If you press the mouse button while over something, it then replaces the default mouse behaviour with a new mouse behaviour. When you lift the button, the new behaviour ends and reverts back to the previous behaviour. This keeps everything simple. There are several mouse behaviours:
<ol>
<li>DefaultBehaviour -- Depending on where you click, it replaces the current behaviour with a new instance of one of the other behaviours.
<li>SelectBoxBehaviour -- if you click on an empty area of the image, it handles drawing the select box. When you lift the button, it selects everything in the rectangle you have drawn.
<li>TransformSelection -- if you clicked on a selection handle, then it transforms the selected shapes as you move the mouse.
<li>MoveEditNodeBehaviour -- If you click a blue edit node, then it moves the node.
<li>DrawLinesBehaviour -- Places points in a line shape until you double-click to end it.
</ol>
These objects can be found in <a href="http://www.hanovsolutions.com/webdraw/DrawView.js">DrawView.js</a>
<p>
<h2>License</h2>
Zwibbler.com is not open source. All source code is Copyright 2010 Hanov Solutions Inc. Drawings produced by you using zwibbler.com remain your property.
<ul><li><a href='?id=135'>How I run my business selling software to Americans</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a><li><a href='?id=10'>Detecting C++ memory leaks</a><li><a href='?id=90'>Regular Expression Matching can be Ugly and Slow</a><li><a href='?id=96'>Bending over: How to sell your software to large companies</a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a></ul>
You don't need a project/solution to use the VC++ debugger
tag:smhanovtechblog,2007:id91
2009-11-05T08:00:00-05:00
2009-11-05T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
You learn a lot of things on the job as a programmer. Years ago, at my first coop position, I was a little confused when my boss went to Visual C++, and tried to open the .EXE file as a project. <i>What a dolt!</i> I thought. <i>That's not going to work.</i>
<p align=center>
<img src="images/vc2008.png">
</p>
<p>
Luckily I kept my mouth shut. You don't need to create projects or solution files to use Visual C++ as a debugger. Just open up the EXE file and run it. If it has debugging information, you can also manually open up the source files and create break points and everything.
<p>
<ul><li><a href='?id=138'>Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a><li><a href='?id=65'>Drawing Graphs with Physics</a><li><a href='?id=37'>Spoke.com scam</a><li><a href='?id=80'>Comment spam defeated at last</a></ul>
Boring Date (comic)
tag:smhanovtechblog,2007:id86
2009-10-26T08:00:00-05:00
2009-10-26T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=85">Previous Comic</a> | Next Comic </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20040227.png"></p>
It's a re-run, from before I used computerized lettering.
<p><ul><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=131'>[comic] Appreciation of xkcd comics vs. technical ability</a><li><a href='?id=65'>Drawing Graphs with Physics</a><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a></ul>
barcamp (comic)
tag:smhanovtechblog,2007:id85
2009-10-19T08:00:00-05:00
2009-10-19T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=84">Previous Comic</a> | <a href="?id=86">Next Comic </a></p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20091005.png"></p><ul><li><a href='?id=90'>Regular Expression Matching can be Ugly and Slow</a><li><a href='?id=52'>Automatically remove wordiness from your writing</a><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=82'>The PenIsland Problem: Text-to-speech for domain names</a><li><a href='?id=128'>Why you should go to the Business of Software Conference Next Year</a><li><a href='?id=116'>Fun with Colour Difference</a></ul>
How IE <canvas> tag emulation works
tag:smhanovtechblog,2007:id88
2009-10-15T08:00:00-05:00
2009-10-15T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
The <canvas> tag is the current fangled way of displaying vector graphics in a web browser. Before, all graphics were images, Flash animations, or even thousands of one-pixel <div>s. Finally, Internet browsers have <a href="http://en.wikipedia.org/wiki/Logo_%28programming_language%29#Syntax">caught up to the 1970s</a> and will be able to draw lines and curves programmatically, and you don't have to pay <a href="http://www.adobe.com/products/flash/">$699 USD</a> for the priviledge.
<p>
At the time of this writing, Internet Explorer at version 8.0 still lacks the <canvas> tag. But you can easily add the capability by including <a href="http://code.google.com/p/explorercanvas/source/browse/trunk/silverlight/excanvas.js?r=48">a short javascript file</a> in your page. At first glance, that's astounding. <b>How do you implement an entire vector graphics API in a few lines of Javascript?</b>
<p>
Actually, IE has had the ability to do vector graphics for years.
For IE 5.0, Microsoft was aware at how useful it would be, and also keenly aware of how long standardization takes, so they went ahead and implemented something called VML (Vector Markup Language), after it had been submitted to the standards process. SVG is simply VML (combined with some competing submissions) after it went through the process of being standardized.
<p>
For example, this code will draw an ellipse in VML.
<pre>
<style>v: * { behavior:url(#default#VML); display:inline-block }</style>
<xml:namespace ns="urn:schemas-microsoft-com:vml" prefix="v" />
<v:oval style="width:100pt;height:50pt" fillcolor="red">
</v:oval>
</pre>
<style>v: * { behavior:url(#default#VML); display:inline-block }</style>
<xml:namespace ns="urn:schemas-microsoft-com:vml" prefix="v" />
<v:oval style="width:100pt;height:50pt" fillcolor="red">
</v:oval>
<p>
In 2005,
<a href="http://me.eae.net/archive/2005/12/29/canvas-in-ie/">Emil A Eklund</a> created <a href="http://excanvas.sourceforge.net/">a simple translation layer</a> in Javascript that emulates canvas.
<blockquote>
IE doesn’t support SVG natively either, it does support something called VML though, and it’s been around since the 5.0 days, if I remember correctly.
VML does pretty much the same thing as SVG (as far as basic drawing is concerned).
<p>
Using VML, in combination with behaviors, it should therefor be possible to emulate a subset of SVG or Canvas in IE. That’s the idea I got a few days ago when working on a basic drawing abstraction, and according to Google a few others have thought along those lines as well. Couldn’t find any actual implementation though so I decided to make my own, how hard could it be?
<p>
If such implementation could be created it would open up the world of client side drawing to web site developers and allow all kind of neat widgets to be developed.
<p>
</blockquote>
So there is it. The ExplorerCanvas script isn't magic. It inserts VML tags into your web page when you call its various drawing routines.
<p>
<ul><li><a href='?id=119'>Throw away the keys: Easy, Minimal Perfect Hashing</a><li><a href='?id=131'>[comic] Appreciation of xkcd comics vs. technical ability</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=50'>Why Perforce is more scalable than Git</a><li><a href='?id=90'>Regular Expression Matching can be Ugly and Slow</a></ul>
I didn't know you could mix and match (comic)
tag:smhanovtechblog,2007:id84
2009-10-12T08:00:00-05:00
2009-10-12T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=83">Previous Comic</a> | <a href="?id=85">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20091004.png"></p><ul><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=120'>Succinct Data Structures: Cramming 80,000 words into a Javascript file.</a><li><a href='?id=52'>Automatically remove wordiness from your writing</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=117'>Why don't web browsers do this?</a><li><a href='?id=85'>barcamp (comic)</a><li><a href='?id=50'>Why Perforce is more scalable than Git</a></ul>
Sign here (comic)
tag:smhanovtechblog,2007:id83
2009-10-05T08:00:00-05:00
2009-10-05T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=78">Previous Comic</a> | <a href="?id=84">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20091003.png"></p><ul><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=55'>How wide should you make your web page?</a><li><a href='?id=75'>Pitching to VCs (comic)</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=71'>Using the Acer Aspire One as a web server</a><li><a href='?id=68'>When programmers design web sites (comic)</a></ul>
It's a dirty job... (comic)
tag:smhanovtechblog,2007:id78
2009-08-24T08:00:00-05:00
2009-08-24T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=77">Previous Comic</a> | <a href="?id=83">Next Comic </a></p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20090722.png"></p><ul><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=135'>How I run my business selling software to Americans</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a><li><a href='?id=97'>Creating portable binaries on Linux</a><li><a href='?id=120'>Succinct Data Structures: Cramming 80,000 words into a Javascript file.</a><li><a href='?id=50'>Why Perforce is more scalable than Git</a></ul>
The PenIsland Problem: Text-to-speech for domain names
tag:smhanovtechblog,2007:id82
2009-08-20T08:08:08-05:00
2009-08-20T08:08:08-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
"expertsexchange.com" is a domain name that can be read in multiple, unintended ways. Howshouldatexttospeechsystemresolvethisambiguity?
<p>
Recently, I was contracted to run a list of domain names through the custom-built pronunciation engine that powers my <a href="http://stevehanov.ca/rhyme">rhyming web site</a>. On the first attempt, I found that the results were embarrassingly bad. A quick inspection revealed the problem: most domain names are severalwordsstucktogether.
<p>
When a <a href="http://scholar.google.ca/scholar?q=pronunciation+by+analogy&hl=en&btnG=Search">pronunciation by analogy</a> system encounters an unknown word, it searches its knowledge base for words that look similar, and tries to stitch together their pronunciations. In this case, it was doing just what it was supposed to do. For example, lots of words end with an 'e', and usually that 'e' is silent when at the end of a word. But stick another word on, and the system would try to pronounce the 'e', just like a six-year-old learning to read by sounding out each letter. Most people, on the other hand, would recognize the two words and say them each individually.
<p>
Try these domains in the AT&T text to speech system, which many consider to be the best in the world, at <a href="http://www.research.att.com/~ttsweb/tts/demo.php">http://www.research.att.com/~ttsweb/tts/demo.php</a>.
<ul>
<li>thepiratebay.com (sounds like <i>separately</i>?)
<li>mydreamcloset.com (huh?)
<li>torrentspy.com (sounds like a polish name)
<li>123greetings.com (AT&T is ridiculous with this one)
</ul>
This world-class system mispronounces them all, even when given the huge hint of the ".com" at the end.
<p>
Time for a bit of <a href="?id=9">dynamic programming</a>. After finding an appropriate scoring function, we can break up text the same way a human reader would. We also use some simple heuristics to say numbers properly.
<p>
Although I don't have a speech synthesizer, you can check the raw pronunciation output using this form. The phonemes correspond to the ones in the <a href="http://www.speech.cs.cmu.edu/cgi-bin/cmudict">CMU pronouncing dictionary.</a>
<p>
<script type="text/javascript">
function makeAjaxRequest( strUrl, params, fnCallBack, param )
{
var self = this;
try {
self.xmlHttpReq = new XMLHttpRequest();
} catch ( trymicrosoft ) {
try {
self.xmlHttpReq = new ActiveXObject("Msxml2.XMLHTTP");
} catch(othermicrosoft) {
try {
self.xmlHttpReq = new ActiveXObject("Microsoft.XMLHTTP");
} catch(failed) {
self.xmlHttpReq = null;
}
}
}
self.xmlHttpReq.open('GET', strUrl, true);
self.xmlHttpReq.onreadystatechange = function() {
if (self.xmlHttpReq.readyState == 4) {
if ( fnCallBack ) {
fnCallBack( self.xmlHttpReq.responseText, param );
}
}
}
self.xmlHttpReq.send( null );
}
function onAjaxDone( response, param )
{
var result = eval("("+response+")");
var text = result.split + "n" + result.pron + "n";
$("#result").text(text);
}
function onsubmitmyform()
{
var wordx = document.getElementById("word").value;
makeAjaxRequest(
"http://stevehanov.ca/cgi-bin/talk?getpron=1&domain=1&word=" + wordx,
null,
onAjaxDone,
wordx
);
}
</script>
<div>
<form action="javascript:onsubmitmyform()">
<input id=word type="text" size="60" name="word" value="thepiratebay" />
<input type=submit value="Lookup" /><br>
</form>
<pre id="result">
</pre>
</div>
<ul><li><a href='?id=134'>0, 1, Many, a Zillion</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=56'>How a programmer reads your resume (comic)</a></ul>
Pitching to VCs #2 (comic)
tag:smhanovtechblog,2007:id77
2009-08-17T08:00:00-05:00
2009-08-17T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=76">Previous Comic</a> | <a href="?id=78">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20090721.png"></p><ul><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=85'>barcamp (comic)</a><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=116'>Fun with Colour Difference</a></ul>
Building a better rhyming dictionary
tag:smhanovtechblog,2007:id81
2009-08-13T08:00:00-05:00
2009-08-13T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Back in 2007, I created a <a href="http://stevehanov.ca/blog/index.php?id=8">rhyming engine</a> based on the public domain <a href="http://icon.shef.ac.uk/Moby/mpron.html">Moby pronouncing dictionary</a>. It simply reads the dictionary and looks for rhyming words by comparing the suffix of the words' pronunciations. Since that time, I have made some improvements.
<p align=center>
<a href="http://rhymebrain.com">
<img src="kwijibo.png" alt="rhyme any word" /></a>
</p>
<p>
Using a comnbiation of techniques from artificial intelligence, math, and linguistics, the rhyming engine can now figure out how to say any word that you enter. That means if you enter a word that is not in the dictionary, it will still be able to find some rhymes.
<p>
Rather than looking for technically perfect rhymes, it suggests words that would sound good together in song or poetry. For example, we sometimes ignore consonants, as suggested by this <a href="http://acl.ldc.upenn.edu/P/P85/P85-1034.pdf">1985 paper</a>. That way, <i>fervently</i> will rhyme with <i>urgently</i> despite the v/g mismatch.
<p>
There is a legal advantage to this technique as well. Many of the standard word lists used by natural language processing researchers include words from an old edition of the Oxford dictionary, and so <a href="ftp://svr-ftp.eng.cam.ac.uk/pub/comp.speech/dictionaries/beep-1.0.README">cannot be used for "commercial purposes"</a>. That's why both <a href="http://www.rhymezone.com">Rhymezone</a> and <a href="http://www.rhymer.com">Write Express</a> have a relatively limited dictionary size. My rhyming engine can sidestep this issue, since it only needs to be seeded with a small number of words from unrestricted sources, and it can then import words in bulk, and guess the pronunciations without using any restricted content.
</p>
<p>
I couldn't resist doing some premature optimization. It uses one of my favourite data structures -- <a href="http://en.wikipedia.org/wiki/Trie">the trie</a>. The program starts, reads the entire 260,000 word database, and completes in 60 ms on my <a href="images/acer.jpg">netbook web server</a>. It takes about 8 MB of memory. I guess that equates to about 0.48 mega-byteseconds per request.
<!--
<h2>Denouement</h2>
<p>
I leave you with a major inspiration for this project, from the Star Trek episode, "Schisms".
<p align=center>
<b>Ode to Spot</b>
<br>
Felis Cattus, is your taxonomic nomenclature,<br>
an endothermic quadruped carnivorous by nature?<br>
Your visual, olfactory and auditory senses<br>
contribute to your hunting skills, and natural defenses.<br>
<br>
I find myself intrigued by your subvocal oscillations,<br>
a singular development of cat communications<br>
that obviates your basic hedonistic predilection<br>
for a rhythmic stroking of your fur, to demonstrate affection.<br>
<br>
A tail is quite essential for your acrobatic talents;<br>
you would not be so agile if you lacked its counterbalance.<br>
And when not being utilized to aide in locomotion,<br>
it often serves to illustrate the state of your emotion.<br>
<br>
O Spot, the complex levels of behaviour you display<br>
connote a fairly well-developed cognitive array.<br>
And though you are not sentient, Spot, and do not comprehend,<br>
I nonetheless consider you a true and valued friend.<br>
</p>
-->
<h2>Why is this hard?</h2>
Text to speech for English is still a hard problem to solve, and it is an active area of research. Consider the words rough, through, bough, thought, dough, cough, or photOgraph, photOgraphy, or physics, lymphatic, and loophole. In the 80's, and still today in many cases, text to speech is done by hiring specially trained linguists to develop the thousands of rules necessary to create pronunciations. It is only in the last 10 years or so that this task has been automated. My system has over 200,000 hints on how to interpret each part of a word given its context. With further refinements, this could probably be reduced to tens of thousands, which is still a lot.
<h2>Further reading</h2>
<ul>
<li><a href="?id=52">Automatically remove wordiness from your writing</a>
<li><a href="?id=9">What does your phone number spell?</a>
<li><a href="?id=63">Keeping abreast of pornographic research in computer science</a>
<li><a href="?id=22">Exploring sound with wavelets</a>
</ul>
<p>
<ul><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=132'>20 lines of code that will beat A/B testing every time</a><li><a href='?id=65'>Drawing Graphs with Physics</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a></ul>
Does Android team with eccentric geeks? (comic)
tag:smhanovtechblog,2007:id76
2009-08-10T08:00:00-05:00
2009-08-10T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=75">Previous Comic</a> | <a href="?id=77">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20090719.png"></p><ul><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=60'>Is 2009 the year of Linux malware?</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=52'>Automatically remove wordiness from your writing</a></ul>
Comment spam defeated at last
tag:smhanovtechblog,2007:id80
2009-08-06T08:00:00-05:00
2009-08-06T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
For years when running this blog, I would have to log in each day and delete a dozen comments due to spam. This was a chore, and <a href="http://stevehanov.ca/blog/index.php?id=18">I tried many ways</a> to stem the tide.
<p>
Finally, a few months ago, I found a way that worked 100% of the time. <a href="comments.txt" rel="nofollow">This raw text file</a> shows what I'm up against, containing all server variables and full text of every comment I've gotten in the last couple of months.
<p>
Here's the code for the comment form below. Can you spot my solution? (No, it's not the "http:" part, which almost worked)
<p>
<pre>
<div class="roundedcornr_top_473174"><div></div></div>
<div class="roundedcornr_content_473174">
<div id=commentBox class=commentBox>
<div class=pad>
<h2>Post comment</h2>
<form action="/blog/index.php" method=POST onsubmit="return validateCommentForm(this);">
Real Name: <input type=text name=displayname /><br>
<span style="visibility:hidden"> Your Email (Not displayed): <input type=text name="email"/></span><br>
Text only. No HTML. If you write "http:" your message will be ignored.
<br>
<textarea cols=60 name=comment rows=10 wrap=soft ></textarea><br>
<input type=submit value="Post" />
<input type=hidden name=id value="75">
</form>
</div>
<div class=comment>
</div>
</div>
</div>
</pre><ul><li><a href='?id=76'>Does Android team with eccentric geeks? (comic)</a><li><a href='?id=66'>Test Driven Development without Tears</a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=120'>Succinct Data Structures: Cramming 80,000 words into a Javascript file.</a><li><a href='?id=10'>Detecting C++ memory leaks</a><li><a href='?id=53'>cairo blur image surface</a></ul>
Pitching to VCs (comic)
tag:smhanovtechblog,2007:id75
2009-08-03T08:00:00-05:00
2009-08-03T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=74">Previous Comic</a> | <a href="?id=76">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20090712.png"></p>
<div style="font-size:smaller">
May contain characters from <a href="http://www.xkcd.com/">xkcd</a>, used under <a href="http://creativecommons.org/licenses/by-nc/2.5/">license</a>. However, for this work, all rights are reserved. Coyright © 2009 Steve Hanov
</div><ul><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=11'>Cell Phones on Airplanes</a><li><a href='?id=90'>Regular Expression Matching can be Ugly and Slow</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a></ul>
How QBASIC almost got me killed
tag:smhanovtechblog,2007:id79
2009-07-30T08:00:00-05:00
2009-07-30T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Back in high school, I had too much free time, so I decided to play a joke on my computer teacher. I created <a href="dos.bas">an exact clone of the school's DOS system</a> using QBasic. It would pretend to execute three commands: DIR, DEL *.*, and FORMAT.
The simulation was so realistic that during development, I was kicked out of the lab. Usually students would be playing <a href="http://www.dosgamesarchive.com/download/secret-agent-the-hunt-for-red-rock-rover/">Secret Agent</a> or <a href="http://en.wikipedia.org/wiki/Jill_of_the_Jungle">Jill of the Jungle</a>.
<p>
The day arrived when my project was ready to be unleashed upon the world. I waited until the teacher was hovering nearby and then I started my application, running the FORMAT command on the network drive. Some classmates were watching the screen and she hurried over to see what all the fuss was about.
<p align=center><img src="dos.png" alt="DOS screen formatting a disk"></p>
<p>
The reaction was immediate. She stared at the screen, eyes wide open, and mouth agape, as the terrible seconds ticked by. <b>At that moment I regretted my deception and tried to abort the demo.</b> But QBasic didn't understand CTRL-C during the SLEEP command. Pressing CTRL-C just interrupted the current SLEEP, so <i>it caused the percentage to advance faster</i>. I had to hold down the abort keys and wait until it advanced to 100% before I could prove that everything was really okay.
<p>But then it said:</p>
<div style="background:black;color:#C0C0C0;font-family:terminal,lucida console,monospace,courier;font-size:12px">
Unable to read from drive X:<br>
Abort, Retry, Fail? <blink>_</blink>
</div>
<p>
That was the closest I've ever come to being murdered.
</p>
<ul>
<li><a href="dos.bas">Source code for Fake DOS</a>
<li>My QBASIC implementations of <a href="http://www.powerbasic.com/support/downloads/files/PacMan.zip">Pacman</a> and <a href="http://jlounge.classicgaming.gamespy.com/other/jman-bas.zip">Jumpman (with EXE, use speed 170)</a>
<li>My <a href="http://stevehanov.ca/blog/?id=92">QBASIC implementation</a>
</ul>
<ul><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=8'>A Rhyming Engine</a><li><a href='?id=75'>Pitching to VCs (comic)</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a></ul>
Blame the extensions (comic)
tag:smhanovtechblog,2007:id74
2009-07-27T08:00:00-05:00
2009-07-27T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=72">Previous Comic</a> | <a href="?id=75">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20090711.png"></p><ul><li><a href='?id=146'>O(n) Delta Compression With a Suffix Array</a><li><a href='?id=55'>How wide should you make your web page?</a><li><a href='?id=9'>What does your phone number spell?</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a></ul>
How to run a linux based home web server
tag:smhanovtechblog,2007:id73
2009-07-23T08:00:00-05:00
2009-07-23T08:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
There are plenty of places you can go if you just want to put up some static web pages for free, or very low cost. But costs go up very quickly if you need to do any more than that, or if you get spikes in traffic. Sometimes you need complete control over the server, and don't want to pay $20 to $40 a month for a VPS. In this article, I'll describe step by step how to set up a home web server using Ubuntu, capable of handling modest spikes in traffic.
<p>
There are several things you need:
<ul>
<li>Connection
<li>Hardware
<li>Domain name
<li>DNS server
<li>Web server software properly configured for your system
</ul>
<h2>The Connection</h2>
<p>
Many large internet providers forbid running any servers in their usage agreement. They might not actually check for a long time, but it's no fun to have your Internet cut off without warning after months with no problems. You can only do this if you are lucky enough to find an ISP that allows servers. In Canada, try <a href="http://www.teksavvy.ca">Teksavvy</a>.
<p>
You will want as high an uplink speed as you can get. Unless your website is extremely popular, bandwidth caps will not be an issue, but be aware of what they are.
<h2>The Hardware</h2>
<p>
Over your home internet connection, traffic will be naturally smoothed by the low uplink bandwidth. Incoming requests will be queued and served in order, not all at once. CPU speed is not going to be an issue, and a one GHz machine is plenty of power. Although this blog's web server has only 512 MB of memory, I recommend at least 1 GB of RAM.
<p>
Your router will also be an important part of the system, but quality varies. I have a Linksys WRT54GL. I downloaded and installed <a href="http://www.dd-wrt.com">dd-wrt</a> on it. The new firmware unlocks a lot of hidden features, including dynamic DNS updating. However, this is not strictly necessary, because your linux system can handle this step just as easily.
<h2>Domain name</h2>
<p>
You can get a domain name for 10 to 15 dollars. Try DomainsAtCost or GoDaddy. After you buy one from a registrar, you will get an account with them. After logging in, you can renew the domain name. But the most important priviledge is to be able to set the name server. That will have to wait until the next step...
<p>
<img src="http://upload.wikimedia.org/wikipedia/commons/thumb/2/24/Warning_icon.svg/420px-Warning_icon.svg.png" width=40 align=left style="margin:5px">A few months after getting your domain name you will start getting emails from companies claiming they have registered it in another country, and offering to sell it back to you. You can safely ignore these scams.
<h2>DNS server</h2>
<p>
Because your IP address will often change, you will need a dynamic DNS updating service. DynDns.com provides very good service, with updates taking effect within seconds. However, they do charge a fee. <a href="http://www.zoneedit.com">ZoneEdit</a> provides a free service for up to five domains, although updates can take several minutes to propagate.
<p>
After signing up with a dynamic DNS updating service, you will get:
<ul>
<li>a username and password
<li>IP addresses of nameservers
</ul>
Save the username and password for later. Go back to your domain name registrar and enter the nameservers into their system.
<p>
<h2>Configuring your server</h2>
Install the latest Ubuntu Desktop edition, and then add the additional software you will need for web serving:
<pre>
sudo apt-get install mysql-client mysql-server php5 apache2 php5-mysql ddclient
</pre>
<p>
You will be prompted several times for your mysql root password. This is not the same as your linux root password. As long as you are behind a firewall, it is okay to press enter and leave it blank.
<p>
Now check that your web server works using a web browser browsing to <a href="http://localhost">http://localhost</a>. Apache should show a default page telling you that it works.
<h2>Fixing ddclient</h2>
DDclient is a program that continually monitors your true IP address. When it changes, it updates your dynamic DNS service so that your domain name will resolve to your home web server.
<p>
Unfortunately, the version of ddclient that you can install on Ubuntu 9.04 (and most possibly later ones) is screwed up. Lets fix it.
<pre>
wget http://downloads.sourceforge.net/sourceforge/ddclient/ddclient-3.8.0.tar.gz
tar xvzf ddclient-3.8.0.tar.gz
sudo cp ddclient-3.8.0/ddclient /usr/sbin/ddclient
sudo ln -s /etc/init.d/ddclient /etc/rc2.d/S99ddclient
</pre>
<p>
Edit /etc/ddclient/ddclient.conf. If you use zoneedit, the file should look like this. Replace username and password and domain with your true values.
<pre>
protocol=zoneedit1
use=web
server=dynamic.zoneedit.com
login=yourlogin
password='yourpassword'
yourdomain.com
</pre>
<p>
Now erase the ddclient cache and restart it.
<pre>
sudo rm /var/cache/ddclient/*
sudo /etc/init.d/ddclient stop
sudo /etc/init.d/ddclient start
</pre>
<p>
After a few minutes you should be able to get to your web server by entering your domain name. If things aren't working, running 'sudo ddclient --verbose' will help you figure out what it's doing.
<h2>Configuring Apache2</h2>
There is one last thing. Because your machine has limited memory and bandwidth, you will need to set some parameters in /etc/apache2/apache2.conf.
<p>
The <b>KeepAlive</b> setting keeps your server busy for a few seconds after a page is served, allowing clients to download images without starting a new connection. The default setting is far too long, and it would cause your server to choke under heavy load. Set KeepAliveTimeout to 1 or 2 seconds.
<p>
The <b>MaxClients</b> setting determines how many connections your web server can handle at one time. Each connection takes around 5 megabytes of unshared RAM, so you will have to set this value taking into account the amount of RAM on your machine, while leaving room for other system processes. For 1 GB, a nice safe value is 130, but your mileage may vary. If your server becomes unresponsive, reboot it and lower this number.
<p>
After any change to server settings, use "sudo apache2ctl restart" to safely restart everything. Your users won't notice a thing.
<h2>Optimize for bandwidth</h2>
When your bandwidth is limited, the number of visitors you can handle is directly proportional to size of the page, and all the images. If you have 120 KB/s of uplink bandwidth, and your page is 120 KB in size, then you can handle no more than one visitor per second. The single most important thing you can do is make your images as small as possible. Use low quality JPG for photos, and 32 or 256 colour PNG files for everything else.
<p>
Type 'sudo a2enmod deflate' and restart apache to allow your HTML, scripts, and css files to be transmitted compressed.
<p>
If you use common Javascript libraries, use the <a href="http://code.google.com/apis/ajaxlibs/documentation/index.html#jquery">Google hosted versions of them.</a>
<p>
Include all your own CSS and Javascript in the same file as your HTML, so that only one page needs to be requested. During times of heavy load, 99.9% of your visitors only load a single page of your web site anyway, so it makes no sense to split things up into different files.
<h2>In conclusion</h2>
If you need 99.999% reliability, you should look for a hosted VPS solution. However, for modest needs, you can achieve 98.9% reliability for no cost other than what you are paying your internet provider.
<h2>Further reading...</h2>
<ul>
<li><a href="http://stevehanov.ca/blog/index.php?id=71">Using the Acer Aspire One as a web server</a>
<li><a href="http://stevehanov.ca/blog/index.php?id=17">Run your own Web-Comic Aggregator</a>
</ul>
<p>
</p><ul><li><a href='?id=96'>Bending over: How to sell your software to large companies</a><li><a href='?id=26'>A simple command line calculator</a><li><a href='?id=71'>Using the Acer Aspire One as a web server</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a><li><a href='?id=144'>Finding Bieber: On removing duplicates from a set of documents</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=75'>Pitching to VCs (comic)</a></ul>
Microsoft's generosity knows no end for a year (comic)
tag:smhanovtechblog,2007:id72
2009-07-19T14:24:53-05:00
2009-07-19T14:24:53-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center><a href="?id=68">Previous Comic</a> | <a href="?id=74">Next Comic </a></p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="../comics/comic_20090720.png"></p><ul><li><a href='?id=13'>Draw waveforms and hear them</a><li><a href='?id=73'>How to run a linux based home web server</a><li><a href='?id=127'>Four ways of handling asynchronous operations in node.js</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=7'>Rules for Effective C++</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a></ul>
Using the Acer Aspire One as a web server
tag:smhanovtechblog,2007:id71
2009-07-18T12:08:19-05:00
2009-07-18T12:08:19-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<img src="images/acer.jpg" align=right>
My web site has been going up and down over the night. I've intentionally been trying to elicit reddit traffic so I can test different parameters for apache server optimization, to handle high traffic over slow connections.
<p>
A netbook can be ideal for a home web server. They are cheap, and use less power
than a CFL light bulb. The only trade-off is that it won't reboot on a power failure. Fortunately, the built in UPS will sustain it through all but the longest power outages.
<p>
My <a href="?id=29">Via Artigo</a> died several weeks ago while I was using it. I have no air conditioner, so it must have been about 30 degrees which may have contributed to its demise.
I took it apart and poked at it, but the monitor and disk light wouldn't light
up. So ended up getting a <a href="http://www.canadacomputers.com/index.php?do=ShowProdList&cmd=pl&id=896.862.656&mfg=all&sort=3">$179</a> refurbished Acer Aspire One, which is now
happily serving up this web site.
<p align=center><img src="images/artigo.jpg"> </p>
<p>
<h3>Specs</h3>
<ul>
<li>1.6 GHz Atom processor (with hyperthreading)
<li>512 MB memory
<li>8 GB SSD
<li>Wifi, ethernet, VGA ports
<li>3 USB ports and 2 SD card slots (?!)
</ul>
<h2>Initial experiences</h2>
<p>
I turned it on briefly and it had a linux based OS on it that boots up in a
less than 5 seconds and looks confusingly like Windows XP. Enough about that.
<p>
Because I only install Ubuntu on <a href="?id=48">USB keys</a>, I was able to
plug it in, boot up, and my web site was up and running in only minutes.
<p>
The fan is the quietest I have ever heard on any laptop. It makes almost no
sound at all.
<h3>Lousy performance as a desktop system</h3>
<p>
My old Artigo had a Via 1 GHz CPU and 1 GB of memory, and it could be used as a desktop system. Not so with the Acer. Even with the increased
processor power, the decrease in RAM eats up all the benefit. After installing
Xubuntu, the system is too sluggish to handle common desktop tasks.
It badly needs a RAM upgrade, but this <a href="http://www.youtube.com/watch?v=-EfzckyZMTk">requires you to completely disassemble it.</a>.
<p>
The tiny speakers are junk. Imagine listening to a movie through ear-buds one metre away. That's how bad it is. But it doesn't matter, because it's a server.
<p>
<h3>Handling Server Load</h3>
<p>
With Ubuntu 9.04, after a day of serving up moderate web activity, the Linux Atheros
Wifi driver was stuck in a bad state so it was offline. I switched it to a
plug-in cable and haven't had any more issues.
<p>
The system went non-responsive an hour after I posted my last comic strip. It
had a blank screen and needed rebooting. However, I hadn't
optimized apache2 yet. Using the approriate settings, I believe that it can handle about
between 30 and 40 visits/minute, or anything <a
href="http://www.reddit.com/r/programming/">reddit</a> and <a
href="http://news.ycombinator.com">ycombinator</a> can throw at it. I'm still tweaking with the apache parameters, and I'll post something about them soon.
<h3>Things to watch out for when running a home web server</h3>
<ul>
<li>Ensure that MaxClients is set to a reasonable value (memory divided by 7 MB?) in your apache2.conf file. If you come home to find your machine non-responsive and thrashing, this is the cause. To be ultra-safe, you can flip the "KeepAlive" setting to Off, but your pages will load more slowly.
<li>If serving over a dynamic IP address, make sure your machine is configured to automatically update it. I installed ddclient to do this, but on Ubuntu it wasn't automatically restarting after a reboot.
<li>As always, be careful with large images. Save photos as low quality JPGs, and screenshots as 16-colour .PNG files. Don't put up anything over 50 kb.
<li>The limiting factor is bandwidth, not CPU. Enable gzip compression using 'a2enmod deflate'
</ul>
<ul><li><a href='?id=13'>Draw waveforms and hear them</a><li><a href='?id=123'>Zero load time file formats</a><li><a href='?id=137'>Asana's shocking pricing practices, and how you can get away with it too</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=91'>You don't need a project/solution to use the VC++ debugger</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=98'>Asking users for steps to reproduce bugs, and other dumb ideas</a></ul>
When programmers design web sites (comic)
tag:smhanovtechblog,2007:id68
2009-07-13T07:19:52-05:00
2009-07-13T07:19:52-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align="center"><a href="?id=70">Previous Comic</a> | <a href="?id=72">Next Comic</a> </p>
<p align="center">
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img src="http://imgur.com/Z7ZzK.png"></p>
<ul><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=127'>Four ways of handling asynchronous operations in node.js</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=66'>Test Driven Development without Tears</a><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a></ul>
Finding great ideas for your startup
tag:smhanovtechblog,2007:id70
2009-07-08T21:36:26-05:00
2009-07-08T21:36:26-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align="center"><a href="?id=56">Previous Comic</a> | <a href="?id=68">Next Comic</a> </p>
<p align=center>
<span style="font-family:impact;font-size:20px">STARTUP INK</span><br>
<img border=0 src="../comics/comic_20090708.png">
</p>
<p>
<i>"I just don't have any ideas."</i> This is the #1 stumbling block for budding entrepreneurs. Here are a few techniques to get the creative juices flowing.
<h3>Copy somebody else, but fix the problems</h3>
<p>
If you have a lack of ideas, you might be mentally discarding lots of things. Ideas don't have to sound good to be successful. If you take a look at this <a href="http://ycombinator.com/ideas.html">list of ideas from 2008</a>, none of them are original or earth shattering.
<p>
If somebody's already doing it, don't let that stop you. Instead, do it better! For years, the technical question and answer site that came up most often in searches was <a href="http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/MS-SharePoint/Q_24542014.html#a24771974">Experts Exchange</a>. But it obnoxiously demands payments all the time, and is notoriously hard to read. Last year, <a href="http://stackoverflow.com">StackOverflow</a> popped up, with its large, user-friendly fonts and zero costs, and quickly rose to popularity.
<h3>Copy an idea and fill a niche</h3>
<p>
Ebay is a one stop shop for buying anything online. You could certainly <a href="http://reviews.ebay.com/Listing-and-Selling-Gift-Cards-within-eBay-Guidelines_W0QQugidZ10000000001677562">sell gift cards on Ebay</a>, but why go to the trouble when you can sell through <a href="http://www.giftah.com/">Giftah</a>, a whole site devoted to pawning unwanted gift cards. The founders of Giftah were unfazed by the many existing options, and built their site anyway.
<h3>Implement ideas from academia</h3>
There is a clear separation between the practical world of business, and the "publish or perish" world of academia. The research journals are filled with ideas that are ripe for the plundering. Entire branches of research from decades ago were simply dropped because they were a solution without a problem, or the author achieved tenure. <a href="http://scholar.google.com">Start here,</a> and you might find something useful.
<p>
This is one of the harder paths of entrepreneurship. Still, if you're smart enough to develop a complete understanding of a complex subject, and translate it into something useful and practical, you can turn obscurity into opportunity. Google did it with PageRank. More recently, <a href="http://facemining.pittpatt.com/">Pittsburgh Pattern Recognition</a> is a company that went this route, taking state of the art facial mining software and applying it to old episodes of Star Trek as an effective demonstration of their technology.
<h3>Solve a business problem</h3>
<p>
Are you selling to consumers? It's a great start because you're a consumer and you know what you like, and its nice to build something that your friends can use. But many startups quickly switch gears and sell their services to businesses instead. Why? Simple scaling. You have to be big to handle sales marketing support, while still finding the resources to develop your core technology. Dealing with ten thousand customers requires a lot more infrastructure than dealing with ten.
<p>
<a href="http://en.wikipedia.org/wiki/Quack.com">Quack.com</a> started off with the dream running a phone portal, to let people call in and get their email, news, and other information using voice recognition. But their first paying customer was the Lycos search engine, and later on they were bought by America Online, which gave them the resources to scale. (Unfortunately things didn't work out under AOL's direction.)
<p>
Businesses expect a more complete solution. For example: Kelly sells popcorn to consumers in a mall for $3 a bag, works 8 hours a day, and after rent makes $200 a day. "Grandpa" Joe sells popcorn to business -- for $500 he'll cart his popcorn machine to your corporate event. Over three hours, he'll make cheery conversation while doling out the yellow goodness, and clean up when he leaves, and all it takes is a phone call.
<p>
To jumpstart things, use the <a href="http://en.wikipedia.org/wiki/Fortune_cookie#In_popular_culture">fortune cookie trick</a>. Take an idea that is traditionally marketed to consumers, and add "to business" after it. It doesn't take much imagination to figure out a way that a consumer business can be converted to a B2B.
<p>
<table border=1">
<tr><th>We provide...</th><th></th></tr>
<tr><td>
<ul>
<li>social networks
<li>video and photo sharing
<li>contact management
<li>news and blog post aggregation
<li>greeting cards
<li>online training
</ul>
</td>
<td valign=middle>
to business</td>
</tr>
</table>
<h3>Do what you know</h3>
<a href="http://aliasaria.ca/blog/">Ali "Brickbreaker" Asaria</a> is the founder of <a href="http://www.well.ca">Well.ca</a>, Canada's online drugstore, and the son of a pharmacist. Ali noticed that the market for health and beauty products was under-served -- many other online drugstores seem focused on drugs only. He created an extensive business plan, built a web site, sought funding, and now runs a successful company.
<h3>Be flexible</h3>
<p>
The odds are that you will end up building something completely different from what you started with. Mike Lazaridis founded Research in Motion, the creator of BlackBerries, in 1984 to do custom hardware development. An early product was a digital barcode system for Hollywood film editing. Later, they contracted with another company to do <a href="http://www.berryreview.com/2009/02/12/the-history-of-rim-the-blackberry-smartphone-part-1-the-origins/">wireless point of sale terminals</a>. That worked out pretty well, and they were left with a bunch of radio modems laying around the office. The rest is history.
<h2>Start your engines...</h2>
Ideas by themselves are of little worth. It is the hard work, personal risk, sleepness nights, plus a lot of luck and networking that has real value and can help you be successful. How many hundreds of creative minds used flash to stream video, or created tiny video streaming websites, before someone went and built YouTube? Having an idea is the easy part. The real issue is finding the spark of excitement and drive to follow through with it, even when the world is saying you will fail.<br>
<h3>Further reading...</h3>
<ul>
<li><a href="http://www.amazon.com/gp/product/1430210788?ie=UTF8&tag=stehanstecblo-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1430210788">Founders at Work: Stories of Startups' Early Days</a><img src="http://www.assoc-amazon.com/e/ir?t=stehanstecblo-20&l=as2&o=1&a=1430210788" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> - Learn from the experts. This book has tales of successes and lots of failures from lots of entrepreneurs, including the founders of Paypal, Hotmail, and Del.icio.us. The tale of Delicious is especially unusual, because it was just one guy working in the evenings.
<li><a href="http://www.amazon.com/gp/product/0060851139?ie=UTF8&tag=stehanstecblo-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0060851139">Innovation and Entrepreneurship</a><img src="http://www.assoc-amazon.com/e/ir?t=stehanstecblo-20&l=as2&o=1&a=0060851139" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> - According to a commenter, I am <i>copying many of my ideas</i> from Peter Drucker, and this book has many more along the same lines.
</ul>
<ul><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=31'>Free, Raw Stock Data</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a></ul>
Game Theory, Salary Negotiation, and Programmers
tag:smhanovtechblog,2007:id67
2009-06-25T22:41:12-05:00
2009-06-25T22:41:12-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<div style="border:2px solid red"><i>Disclaimer</i>: Use these tips at your own risk. Don't get career advice from bloggers.</div>
<p>
When you get a new job, you can breathe a sigh of relief, but not for long. You have an offer letter in your hand, and it is easy to miss one of the most important opportunities of your life: the starting salary. Here's the tale of two programmers. When getting a job, Goofus didn't negotiate. Gallant asked and got an extra $2500. They both get yearly raises of 3%
<center>
<table border=1>
<tr><th>Year</th><th>Goofus</th><th>Gallant</th></tr>
<tr><td>1</td><td>45000</td><td>47500</td></tr>
<tr><td>2</td><td>46350</td><td>48925</td></tr>
<tr><td>3</td><td>47740</td><td>50392</td></tr>
<tr><td>4</td><td>49172</td><td>51904</td></tr>
<tr><td>5</td><td>50647</td><td>53461</td></tr>
</table>
</center>
<p>
After five years, Gallant has made an extra $13272, enough to get his car paid off, or keep his Macbook software up to date.
<p>
Goofus is in prison because he had to become a spam lord to pay child support for his six kids.
<p>
Everything in life is negotiable. C.E.O.s and corporate executives are simply people that learned this at an early age. The things that are most negotiable are the things written in black and white in indelible ink. They are engraved in silver, carved in stone, simply to trick you into thinking you cannot negotiate. <i>"Just sign here. It's a formality." "It's a preprinted form, it can't be changed."</i>
<p>
Do not be intimidated. Salary negotiation is a game, and the first to give a number loses.
<h2>When it goes wrong?</h2>
<p>
A friend once confided in me one of the biggest mistakes of his life. He got a job in technology, and they asked him his salary expectations. Just being out of school, he gave them a very low number for the type of work, in the low 40s. Then the worst thing happened: They gave him the job, and his lowball salary. He felt unmotivated and ripped off. He wasn't working there long.
<p>
He broke the golden rule of salary negotiation. If you say a number, you lose. If you are asked on a form, leave it blank. If someone is pressing you for a number, just repeat: "I expect to be paid fairly based on my skills." The chances of you mentioning a number and getting it right are low.
<p>
Let's prove it using some <a href="http://en.wikipedia.org/wiki/Game_theory">game theory</a>. In this the rules of this game, there are two possible salaries: high and low. You and the company's recruiter both write your salary expectations separate cards, and then you each flip them over one at a time. Sounds simple, right? Here's the catch: The second person to go gets to change their vote after the first move. If both cards match, you get the job at the agreed upon salary. Here are all possible outcomes if the job applicant goes first:
<table>
<tr><th rowspan=3>Your expectations</th></tr><th colspan=3>Company's expectations</th></tr>
<tr><td></td><th>Low</th><th>High</th></tr>
<tr><th>Low</th><td>LL</td><td>LL (changed)</td><tr>
<tr><th>High</th><td>HL</td><td>HH</td><tr>
</table>
<p>
You only get the high salary 25% of the time, and in one unfortunate case (HL) they have security escort you from the building. Let's look at what happens if the company reveals their card first:
<table>
<tr><th rowspan=3>Company's expectations</th></tr><th colspan=3>Your expectations</th></tr>
<tr><td></td><th>Low</th><th>High</th></tr>
<tr><th>Low</th><td>LL</td><td>LL (changed)</td><tr>
<tr><th>High</th><td>HH (changed)</td><td>HH</td><tr>
</table>
<p>
When the company is the first to give a number, you always get the job, and you have a 50% chance of getting a high salary.
<h2>When it goes right</h2>
<p>
Another friend of mine was working in a job that didn't challenge her, so she applied around and was interviewed for a much better position. The problem was, it wasn't paying that much more than her old job. For the same money, she could keep her old job and play on the Internet much of the time.
<p>
She called the recruiter and said she would love the position and was very excited about it, and looking forward to the new challenges, except for the trifling little detail of the salary, wasn't there <i>anything</i> they could do about that? A few hours later she got the job -- at much higher pay.
<p>
The manager trying to hire you has chosen you over all the other candidates, and this give you the upper hand. In a larger company, the person you will be working for often has no idea what happens after the interview, since you are now dealing with an HR person. If you get away, this HR person has failed, and it will be disappointing to the manager. Use this to your advantage.
<h2>Final tips</h2>
<p>
The economy is turning around, and more jobs are popping up. The fact that you are reading this blog means that you have an interest in your craft, and this puts you in the top 10% of candidates. If you are on the hunt for a job, remember these tips:
<ul>
<li>Know what you are worth. Do salary research on <a href=http://monster.salary.com/>Monster.com</a> or <a href=http://salary.monster.ca/>Monster.ca</a>, or ask your friends if they have any information on pay scales.
<li>The opportunity to negotiate is <i>after you've been interviewed, but before you've accepted.</i> Don't even mention salary during the interview, unless they bring up the subject first. It is in your interest to postpone salary discussion as late as possible, after they are sure you are the best candidate.
<li>During or after the interview, do try to get a sense of whether you are the preferred candidate.
<li>Companies will rarely give you the opportunity to negotiate, or even bring up the topic of salary at all. You will probably have to call them about it. They will say no. Be persistent if you are confident in your skills.
<li>If you are first to mention a number, you are a loser.
<li>If the company mentions a number, then do the following: Repeat it back to them, then <a href="http://books.google.ca/books?id=yrDlOJE0tucC&lpg=PP1&pg=PA44">stare at them while counting to 30 in your head.</a> 90% of the time they will then give you a higher number.
<li>Do try to negotiate, even if the salary surpasses your wildest dreams.
<li>Even if your asking doesn't work right away, it may be remembered and lead to more perks later on.
</ul>
<h2>Further reading</h2>
<ul>
<li><a href="http://www.amazon.com/gp/product/1580087760?ie=UTF8&tag=stehanstecblo-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1580087760">Negotiating Your Salary: How To Make $1,000 A Minute 2006 Edition</a><img src="http://www.assoc-amazon.com/e/ir?t=stehanstecblo-20&l=as2&o=1&a=1580087760" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />: A thin and useful book on how to negotiate. I highly recommend it. There is nothing like this online so you'll have buy it from Amazon or get it from the library.
<li><a href="http://www.joelonsoftware.com/items/2008/11/26.html">Exploding offer season</a> - A dirty trick that all recruiters use on new grads.
<li><a href="http://gandolf.homelinux.org/~smhanov/blog/?id=56">How a programmer reads your resume (comic)</a>
<li><a href="http://www.four-pillars.ca/2009/07/21/salary-history/">Four Pillars</a> - What to do if asked about your current salary.
</ul>
<ul><li><a href='?id=101'>"Your program is stupid. It doesn't work," my wife told me</a><li><a href='?id=9'>What does your phone number spell?</a><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a></ul>
Coding tips they don't teach you in school
tag:smhanovtechblog,2007:id57
2009-06-23T09:45:19-05:00
2009-06-23T09:45:19-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align=center>
<a href="http://gandolf.homelinux.org/~smhanov/comics"><img border=0 src="../comics/comic_20040326.png"></a>
</p>
<p>Here are some C coding tips, because I have been unable to post anything for a while.
<div class=warn>Some of these time-saving shortcuts are intended for small projects or prototyping code.</div>
<h3>The nested ? : trick</h3>
The switch statement is very efficient, and the compiler will often implement it as a table lookup so it doesn't have to do any comparisons. But it sure can tire your fingers in a hurry:
<pre>
switch( number ) {
case 1: str = "one"; break;
case 2: str = "two"; break;
case 3: str = "three"; break;
case 4: str = "four"; break;
case 5: str = "five"; break;
default: str = "unknown number"; break;
}
</pre>
<p>
If getting the code out is more important than its speed, you can use a nested conditional operator to save typing:
<pre>
str = number == 1 ? "one" :
number == 2 ? "two" :
number == 3 ? "three" :
number == 4 || rand() == 42 ? "four" :
number == 5 ? "five" :
"unknown number";
</pre>
<p>
You can freak out your coworkers if you do it all on one line.
<h3>DeMorgan's theorem of negativity</h3>
<pre>
// if blah blah blah blah blah blah....
if ( !(A && B) )
</pre>
<p>
is the same as
</p>
<pre>
// if blah blah
if ( !A || !B )
</pre>
<p>
Pick the one that is easier to understand and read out loud. It is usually the second.
</p>
<h3>Rounding numbers using integer math</h3>
<p>
Suppose you have 2001 boolean variables, so you want to keep them in a bitmap of bytes, 8 at a time. How do
you declare this array?
</p>
<pre>
#define NUMBER_OF_BITS 2001
unsigned char bitmap[ NUMBER_OF_BITS / 8 ]; // FAIL!
</pre>
<p>
The computer uses integer math, so 2001 / 8 is 250 and there is one bit left over. When you store bit 2001,
you will corrupt memory. You can round numbers up like this:
<pre>
unsigned char bitmap[ (NUMBER_OF_BITS + 7) / 8 ];
</pre>
<p>
This works because integer division always rounds down. However, if you add the divisor less one, then you will force it to always round up. It will also work for 32 bit integers:
</p>
<pre>
unsigned int bitmap[ (NUMBER_OF_BITS + 31) / 32 ];
</pre>
<h4>Real rounding via integer division</h4>
Adding something before dividing is a general technique. This code rounds <tt>a</tt> and <tt>b</tt> to the nearest 10.
<pre>
int a = 12;
int b = 16;
printf("Round a: %d", (a + 5) / 10 * 10 );
printf("Round b: %d", (b + 5) / 10 * 10 );
</pre>
<h3>Multiply by arbitrary numbers using shift</h3>
<p>
Sometimes, for good reasons, you need to hand optimize your code. But most of the time, somebody is just showing off, and you will see code like this:
<pre>
b = ( a << 1 );
</pre>
<p>
This is the same as multiplying by 2. Because of the way binary numbers are represented, you can multiply by any power of two by shifting by that power. You can also multiply by other numbers.
<pre>
// b = 10*a, which is 8*a + 2*a
b = ( a << 3 ) + ( a << 1 );
</pre>
<p>
<a href="http://www.hackersdelight.org/divcMore.pdf">Division</a> isn't as nice looking.
<h3>Getting array sizes</h3>
In C, you can get the total size of an array in bytes using the <tt>sizeof</tt> operator. Dividing it by the size of each element will give you the number of elements in the array.
<pre>
MyHugeStructure array[100];
int i;
for( i = 0; i < sizeof(array)/sizeof(*array); i++ )
{
array[i].id = i;
}
</pre>
The downside is that if you ever change the array to be dynamically allocated using <tt>new</tt> or <tt>malloc()</tt>, then sizeof() will only return the pointer size. Of course, in a large project you should have have a defined constant for the number of entries.
<h3>Else Flattening</h3>
<p>
Deeply nested else clauses are hard to understand. But if they look like this, you are in luck:
</p>
<pre>
if ( A ) {
} else {
if ( B ) {
} else {
if ( C ) {
}
}
}
</pre>
<p>
Save money on wide screen monitors by flattening the elses.
</p>
<pre>
if ( A ) {
} else if ( B ) {
} else if ( C ) {
}
</pre>
<h2>Black is White and Up is Down if you're not using a mainstream compiler</h2>
<p>
If your code has to run on a lot of different platforms, you should know that GCC and Microsoft VC++ are very similar and this similarity will trick you into thinking that all compilers are alike. But the C standard leaves a lot of stuff out that will surprise you, if you do work on embedded systems with crappy, but standards-conforming compilers.
<p>
What is sizeof(char)? In most compilers it is 1 byte. Switch compilers, and it might be 2 or 4 bytes. The C standards only says that it cannot be larger than an int.
<p>
What happens if you add something to the maximum integer?
<p>
<pre>
int a = MAX_INT;
a++;
printf("%dn", a);
</pre>
<p>
In mainstream compilers you get a negative number, since the number space wraps around. But the C standard says the result is undefined. So some compilers will leave <code>a</code> at the maximum value, MAX_INT.
<p>
And don't even think about bit-shifting a negative number:
<pre>
int a = -2;
printf("%dn", a >> 1 );
</pre><ul><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=71'>Using the Acer Aspire One as a web server</a><li><a href='?id=138'>Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US</a><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=37'>Spoke.com scam</a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a><li><a href='?id=10'>Detecting C++ memory leaks</a></ul>
When a reporter mangles your elevator pitch
tag:smhanovtechblog,2007:id59
2009-06-01T18:00:30-05:00
2009-06-01T18:00:30-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
If a reporter asks you about your new startup company, be careful what you say.
<ol>
<li>The statement that sounds best will be quoted.
<li>Some of what you say will be re-ordered or deleted.
<li>Long, rambling descriptions will be paraphrased and condensed.
</ol>
<p>
Here is a pitch from a new startup company, taken from an article in The KW Record on <a href="http://news.therecord.com/Business/article/513072">Wednesday, April 1st, 2009</a>:
<blockquote>
<p>
"We are positioning ourselves to disrupt the entire computing experience," said Ted Livingston, co-founder of Unsynced, an emerging company that won free patent-filing services from law firm Miller Thomson LLP at yesterday's competition for the students from the VeloCity residence...
<p>
The initial software will let people keep all of their music files on their BlackBerry and also be able to manage those tunes from any computer without having to download an application like Apple's iTunes.
<p>
"All of your music can be on your BlackBerry, but if you want to play it on a computer anywhere in the world, you just plug it in," said Livingston, a third year engineering student.
</blockquote>
<p>
This pitch starts with a generic, ho-hum opening, the kind that makes my eyes skip down a couple of paragraphs. The reporter is only going to give you a few sentences, so use them wisely, and don't throw them away with verbal fluff about "disrupting the entire computer experience."
<p>
The pitch fails to differentiate itself from existing solutions. This is no different from a $5 USB key. You can keep your music files on it. When you stick it in your computer a player appears and starts playing your music. In fact, the BlackBerry already has mass storage support, so when you plug it in your files appear without any special software.
<p>
To be fair, perhaps Livingston couldn't give too much away, if he is going to use his new, free patent filing services.
<p>
The pitch below did get me excited:
<blockquote>
<p>
Remember how you struggled to not show your disappointment at Christmas, when your Aunt May gave you a gift card to a book store but you really wanted the cash toward your new cellphone?
<p>
<a href="http://www.giftah.com">Giftah.com</a> will replace that disappointment with smiles by helping people sell those unwanted gift cards, said co-founder Nick Belyaev, a fourth year UW math student.
</blockquote>
<p>
I want to use this web site now. Unfortunately, the ideas that sound best may not always be the most successful in practice. According to Joel Spolksy, the ideas that work <a href="https://stackoverflow.fogbugz.com/default.asp?W25795">sound the dumbest</a>:
<blockquote>
<p>
If you explain it, and everyone says "Oh yeah, that would work, I'm surprised
that's not being done," then it <i>is</i> being done. However, if you explain it and they say, "That wouldn't work, because of <i>blah</i>. It could never possibly work. You could never have auctions on the Internet because people are untrustworthy and they will use it to steal your money by pretending to sell you a laptop and not sending you the laptop, so you can't have auctions on the web." But as it turns out, you <i>can</i> have auctions on the web.
<p>
Whatever the idea is, it has to have a fatal flaw at first glance -- or has to sound like a terrible idea. You have to believe in it for some reason, which you just have trouble explaining to anyone except your brother-in-law who joins you in your startup, or your college roommate who doesn't really get it. Because you do need someone to join you, but the idea has to be <i>not obvious and it has to sound bad</i>. <b>Otherwise it's getting done.</b>
</blockquote>
<p>
If you have a company, and a reporter asked you to explain it, what would you say?
<p>
<b>Update in 2011</b>
Ted recently <a href="http://velocity.uwaterloo.ca/news/23-year-old-donates-us1-million-to-support-university-of-waterloo-student-entrepreneurs">donated $1M to the University of Waterloo</a>. So it seems he has improved his elevator pitch in the past two years!<ul><li><a href='?id=3'>Cell Phone Secrets</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=133'>Give your Commodore 64 new life with an SD card reader</a><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=57'>Coding tips they don't teach you in school</a><li><a href='?id=68'>When programmers design web sites (comic)</a></ul>
Test Driven Development without Tears
tag:smhanovtechblog,2007:id66
2009-05-13T20:46:35-05:00
2009-05-13T20:46:35-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Every company that I worked for has its own method of testing, and I've gained a lot of experience in what works and what doesn't. At last, that stack of conflicting confidentiality agreements that I got as a coop student have now all expired, so I can talk about it. (I never signed them anyway.)
<p>
<em>Warning: </em> My recollection of events may be different from what actually occurred. <i>Do you remember what you were doing 10 years ago?</i>
<h2>Don't make testing a pain in the ass.</h2>
Two places I worked at had test systems that were painful to use. They were Microsoft and Soma.
<p>
Microsoft's is the worse of the two. The system of my particular team in 2000 was so bad that developers couldn't even use it. Only dedicated <i>Software Test Engineers</i> had access to it and could run the full suite. Setting it up on a new PC could not be done in a day. Eventually at the end of the summer I managed to get a copy running, and it found lots of bugs in my module, but by then I ran out of time to fix them. Yes, this was all my fault because I should have been running the tests earlier. I didn't run them because they were so darn hard to get a hold of and set up.
<p>
<i>Testing must be done continuously and during development, by developers.</i>
<p>
Soma had the right idea. They were working on a software phone that ran on Linux. Before checking in a change, we were required to write a test for it and run the full regression suite without any failures.
<p>
The software and tests were all written in Java, so to create a test, you'd create a Java object with a procedure that called the high level APIs to start a phone call, and figure out a way to use the feature you were working on.
<p>
Some of the features needed a complex set of steps to invoke. If you wanted to test the user hitting a #-code to add a participant during a five-way conference call, you had to first set up five fake calls using the Java API. There were functions for dialing a number and such. Whenever an API changed, all the test objects would have to be updated.
<p>
The test system was downright painful to use. Early versions ran in real time, with real timeouts. To test that the dial tone only played for 30 seconds before the off-hook beeps started, you had to sit there and wait while the system did nothing until the timer expired. There was a speeded-up mode where timers would expire immediately. But even in speed mode, the Java system was so bogged down that it would take several hours to run everything. While I was there, they were in the process of building a cluster of 100 PCs just to run all the tests.
<p>
The problem was that all of the tests were system level. Programming in Java eventually leads to a system that I call <i>object-soup</i>: More abstraction can seem to be better, but it leads to more classes, and more classes have more references to each other. The system grew so complex that there was no way to reset the state to the middle of a five-way call, without actually performing all of the call setup. Each one had to be set up for every test. Unit testing of an individual class was meaningless, but it was impossible to separate a feature from the rest of the system.
<h2>You can't test GUIs</h2>
<p>
In 1999, Corel's flagship product was its DRAW! Vector graphics suite. As I remember it, testing was completely manual. Of course, developers were expected to test their changes as much as possible, but CorelDRAW is a huge program with thousands of features and modes of operation. After we'd done some cursory testing, we'd check in a change into Visual Source Safe and wait bugs to come in from beta testers. When a bug was submitted, the test specialist for my team, Mona, would reproduce it and create an issue report.
<p>
The system was clearly broken. We had about 100,000 open bugs and even some of the more serious ones (copy text with certain bullet styles results in crash.) had been unfixed for several versions.
<p>
The problem was the lack of any regression testing. We relied too much on developers manually going through all the code paths, and beta testers to submit reports. The engines were physically separated into libraries, but they were tightly coupled into the UI, so automated testing was impossible. If you broke something in a little-used feature, it might be years until someone noticed.
<p>
Regression testing is only feasible if the process is automated, but to this day, I still can't think of a good way to automatically test GUIs. If your program has a graphical user interface, you are stuck with laborious manual testing. The best thing you can do is to separate your program into two parts: a user interface part, and an engine part that does the real work. The user interface part will have to be tested by moving the mouse and going through all the options. The engine part has to have a well defined API, with inputs and outputs that you can test automatically.
<p><i>Note: Most people stop reading at this point.</i>
<h2>Three rules for Test Driven Development</h2>
<a name="good"></a>
<p>
In a modern software company, developers should be running some kind of regression tests with <i>every change they make</i>. If you make it painful, then your developers will be unproductive. If you make it painful and mandatory, then your developers will be unproductive and unhappy. For effective test driven development, you need three things:
<ul>
<li>The test system should support the developer, not the other way around.
<li>A developer must be able to create a new test in under 10 minutes.
<li>The test suite should be capable of running hundreds of tests in under 5 minutes.
</ul>
<p>
If the test system is painful to use and create tests for, then developers will not create tests. There has to be some payoff for spending the effort of creating a test, in terms of finishing and going home early. Otherwise, the tests will be put off to later, or they won't get done, or you will have to have a separate team whos only job it is to create tests, and then you are no longer doing test driven development.
<p>
The test system must support the easy creation of tests. You should be able to take a bug report or some log from the field, run it through a tool, and out pops your test that is ready to run to reproduce the issue. If the tests are written in Java that is quite hard to do. Ideally instead of writing code, you will have some other kind of input, like a list of events that occurred since system startup. You can run the events through your system to exactly reproduce its state. These types of tests take no development effort to create, and the best part is they don't depend on function names and classes. You could rewrite your system from scratch, and as long as it takes the same input, your tests will still work.
<p>
Finally, the test suite should be fast. You're going to want your automated build system to run the tests after every few changes. If you think about it, a developer might make, on average, one or two changes a day. If you have 100 developers, you will have 100 to 200 changes a day. Developers will need to be able to run the tests at their desk, too. If regression tests passing is mandatory before committing a change, then it should take only a few minutes to run, so you don't have developers leaving for a three hour lunch, checking their stocks, or having swordfights.
<p>
<h2>File importer example</h2>
Lets say we are responsible for writing file importers for a word processor. We are fixing a bug in a file importer for Microsoft DOC format. The input is a .DOC file, and the output is a series of function calls that modify a document. So when we read some text, we call document.addText(), and when we get a new font, we call document.setFont(), etc.
<p>
But instead of maintaining a real document and displaying on the screen, we have a generic test document. When we call document.addText(), we just record this fact to a text file. So after our importer runs, we might have something like this:
<pre>
called addText("This document is copyright")
called setFont("Symbol", 10)
called addText("c")
called setFont("Times New Roman", 10 )
called addText("1995")
</pre>
<p>
Suppose we had hundreds of .doc files, taken from actual bug reports by actual users. We put them into a folder called "input" and check them into our source control system. We run them through the test document. This only takes a few seconds, because all its doing is creating text files. We then take these output .txt files, and check them into our source control system.
<p>
If, one year later, I'm making a change and the output of the test suite changes in any way, then it is either a bug, an improvement, or irrelevant. I'll revise the change to fix the problem, or update the checked-in text files with the new results.
<p>
We now have a regression test system that is easy to use and quick to run. The tests are not Java files. They take absolutely no effort to write -- we just save an attachment. In fact, it's easier to fix an issue after creating a test for it, because you can run it over and over again. Developers will naturally create tests as part of doing their job, without even being asked.
<p>
Now we can re-write stuff without fear. We can re-write the file importers from scratch, and make sure they work on all the same documents in exactly the same way. We can also re-write the rest of the system, as long as the interface to addText() and setFont() still works the same way.
<p>
Sure, there are some bad parts. If you change document.setFont() so that it needs a font encoding parameter, you will need to update all the test scripts. But these changes aren't difficult to manage, and the benefits far outweigh the inconvenience.
<h2>In Conclusion</h2>
<p>
If you are setting up a regression test system, it should be effortless to create a new test, and it should be able to run hundreds of tests in five minutes. Most importantly, the test system should make it easier to fix bugs, so developers will naturally want to create new tests. <ul><li><a href='?id=77'>Pitching to VCs #2 (comic)</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=134'>0, 1, Many, a Zillion</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=132'>20 lines of code that will beat A/B testing every time</a><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a><li><a href='?id=128'>Why you should go to the Business of Software Conference Next Year</a></ul>
Drawing Graphs with Physics
tag:smhanovtechblog,2007:id65
2009-05-08T21:42:40-05:00
2009-05-08T21:42:40-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
I use <a href"http://www.graphviz.org">graphviz</a> whenever I need to draw state machine diagrams. Drawing circles connected with lines is a hard problem for computers, because they have to decide where to place the circles so the diagram makes sense. These types of diagrams are called graphs.
<p>
To my surprise, I found that there is a very simple way to arrange graphs that can be expressed in only a few lines of code, using <a href="http://www.cs.ubc.ca/local/reading/proceedings/spe91-95/spe/vol21/issue11/spe060tf.pdf">force-directed placement</a> [Fruchterman, 1991]. We pretend that the nodes on the graph all strongly repel each other. However, on the other hand, nodes that are connected attract each other with a weaker force. It is as if you had a bunch of statically charged styrofoam balls connected with springs, and its very fun to watch.
<p>
Click the image to reset it.<br>
<canvas id="graph" width=800 height=600 onclick="a.reset()"></canvas><br>
<p>
Here's the code that moves the nodes around.
<pre>
Springs.prototype.recalc = function()
{
var width = this.ctx.canvas.width;
var height = this.ctx.canvas.height;
// K is related to how long the edges should be.
var k = 200.0;
// C limits the speed of the movement. Things become slower over time.
var C = Math.log( this.frame + 1 ) * 100;
this.frame++;
// calculate repulsive forces
for ( var vindex = 0; vindex < this.graph.nodes.length; vindex++ )
{
var v = this.graph.nodes[vindex];
// Initialize velocity to none.
v.vx = 0.0;
v.vy = 0.0;
// for each other node, calculate the repulsive force and adjust the velocity
// in the direction of repulsion.
for ( var uindex = 0; uindex < this.graph.nodes.length; uindex++ )
{
if ( vindex == uindex ) {
continue;
}
var u = this.graph.nodes[uindex];
// D is short hand for the difference vector between the positions
// of the two vertices
var Dx = v.x - u.x;
var Dy = v.y - u.y;
var len = Math.pow( Dx*Dx+Dy*Dy, 0.5 ); // distance
if ( len == 0 ) continue;
var mul = k * k / (len*len*C);
v.vx += Dx * mul;
v.vy += Dy * mul;
}
}
// calculate attractive forces
for ( var eindex = 0; eindex < this.graph.edges.length; eindex++ )
{
var e = this.graph.edges[eindex];
// each edge is an ordered pair of vertices .v and .u
var Dx = e[0].x - e[1].x;
var Dy = e[0].y - e[1].y;
var len = Math.pow( Dx * Dx + Dy * Dy, 0.5 ); // distance.
if ( len == 0 ) continue;
var mul = len * len / k / C;
var Dxmul = Dx * mul;
var Dymul = Dy * mul;
// attract both nodes towards eachother.
e[0].vx -= Dxmul;
e[0].vy -= Dymul;
e[1].vx += Dxmul;
e[1].vy += Dymul;
}
// Here we go through each node and actually move it in the given direction.
for ( var vindex = 1; vindex < this.graph.nodes.length; vindex++ )
{
var v = this.graph.nodes[vindex];
var len = v.vx * v.vx + v.vy * v.vy;
var max = 10;
if (this.frame > 20) max = 2.0;
if ( len > max*max )
{
len = Math.pow( len, 0.5 );
v.vx *= max / len;
v.vy *= max / len;
}
v.x += v.vx;
v.y += v.vy;
}
};
</pre>
<p>
Its really fun to play with this method. Here's what happens if we add a bit of gravity to the system. (Click to reset)<br>
<canvas id="graph1" width=800 height=600 onclick="b.reset()"></canvas><br>
<script src="springs1.js" type="text/javascript"></script>
<p>
Coding that made me want to play <i><a href="http://en.wikipedia.org/wiki/The_Incredible_Machine">The Incredible Machine</a></i>.
<ul><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=26'>A simple command line calculator</a><li><a href='?id=131'>[comic] Appreciation of xkcd comics vs. technical ability</a><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=115'>Compressing dictionaries with a DAWG</a><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a></ul>
Keeping Abreast of Pornographic Research in Computer Science
tag:smhanovtechblog,2007:id63
2009-04-25T08:00:09-05:00
2009-04-25T08:00:09-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<img src="lenac.jpg" width=200 align=right>
Burgeoning numbers of Ph.D's and grad students are choosing to study pornography. Techniques for the analysis of "objectionable images" are gaining increased attention (and grant money) from governments and research institutions around the world, as well as Google. But what, exactly, does computer science have to do with porn? In the name of academic persuit, let's roll up our sleeves and plunge deeply into this often hidden area that lies between the covers of top-shelf research journals.
<h2>Lena</h2>
<p>
One cannot do research in image processing without an encounter with Lena (pronounced Lenna). The image of the woman with a feathered hat has become the de-facto test image for many algorithms, and appears in thousands of articles and conference papers. <a href="http://www.cs.cmu.edu/~chuck/lennapg/pcs_mirror/may_june01.pdf">And it is of pornographic pedigree:</a>
<blockquote>
<p>
Alexander Sawchuk estimates that it was in June or July of 1973 when he, then an assistant professor of electrical engineering at the USC Signal and Image Processing Institute (SIPI), along with a graduate student and the SIPI lab manager, was hurriedly searching the lab for a good image to scan for a colleague's conference paper. They had tired of their stock of usual test images, dull stuff dating back to television standards work in the early 1960s. They wanted something glossy to ensure good output dynamic range, and they wanted a human face. Just then, somebody happened to walk in with a recent issue of Playboy.
<p>
The engineers tore away the top third of the centerfold so they could wrap it around the drum of their Muirhead wirephoto scanner, which they had outfitted with analog-to-digital converters (one each for the red, green, and blue channels) and a Hewlett Packard 2100 minicomputer. The Muirhead had a fixed resolution of 100 lines per inch and the engineers wanted a 512 x 512 image, so they limited the scan to the top 5.12 inches of the picture, effectively cropping it at the subject's shoulders.
</blockquote>
<p>
The rest of the story (and the rest of Lena) can be found <a href="http://www.cs.cmu.edu/~chuck/lennapg/">here</a>. Indeed, the 70s marked the beginning of a long relationship between computer science and pornography. However, after the birth of the world wide web, things really got hot and heavy.
<h2>Finding Naked People</h2>
<p>
In the 1990s the world wide web began to explode, pumping information of all kinds into the homes of the technologically savvy at rates as high as 9600 bits per second. It was the time when search engines such as Webcrawler, Altavista, and Yahoo began the arduous task of spidering the scattered bits of information in Internet servers everywhere. The problem was that someone might search for a completely innocuous query such as the <a href="http://www.cl.cam.ac.uk/coffee/qsf/coffee.html">Trojan Room Coffee Pot</a>, and come up with images that were unexpected and inappropriate, and depending on one's tastes, objectionable.
<p>
It's not likely to be on his business card, but David A. Forsyth is an expert in web pornography, having served on the NRC committee for this topic. It is evident from <a href="http://luthuli.cs.uiuc.edu/~daf/">his web page</a> that he has a sense of humour, which explains the superbly descriptive title for his 1996 paper, <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.52.1561&rep=rep1&type=pdf">Finding Naked People</a>. Forsyth was one of the first researchers to study the problem of identifying objectionable content.
<p>
One of Forsyth's research areas is tracking people in images and videos and figuring our their pose. In the general case, the system has to cope with the fact that people can wear clothes. It would be easier if the subjects all wore the same colour, or didn't wear anything at all. <i>Finding Naked People</i> describes a way of first masking out areas of skin. The areas are then grouped together into human figures (visualized by drawing a stick figure on the image). The crux of the paper is the grouping algorithm. The grouper knows rules such as how limbs fit together into a body, and the fact that a person cannot have more than two arms. Using the rules, it figures out how to superimpose a body onto the skin patches. If it can successfully do this, the image is probably a naked person. If it cannot, then it is probably something else, like a lamp.
<p>
Here is a visualization of the skin probability field from the paper, with the grouper output segments superimposed on top:
<p align=center><img src="images/stick-figure.jpg"></p>
More probability masks can be found in <i>Proceedings of the 4th European Conference on Computer Vision</i>, volume II on page 598. Be careful -- the pages tend to stick.
<h2>It's better with more than one</h2>
<i>Finding Naked People</i> piqued a lot of interest in the field of objectionable images, and the skin matching idea is now the first step in many algorithms. However, as James Ze Wang of Stanford notes, "it takes about 6 minutes on a workstation for the figure grouper in their algorithm to process a suspect image passed by the skin filter."
<p>
In their <a href="http://infolab.stanford.edu/~wangz/project/imscreen/JCC98/wang.pdf">System for Screening Objectionable Images</a>, Wang and his colleagues describe the WIPE<sup>TM </sup> method for screening content. They use a wavelet edge detection algorithm to obtain the shape of the image. Edge detection transforms an image into the outlines of the object. Wavelet edge detection allows them to tune it to detect sharp or increasingly blurry edges until well-defined shapes appear.
<p>
<a href="http://en.wikipedia.org/wiki/Image_moments">Image moments</a> allow one to treat any shape as a <a href="http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT2/node3.html">flat, physical object</a> (like a plate). You can figure out the centre of gravity, axis of symmetry, and other properties that don't change when you move, rotate, or change the size of the object. This typically results in a set of 3 to 7 numbers that you can use to compare how similar shapes are. They were used in early OCR (optical character recognition) algorithms circa 1962.
<p>
Wang uses both edge detection and image moments in the analysis. His algorithm is different from modern ones, because an image must pass a series of five YES/NO tests. Future algorithms would combine the detectors using statistical methods and give a probability estimate.
<ol>
<li>If the image is small, it is assumed to be an icon, and allowed. Icons (such as a mail envelope) were frequently used on the world wide web in the 1990s.
<li>If the image contains few continuous tones, it is considered to be a drawing and is allowed to pass.
<li>If a great portion of the colours of the image are human body colors, then the image is rejected as porn. The algorithm is pretty smart -- if a patch identified as skin has lots of edges in it, it is probably not really skin and is removed from the analysis. (This also counts as the texture matching step)
<li>Finally, the edge (outline) image is converted into 21 numbers representing the translation, scale, and rotation invariant moments. If the 21 numbers are to close to anything already in the database, the image is rejected.
</ol>
Here are some examples where the algorithm fails. We have blurred them to protect the eyes of the gentle reader. For high resolution versions, you'll have to refer to <i>
Proceedings of the 4th International Workshop on Interactive Distributed Multimedia Systems and Telecommunication Services</i> on page 20 (the dog-eared one).
<p align=center><img src="images/wang1998.jpg"></p>
<h2>Getting a leg up on skin models</h2>
Skin detection is an important step in porn detection, but figuring out which colours represent skin is a hard problem. Colour depends on the lighting used in the photo, the ethnicity of the participants, and the quality and noise level. Michael J. Jones and James M. Rehg at Compaq <a href="https://eprints.kfupm.edu.sa/66778/1/66778.pdf">studied the problem in detail</a>. They first manually labeled hundreds of images, highlighting all the areas that were skin using a custom drawing application. Once you have billions of pixels that you know are skin, and billions that you know are not, you can easily classify them using introductory math:
<p align=center><img src="images/jones2002-2.png"></p>
The paper describes how to find the probability function, <i>P</i>, using a database of images painstakingly highlighted by an army of enthusiastic research interns. However, as a porn detector, the method needs work.
<p align=center><img src="images/jones2002.png"></p>
It will be obvious to anyone who has bought a digital camera recently how to improve this system. <b>Before reading on, can you spot the solution?</b>
<h2>Taking the ogle out of Google</h2>
In recent years, Google has had its hands full with the problem of pornographic imagery. Henry A. Rowley, Yushi Jing, and Shumeet Baluja at the Mountain View campus, have <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.7839&rep=rep1&type=pdf">developed a system</a> that combines skin detection with a number of different features. After applying face detection, they can deduce that the pixels around the face represent skin colour, and therefore find other skin pixels in the image. If the face is the majority of the image, as in a portrait, the image is safe. They use a colour histogram to detect artificial images such as screen shots. (so dirty cartoons are safe?).
<p>
Doing what only Google could, they must have set a record for the rate of pornographic analysis. They evaluated the speed of the algorithm on a corpus of around 1.5 billion thumbnail images of less than 150x150 pixels. "Processing the entire corpus took less than 8 hours," the team bragged, "using 2,500 computers."
<h2>Bags of visual words (Arm, leg, or <span style="background:black">. . .</span>?)</h2>
In 2008, Thomas Deselaers <i>et al.</i> came up with <a href="http://www-i6.informatik.rwth-aachen.de/publications/download/571/Deselaers-ICPR-2008.pdf">a unique way of finding porn</a>, from the world of artificial intelligence. Large news databases can automatically classify news articles based on the words in them. Articles containing the names of political figures or sports jargon can be easily categorized by machines, that don't need to really understand what the article is about. Techniques exist so that the machines can learn on their own which words or names are important. The same methods can be applied to images, using <i>visual words</i>.
<p>
To create the visual vocabulary, they extract image patches around "points of interest", parts of the image that are likely to contain features. They are then scaled to a common size, and analyzed using <a href="http://en.wikipedia.org/wiki/Principal_components_analysis">PCA</a> to find commonalities. It is similar to <a href="http://en.wikipedia.org/wiki/Eigenface">face detection</a>, but for things that aren't faces. It also takes colour into account in the analysis. Because colour is a part of the "vocabulary" already, skin detection is unnecessary.
<p>
Using this technique, Deselaers is even able to go beyond simple YES/NO classification and reach a new level of precision. The algorithm can rate images into one of five categories of increasing levels of offensiveness, from benign, to lightly dressed, to partly nude, fully nude, and porn. The paper contains examples from each category, and is guaranteed to offend somebody.
<h2>Corpus non indutus</h2>
<p>
At the end of the Google paper, the authors speculate on how to spur further advances:
<blockquote>
...because of the ubiquity of the Internet, search engines, and the widespread proliferation of electronic images, adult-content detection is an important problem to address. To improve the rate of progress in this field it would be useful to establish a large fixed test set which can be used by both researchers and commercial ventures.
</blockquote>
<p>
Yes, bring on the grant-sponsored porn, so that researchers can make the world a better place. But despite the years of study, one question remains unanswered: if such a corpus existed, how would we find it?
<h3>For a good time, read this</h3>
<ul>
<li><a href="http://gandolf.homelinux.org/~smhanov/blog/index.php?id=62">Exploiting perceptual colour difference for Edge Detection</a>
<li><a href="http://gandolf.homelinux.org/~smhanov/blog/index.php?id=52">Automatically remove wordiness from your writing</a>
<li><a href="http://gandolf.homelinux.org/~smhanov/blog/index.php?id=8">A Rhyming Engine</a>
<li><a href="http://gandolf.homelinux.org/~smhanov/cs886_hanov_2007.pdf">Automatically figure out how to pronounce words</a>
<li><a href="http://gandolf.homelinux.org/~smhanov/cs698_wavelet_project.pdf">Wavelets and Edge Detection</a>
</ul>
<ul><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=135'>How I run my business selling software to Americans</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=57'>Coding tips they don't teach you in school</a><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a></ul>
Exploiting perceptual colour difference for edge detection
tag:smhanovtechblog,2007:id62
2009-04-16T19:42:41-05:00
2009-04-16T19:42:41-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p style="font-size:small"><a href="https://www.expertoautorecambios.es/science/?p=1556">Estonian Translation</a></p>
<p>
I once took computer vision class. Every algorithm we learned was done on grayscale images, as if it were 1950 and we couldn't afford those new fangled colour VDTs. I put up my hand and asked why we discard all of this lovely colour information. The prof answered that some people have tried it, but it doesn't give much benefit.
<p>
But in some situations, colour can be important. Here's an image in GIMP:
<p align=center><img alt="Green circle on red square" src="gimp-colour.png"></p>
<p>
First, we convert to grayscale:
<p align=center><img src="gimp-grayscale.png" alt="completely gray square"></p>
<p>
Umm......... I don't think we need to go any further here. 'Nuf said.
<h2>Traditional approaches for colour</h2>
<p>
Computer vision researchers use two approaches for dealing with colour. The first is to break the image into red, green, and blue images, do the algorithm three times, and combine the results. The second is to treat the RGB values as a vector and subtract the vectors to determine colour difference. These approaches can work, but they don't take into account human perception of colour.
<p>
The RGB values in an image have very little to do with how the human eye works. They have more to do with the voltage fired at phosphors in an 80's era CRT display than human vision. (Let's conveniently ignore the fact that LCDs have completely different properties. <a href="http://en.wikipedia.org/wiki/SRGB">Everybody else does.</a>)
<p>
To compare two colours, you have to first convert them to a different colour space. If you find yourself subtracting RGB values, you're doing it wrong.
<p>
<h2>Colour Difference</h2>
<p>
<img align=left src="http://upload.wikimedia.org/wikipedia/commons/thumb/8/83/CIE_1976_UCS.png/300px-CIE_1976_UCS.png" alt="CIE Luv Colour Space">People can distinguish skin colours very well, so we can tell if a potential mate is healthy or not. This is more important than distinguishing different shades of green leaves, which would be <a href="http://vision.psychol.cam.ac.uk/jdmollon/papers/Regan(2001)PTRSB.pdf">mere distractions</a> from the delicious bouquet of fruits hanging in trees. Describing the exact shade of an azure sky is simply not required for survival. We are hypersensitive to differences in yellows, reds, and oranges.
<p>
In 1915, Albert Munsell created a <i>Colour Atlas</i>, a book of colour paint samples. He came up with a way of logically arranging them that made sense to his eye. Each colour was indexed by hue, saturation, and value (also known as lightness). We still use his system today, and most of modern colour theory is based on Munsell's ideas.
<p>
The CIE is an international authority in charge of standardizing things to do with light and colour. In 1960 they created a <a href="http://dba.med.sc.edu/price/irf/Adobe_tg/models/cieluv.html">mathematical arrangement of colour</a>, and improved it in 1975. In this space, the more different two colours are, the farther apart they are in the drawing. The arrangement is not perfect, but the <a href="http://www.brucelindbloom.com/index.html?ColorCalculator.html">math is simple enough to work with</a>. It's no good to create an international standard that nobody understands.
<p>
<h2>Finding outlines in images</h2>
Edge detection is usually the first step in any computer vision algorithm, because it converts an image into a black and white outline that is easier to work with.
<p align=center><img hspace=5 width="200px" src="http://upload.wikimedia.org/wikipedia/en/thumb/f/f0/Valve_original_%281%29.PNG/300px-Valve_original_%281%29.PNG"><img hspace=5 width="200px" src="http://upload.wikimedia.org/wikipedia/en/thumb/d/d4/Valve_sobel_%283%29.PNG/300px-Valve_sobel_%283%29.PNG"></p>
<ul>
<li>
You can count the vertical and horizontal edges and figure out if a photo is taken in a <a href="http://eprints.kfupm.edu.sa/54311/1/54311.pdf">city or a forest.</a> City buildings are often square.
<li>
You can treat it as a flat, physical object and figure out the <a href="http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT2/node3.html">center of gravity and angle of symmetry,</a> and get some numbers that you can compare two images with, even if one is rotated or scaled.
<li>
You can <a href="http://www.cim.mcgill.ca/~dparks/CornerDetector/algorithms.htm">mark strong intersecting edges as corners</a>, and compare the locations of corners in two images to figure out how to line them up, or if they are the same object.
</ul>
<p>
A nice, easy way to perform edge detection is to use the sobel operator:
<p align=center>
<img src="http://upload.wikimedia.org/math/4/3/2/4324cd5daf7569063b9d18beb05d8e69.png">
</p>
<p>
Translation: We create two new images, Gx, and Gy. The pixels in the new images get their values from the difference in the values of the pixels around their corresponding position in the old image. In the Gx image, the differences are in the right to left direction. In the Gy image, the differences are in the up and down direction. After the two new images, are created, we combine them into one by adding the pixels together in a special way. We square the values, add them together, and take the square root. The result is the edge-detected image.
<p>
I like the sobel algorithm because it is easy to adapt to color difference. Instead of subtracting the pixel gray values, we subtract the two colours using the distance in <a href="http://dba.med.sc.edu/price/irf/Adobe_tg/models/cieluv.html">L*u*v</a> colour space, which is almost perceptually uniform.
<h2>Results</h2>
<p>
I implemented the two algorithms in <a href="sobel.c">400 lines of C code</a>. When the program is run with a given PNG image, it reads it and performs edge detection first on the grayscale version, and then on the colour version using colour differences. In both cases, the images are normalized so that the highest difference is white, and the lowest is black. Here are the results.<br>
<p>
<i>Note: The grayscale image appears blurred, and the colour one does not. This is an error in the blog entry and should be corrected later. The implementation blurs </i>both versions<i> before performing edge detection, to reduce noise.</i>
<p>
<table>
<tr><td>Grayscale original<br><img src="house-gray.png"></td><td>Traditional Edge Detection<br><img src="house-sgray.png"></td></tr>
<tr><td>Colour Original<br><img src="house-colour.png"></td><td>Colour Edge Detection<br><img src="house-colourdiff.png"></td></tr>
</table>
<table>
<tr><td>Grayscale original<br><img src="lena-gray.png"></td><td>Traditional Edge Detection<br><img src="lena-graydiff.png"></td></tr>
<tr><td>Colour Original<br><img src="lena-colour.png"></td><td>Colour Edge Detection<br><img src="lena-colourdiff.png"></td></tr>
</table>
<h1>Further Reading</h1>
<ul>
<li><a href="http://imaging.utk.edu/~koschan/paper/ACCV95Proc.pdf">A comparative study on colour edge detection.</a> The authors <i>did not</i> use a perceptually uniform colour space. They subtracted RGB values directly, which may explain their disappointing results.
<li><a href="http://gandolf.homelinux.org/~smhanov/cs698_wavelet_project.pdf">Wavelets and Edge Detection</a> - A project I did a few years ago on edge detection using another method.
</ul><ul><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=60'>Is 2009 the year of Linux malware?</a><li><a href='?id=54'>Usability Nightmare: Xfce Settings Manager</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=26'>A simple command line calculator</a></ul>
Experiment: Deleting a post from the Internet
tag:smhanovtechblog,2007:id61
2009-04-12T13:40:20-05:00
2009-04-12T13:40:20-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Once you post something on the Internet, it is hard to get rid of it. As an experiment, I deleted one of my past posts, and I tried to remove all traces of it.
<p>
I selected my post about <i>Technical Interview</i> tips, because it is mildly popular, but never did very well. It was on reddit for only a couple of hours. Yet it regularly received a lot of hits from Google looking for interview tips for RIM. In my opinion the writing needed work, so I deleted it. Forever.
<p>
First, I removed it from my blog. I have a checkbox that says whether a post is shown or not. Unchecking it removes it from the main page, and whenever people try to see it, they get the main article listing instead.
<p>
<h2>RSS Reader caches</h2>
That wasn't good enough, because the article was still available in RSS readers. When Google reader retrieves my blog entries, it simply merges the updated ones with its own database. The <a href="http://www.ietf.org/rfc/rfc4287.txt">atom specification</a> does not define any way to delete posts, but it does allow updates. I had to put the post back, but remove its contents. Then, when the RSS reader did the merge, it would update its database to contain the empty post.
<p>
<h2>Google Cache</h2>
<p>
My post still appeared in Google, and you could read it by clicking on the cached link. To remove it from the Google cache, I had to make the page return a HTTP 404 error. I tried using the .htaccess file:
<pre>
redirect gone /~smhanov/blog/?id=43
</pre>
<p>
Unfortunately this had no effect on my web server. Apparently .htaccess doesn't apply to php scripts. I had to physically change <a href="index.txt">my blog software</a> to return a 404 HTTP status if that entry is retrieved:
<pre>
if ( $_GET['id'] == '43' ) {
header("HTTP/1.0 404 Not Found");
exit;
}
</pre>
<h2>Reddit</h2>
Comments about the post appeared on <a href="http://www.reddit.com/r/programming/comments/7t6cz/technical_interviews_what_we_really_care_about/">reddit</a>. Since I was the original submitter to reddit, I have the option to delete it:
<p align=center><img src="delreddit.png"></p>
<p>
Clicking delete didn't work as advertised. You can still get to the post, but it is marked as [deleted]. This is a real problem on Reddit's part, because people might post something under the mistaken belief that they can remove it later. The button should be descriptive of what it will actually do. Software shouldn't lie.
<h3>Conclusion</h3>
The main text of the article is nowhere to be found. The problem is any comments or blog reactions will still be there, although they will have broken links. The experiment is a partial success.
<p>
The best way to hide your embarrassing past is to bury them with new things. For example, if you <a href="http://www.google.com/search?hl=en&q=steve+hanov&btnG=Google+Search">search for my name</a> you won't find my <a href="http://www.lionking.org/~fanfic/Stories/hanov1.html">Lion King fan-fiction</a> anywhere in the first few pages of results.
<ul><li><a href='?id=60'>Is 2009 the year of Linux malware?</a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=38'>UMA Questions Answered</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=70'>Finding great ideas for your startup</a></ul>
Is 2009 the year of Linux malware?
tag:smhanovtechblog,2007:id60
2009-04-03T23:24:27-05:00
2009-04-03T23:24:27-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
It is <a href="http://www.internetling.com/2008/03/09/anti-virus-software-is-dead-use-linux/#comment-274">common knowledge</a> that Linux users needn't worry about viruses because users don't run as root. I've never understood the reasoning behind this. Here are a few of the malicious things that a program can do without being root on Ubuntu 8.10:
<ul>
<li>Start a program every time you login<br><code>Add an entry to .config/autostart</code>
<li>
Configure firefox to route all traffic through a remote proxy<br><code>Change a line in .mozilla/firefox/*/prefs.js</code>
<li>
Replace everything in your "System Settings" menu with a command that asks you for your password, then does <i>something else</i> before invoking the real program. <br><code>Add a file to .local/share/applications</code>
<li>
Download and install other programs in the background <br><code>Putting them in .gnome2/system32 seems somehow appropriate</code>
<li>
Run a server of any kind (web/ftp/irc/etc)<br><code>Just pick a port above 1024, and <a href="http://en.wikipedia.org/wiki/Universal_Plug_and_Play#Lack_of_Default_Authentication">update the firewall with uPnp</a></code>
<li>
Install a new firefox plugin <br><code>put it in .mozilla/firefox/*/extensions/ <br>call it "Ubuntu System Integration Plugin Helper"</code>
</ul>
<p>
Once malware has its grubby code all over your home folder, you are one fake dialog box away from giving it complete control over your system:
<p align=center><img src="firefox-malware.png" border=1 alt="Firefox with suspicious dialog box asking for root password"></p>
<p>
<b>If you have ever run a program or script that wasn't included in your distribution, then you could have been infected with malware.</b> (You weren't.)
<ul>
<li>Have you installed a zipfile full of video codecs from <a href="http://www.mplayerhq.hu/MPlayer/releases/codecs/">somewhere?</a>
<li>Have you ever run a script from ubuntuforums that <a href="http://ubuntuforums.org/showthread.php?t=476924">promised to make it easy to get something working?</a>
<li>Have you ever been <a href="http://www.howtoforge.com/ubuntu_internet_explorer">told by a web site</a> to add a line to your sources.list file? Instant rootkit!
</ul>
If you are interested in more examples, <a href="TheMalwareProject.pdf">The Malware Project (PDF)</a> is a great read that takes you <i>step by step through an actual social engineering experiment with users</i>. The results will surprise you.
<p>
Ubuntu in particular must be very enticing for malware writers, because:
<ul>
<li><b>It is easy to get new users to run things.</b> There are thousands of annoyances with desktop linux that can only be fixed by <i>dropping to the command line</i>, or downloading something to do it for you.
<li><b>It has a rich, portable API.</b> Malware writers have access to all unix commands and a rich programming environment that is guaranteed to be available on every desktop, allowing them to search and change any file in your home folder, or even implement complex network protocols.
<li><b>Open source makes it easy to copy other programs</b>. If you can change sources.list, you can then replace <code>top</code>, <code>ps</code>, and <code>System Monitor</code> with exact clones that neglect to display your processes. This is much easier than hacking up the Windows Task Manager internal memory. Or just do everything in kernel mode for ultimate captcha cracking, DDOS power.
<li><b>People are unprepared.</b> The "fact" that linux can't get viruses is <a href="http://www.internetling.com/2008/03/09/anti-virus-software-is-dead-use-linux/">constantly repeated</a> all <a href="http://linuxmafia.com/~rick/faq/index.php?page=virus">over</a> the <a href="https://help.ubuntu.com/community/Linuxvirus">web</a>.
</ul>
<p>
Is 2009 the year of the linux desktop malware? How long until we see headlines like, "Researchers find massive botnet based on linux 2.30"?
<h2>Further Reading</h2>
<ul>
<li><a href="http://www.geekzone.co.nz/foobar/6229">How to write a Linux virus in 5 Easy Steps</a> - Desktop launchers will be the first wave of attack. You can send a launcher by email. Gnome will execute it regardless of permissions, and the icon and extension can be <a href="TheMalwareProject.pdf">anything you want!</a>
</ul><ul><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=14'>Experiments in making money online</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=63'>Keeping Abreast of Pornographic Research in Computer Science </a><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a></ul>
Email Etiquette
tag:smhanovtechblog,2007:id58
2009-04-01T20:20:03-05:00
2009-04-01T20:20:03-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
If you begin your emails with "Hi, <name>!" then they will seem less rude.
</p>
<pre>
> steve, we were testing your changes and they errored. can you pls
> look at these tracefiles?
> cam
You dolt, you set HPLX_PARAMETER_7=A0, but it should be set to AO.
Please re-run the test.
</pre>
<p>
Compare to:
</p>
<pre>
> steve, we were testing your changes and they errored. can you pls
> look at these tracefiles?
> cam
Hi, Cam!
You dolt, you set HPLX_PARAMETER_7=A0, but it should be set to AO.
Please re-run the test.
</pre>
<p>
See what I mean?
</p>
<ul><li><a href='?id=109'>Cross-domain communication the HTML5 way</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=3'>Cell Phone Secrets</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a><li><a href='?id=140'>A little VIM hacking</a><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a></ul>
How a programmer reads your resume (comic)
tag:smhanovtechblog,2007:id56
2009-03-26T21:11:26-05:00
2009-03-26T21:11:26-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p align="center">
<a href="http://stevehanov.ca/comics/index.php?whichdate=20050722">Previous Comic</a> | <a href="?id=70">Next Comic</a> </p>
<p align=center>Click to enlarge<br><a href="resume_comic.png"><img src="resume_comic.png" width="700"></a></p>
<p>
<a href="http://www.aoky.net/articles/steve_hanov/how_a_programmer_reads_your_resume.htm">Japanese Translation</a> by <a href="http://www.aoky.net/">Yasushi Aoki</a>
</p>
<p>
Unauthorized <a href="http://www.aqee.net/2010/07/14/how-a-programmer-reads-your-resume/">Chinese translation</a>, I think.
</p>
<h2>Here are the real tips</h2>
<ul>
<li><a href="http://inter-sections.net/2007/11/13/how-to-recognise-a-good-programmer/">How to recognize a good programmer</a>
<li><a href="http://www.joelonsoftware.com/items/2009/01/02b.html">Another Resume Tip</a> - From Joel on Software
<li><a href="http://steve-yegge.blogspot.com/2007/09/ten-tips-for-slightly-less-awful-resume.html">Ten Tips for a Slightly Less Awful Resume</a> - Advice from Steve Yegge. An entertaining read.
<li><a href="http://www.joelonsoftware.com/articles/ResumeRead.html">Getting your resume read</a> - From Joel on Software
</ul><ul><li><a href='?id=83'>Sign here (comic)</a><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=148'>How to detect if an object has been garbage collected in Javascript</a><li><a href='?id=65'>Drawing Graphs with Physics</a><li><a href='?id=146'>O(n) Delta Compression With a Suffix Array</a><li><a href='?id=70'>Finding great ideas for your startup</a></ul>
How wide should you make your web page?
tag:smhanovtechblog,2007:id55
2009-03-22T15:58:15-05:00
2009-03-22T15:58:15-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Based on 22500 unique IP addresses over the past week, reddit users have these browser widths:
<p align=center><img src="screenres.png"></p>
The numbers on the bottom are browser widths (minus 40), and the numbers on the side are the counts of unique visitors with that width.
<h3>Data collection</h3>
<p>
At the time this blog posting was made, my blog had <a href="?id=33">hand-drawn borders</a> in the page title. They were generated by a CGI program on the fly. For each visitor, a number about 40 pixels below your browser width is recorded in my server logs. I wrote a perl script to create a histogram for unique visitors, and pasted the result into gnumeric to create the chart.
<h3>Results</h3>
<p>
The peaks seem to correspond to 1024, 1280, 1400, 1600, 1920 screen widths, with 1280 being the greatest peak. This indicates that most people have their browser window maximized.
<p>
There is also a section of uniformity between 1000 and 1280. This seems to be a sweet spot among those who do not maximize their browsers.
<p>
Surprisingly, there are still a few people running at 800x600 resolutions. And one at over 2500 pixels across.
<ul><li><a href='?id=119'>Throw away the keys: Easy, Minimal Perfect Hashing</a><li><a href='?id=146'>O(n) Delta Compression With a Suffix Array</a><li><a href='?id=101'>"Your program is stupid. It doesn't work," my wife told me</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=90'>Regular Expression Matching can be Ugly and Slow</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=2'>UMA's dirty secrets</a></ul>
Usability Nightmare: Xfce Settings Manager
tag:smhanovtechblog,2007:id54
2009-03-21T22:22:04-05:00
2009-03-21T22:22:04-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Quick! Where do you go to increase the text size in all your applications? Can you pick the right button on the first try? <i>Do you feel lucky, <b>punk?</b></i>
</p>
<p align=center><img src="xfce-settings.png"></p>
I might try:
<ol>
<li><b>Desktop</b>, because it's at the top, I see it first, and text is a part of my desktop experience.
<li><b>Display</b>, because that's where the setting is in Windows XP.
<li><b>User Interface</b>, because <i>everything I have ever wanted to change</i> is a user interface setting.
<li><b>Panel</b>, because I also want to change the text size on the panel.
<li><b>Window Manager</b>, because windows can display text on them.
<li><b>Window Manager Tweaks</b>, because I want to <i>tweak</i> to how text is shown.
</ol>
<p>
<i>Or maybe you're in the wrong settings manager.</i> The first hurdle was in the Settings menu. Text settings aren't mentioned at all, although some things are in <i>two or three places</i>. I'm glad I'm not setting up a printer!
</p>
<p align=center><img src="xfce-menu.png"></p>
<p>
It turns out that we chose wisely. Unfortunately, in the Xfce Settings Manager, text sizes are distributed between two categories. Choose <i>Window Manager</i> for the window title bars, and <i>User Interface</i> for everything else.
</p>
<p>
Can't you come up with a better way of organizing settings than <b>Windows 3.1?</b> Has the state of the art in Settings dialogs not advanced since 1992? </p>
<p align=center><img src="win31.png"></p>
<p>
At least Microsoft's control panel was unambiguous. It even had a description in the status bar of what you were going to click on.
<h2>Today's Usability Tips</h2>
<ul>
<li>Did you notice that I never mentioned the word <i>font</i>? New users might call things by different names. More people in the world know what "text size" means than "font size".
<li>Manuals have gone the way of the floppy disk. Instead of writing a manual, spend time just watching someone use your software, so you won't need one in the first place.
<li>People read the screen from the top down. Take advantage of this, and put the most common settings first.
<li>If your software has many settings, thoughtlessly dividing them into categories is a sure way of making them more confusing.
<li>If you are copying user interface layout from Windows 3.1, at least do it right.
</ul>
<ul><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=85'>barcamp (comic)</a><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=98'>Asking users for steps to reproduce bugs, and other dumb ideas</a></ul>
cairo blur image surface
tag:smhanovtechblog,2007:id53
2009-03-14T13:34:45-05:00
2009-03-14T13:34:45-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<center><img src="blur.png"></center>
This really should have been included in <a href="http://www.cairographics.org">cairo</a>. Instead, everyone that wants to have shadows has to <a href="http://taschenorakel.de/mathias/2008/11/24/blur-effect-cairo/">roll their own</a> blur function. Here's my take on it. I'll even release this into the public domain.
<p>
This is used in the back-end for <a href="http://www.websequencediagrams.com/?lz=QWxpY2UtPkJvYjogQXV0aGVudGljYXRpb24gUmVxdWVzdApub3RlIHJpZ2h0IG9mIAAlBUJvYiB0aGlua3MgYWJvdXQgaXQuCkJvYi0tPgBMBQA5E3Nwb25zZQ&s=modern-blue">www.websequencediagrams.com</a>.
<pre>
// to build:
// gcc -I/usr/include/cairo -lcairo -o blur blur.c
#include <stdlib.h>
#include <memory.h>
#include <stdio.h>
#include "cairo.h"
void cairo_image_surface_blur( cairo_surface_t* surface, double radius )
{
// Steve Hanov, 2009
// Released into the public domain.
// get width, height
int width = cairo_image_surface_get_width( surface );
int height = cairo_image_surface_get_height( surface );
unsigned char* dst = (unsigned char*)malloc(width*height*4);
unsigned* precalc =
(unsigned*)malloc(width*height*sizeof(unsigned));
unsigned char* src = cairo_image_surface_get_data( surface );
double mul=1.f/((radius*2)*(radius*2));
int channel;
// The number of times to perform the averaging. According to wikipedia,
// three iterations is good enough to pass for a gaussian.
const MAX_ITERATIONS = 3;
int iteration;
memcpy( dst, src, width*height*4 );
for ( iteration = 0; iteration < MAX_ITERATIONS; iteration++ ) {
for( channel = 0; channel < 4; channel++ ) {
int x,y;
// precomputation step.
unsigned char* pix = src;
unsigned* pre = precalc;
pix += channel;
for (y=0;y<height;y++) {
for (x=0;x<width;x++) {
int tot=pix[0];
if (x>0) tot+=pre[-1];
if (y>0) tot+=pre[-width];
if (x>0 && y>0) tot-=pre[-width-1];
*pre++=tot;
pix += 4;
}
}
// blur step.
pix = dst + (int)radius * width * 4 + (int)radius * 4 + channel;
for (y=radius;y<height-radius;y++) {
for (x=radius;x<width-radius;x++) {
int l = x < radius ? 0 : x - radius;
int t = y < radius ? 0 : y - radius;
int r = x + radius >= width ? width - 1 : x + radius;
int b = y + radius >= height ? height - 1 : y + radius;
int tot = precalc[r+b*width] + precalc[l+t*width] -
precalc[l+b*width] - precalc[r+t*width];
*pix=(unsigned char)(tot*mul);
pix += 4;
}
pix += (int)radius * 2 * 4;
}
}
memcpy( src, dst, width*height*4 );
}
free( dst );
free( precalc );
}
int main(int argc, char* argv[])
{
cairo_surface_t* surface;
cairo_t* ctx;
cairo_text_extents_t text_extents;
cairo_font_extents_t font_extents;
double FontSize = 100;
double radius = 7;
double width, height;
if ( argc != 3 ) {
printf("Syntax: %s <outfile.png> "<text to display>"n", argv[0]);
return -1;
}
// Get text size.
surface = cairo_image_surface_create( CAIRO_FORMAT_ARGB32, 10, 10 );
ctx = cairo_create( surface );
cairo_set_font_size( ctx, FontSize );
cairo_font_extents( ctx, &font_extents );
cairo_text_extents( ctx, argv[2], &text_extents );
height = font_extents.ascent + font_extents.descent + radius * 2;
width = text_extents.x_advance + radius * 2;
cairo_destroy( ctx );
cairo_surface_destroy( surface );
// Draw text.
surface = cairo_image_surface_create( CAIRO_FORMAT_ARGB32, width, height );
ctx = cairo_create( surface );
cairo_set_font_size( ctx, FontSize );
cairo_move_to( ctx, 0 + radius, font_extents.ascent + radius );
cairo_show_text( ctx, argv[2] );
cairo_fill( ctx );
cairo_image_surface_blur( surface, 5 );
cairo_move_to( ctx, 0, font_extents.ascent );
cairo_show_text( ctx, argv[2] );
cairo_fill( ctx );
cairo_destroy( ctx );
cairo_surface_write_to_png( surface, argv[1] );
cairo_surface_destroy( surface );
return 0;
}
</pre><ul><li><a href='?id=105'>Finding awesome developers in programming interviews</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=73'>How to run a linux based home web server</a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a></ul>
Automatically remove wordiness from your writing
tag:smhanovtechblog,2007:id52
2009-03-04T11:37:18-05:00
2009-03-04T11:37:18-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<style>
.add {
}
.del {
display: none;
}
.addShown {
background: yellow;
}
.delShown {
background: yellow;
text-decoration: line-through;
display: inline;
}
</style>
<form name=f2 action="index.php" method="POST">
<textarea cols=80 rows=10 name="word"/>I wish to take this opportunity to say that I think, in the final analysis, we should address this problem at the earliest possible date.</textarea>
<br>
<input type=button value="Apply Zinsser Transform" onclick='zinsser_xmlhttpPost()'/>
<div style="color:red; text-decoration: blink;" id=status> </div>
<div id=debug> </div>
</form>
<a href="javascript:void(0)" onclick="toggleDiffs()">Show/Hide
Changes</a>
<div id="zinresults"></div>
<script type="text/javascript">
function zinsser_getquerystring() {
var form = document.forms['f2'];
var word = escape(form.word.value);
return word;
}
function zinsser_xmlhttpPost() {
zinsser_updatePage('[2,"Working..."]');
var strURL = "http://stevehanov.ca/cgi-bin/zinsser.cgi";
var xmlHttpReq = false;
var self = this;
// Mozilla/Safari
if (window.XMLHttpRequest) {
self.xmlHttpReq = new XMLHttpRequest();
}
// IE
else if (window.ActiveXObject) {
self.xmlHttpReq = new ActiveXObject("Microsoft.XMLHTTP");
}
self.xmlHttpReq.open('POST', strURL, true);
self.xmlHttpReq.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
self.xmlHttpReq.onreadystatechange = function() {
if (self.xmlHttpReq.readyState == 4) {
zinsser_updatePage(self.xmlHttpReq.responseText);
}
};
self.xmlHttpReq.send("text="+zinsser_getquerystring());
}
function zinsser_updatePage(str) {
var a = eval( str );
if (a) {
div = $("div#zinresults");
div.empty();
p = $("<p>");
div.append(p);
for (var i = 0; i < a.length; i++ ) {
if ( a[i] == 0 ) {
var span1 = $("<span>").addClass("del").text(a[i+1]);
var span2 = $("<span>").addClass("add").text(a[i+2]);
p.append(span1);
p.append(span2);
i += 2;
} else if ( a[i] == 1 ){
var span1 = $("<span>").addClass("del").text(a[i+1]);
p.append(span1);
i += 1;
} else if ( a[i] == 2 ) {
var span1 = $("<span>").addClass("add").text(a[i+1]);
p.append(span1);
i += 1;
} else if ( a[i] == 3 ) {
var span1 = $("<span>").text(a[i+1]);
p.append(span1);
i += 1;
} else if ( a[i] == 4 ) {
p = $("<p>");
div.append(p);
}
}
}
}
function toggleDiffs()
{
$(".add").toggleClass("addShown");
$(".del").toggleClass("delShown");
}
</script>
<p>I recently started re-reading William Zinsser's
<a href="http://www.amazon.com/gp/product/0060891548?ie=UTF8&tag=stehanstecblo-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0060891548">On Writing Well</a><img src="http://www.assoc-amazon.com/e/ir?t=stehanstecblo-20&l=as2&o=1&a=0060891548" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" />. Zinsser emphasizes simplicity in writing. To reduce wordiness, he implores the writer to remove needless words and phrases:
<p>
<blockquote>
"I might add," "It should be pointed out," "It is interesting to note that" how many sentences begin with these dreary clauses announcing what the writer is going to do next? If you might add, add it. If it should be pointed out, point it out. If it is interesting to note, make it interesting. Being told that something is interesting is the surest way of tempting the reader to find it dull; are we not all stupefied by what follows when someone says, "This will interest you"? As for the inflated prepositions and conjunctions, they are the innumerable phrases like "with the possible exception of" (except), "due to the fact that" (because), "he totally lacked the ability to" (he couldn't), "until such time as" (until), "for the purpose of" (for).
</blockquote>
<p>
It's not only dry corporation speak that you should worry about. <b>Actually, what I mean to say is that</b> a little <b>bit of</b> wordiness <b>totally</b> creeps into informal writing <b>way</b> more than you'd think. If you do any <b>sort of</b> writing on the web, you <b>seriously</b> need to think about editing, and <b>more often than not,</b> this tool can help point out some bad habits.
<p>
You might be concerned that your writing will lose its personality. Zinsser <a href="http://www.cla.wayne.edu/polisci/kdk/general/sources/zinsser.htm">goes on to say</a>:
<blockquote>You will reach for gaudy similes and tinseled adjectives, as if "style" were something you could buy at a style store and drape onto your words in bright decorator colors. (Decorator colors are the colors that decorators come in.) Resist this shopping expedition: there is no style store. ... Style is organic to the person doing the writing, as much a part of him as his hair, or, if he is bald, his lack of it. Trying to add style is like adding a toupee.</blockquote>
<p>
You don't want your blog to wear <i>a toupee</i>, do you? Writing style isn't about needless words. Once you remove them, your thoughts will shine through, clearer and more powerful, and then you can then build them back up. This takes time, but your readers will appreciate it.
<p>
By using sources on the web, I came up with about 600 simple substitution rules to cut out wordy phrases, and encoded them into a python script.
Along with other sources, I used Jeff Atwood's <a href="http://www.codinghorror.com/blog">Coding Horror</a> blog to train it, <del>[edit] as he seems to have a high wordiness factor</del>, because I wondered if I could get a web celebrity to notice my little blog, and it totally worked.
<p>
Try it out above. Paste your entire blog article, essay, or email into it. Download the <a href="http://stevehanov.ca/cgi-bin/zinsser.cgi?quine=1">python source</a> here.
<p>
Unfortunately Internet explorer 6 has some problem with my code...
<h4>You might also be interested in:</h4>
<ul>
<li><a href="http://gandolf.homelinux.org/~smhanov/blog/?id=8">A rhyming engine</a>
<li><a href="http://gandolf.homelinux.org/~smhanov/blog/?id=9">What does your phone number spell?</a>
<ul><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=77'>Pitching to VCs #2 (comic)</a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a><li><a href='?id=26'>A simple command line calculator</a><li><a href='?id=9'>What does your phone number spell?</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a></ul>
Why Perforce is more scalable than Git
tag:smhanovtechblog,2007:id50
2009-02-23T21:31:22-05:00
2009-02-23T21:31:22-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Okay, say you work at a company that uses Perforce (on Windows). So you're happily tapping away using perforce for years and years. Perforce is pretty fast -- I mean, it has this "nocompress" option that you can tweak and turn on and off depending on where you are, and it generally lets you get your work done. If you change your client spec, it synchronizes <i>only the files it needs to</i>. Wow, that's blows the mind! Perforce is great, why would you ever need anything else? And its way better than CVS.
<p>
Suddenly you have to clone something with <i>git</i>, and BAM! The world is changed. You feel it in the water. You feel it in the earth. You smell it in the air. Once you've experienced git, there is no going back, man. Git is the stuff man. You might have checked out firefox -- but have you checked out firefox <i>ooon GIT</i>?
<p>
So many really obvious things are missing in p4. Want to restore your source tree to a pristine state? "git clean -fd". Want to store your changes temporarily to work on something else? "git stash". Share some code with a cube-mate without checking in? "git push". Want to automatically detect out of bounds array accesses and add missing semicolons to all your code? "git umm-nice-try"
<p>
Branching on git is like opening a new tab in a browser. It's a piece of cake. You can branch for <i>EVERY SINGLE BUGFIX</i>. And you wrote the code, so you get to merge it back in, because <i>you are the expert</i>.
<p>
Branching on Perforce is kind of like performing open heart surgery. It should only be done by professionals: experts in the art who really know what they are doing. You have to create a "branch spec" file using a special syntax. If you screw up, the entire company will know and forever deride you as the idiot who deleted "//depot/main". The merging is done by gatekeepers. Hope they know what they're doing!
<p>
Now, if you have been using git for a few days you might discover this tool called "git-p4". "AHA!" you might say, "I can import from my company's p4 server into git and work from that, and then submit the changes back when I am done," you might say. But you would be wrong, for a number of reasons.
<h3>git-p4 can't handle large repositories</h3>
<p>
Really. It's just a big python script, and it works by downloading the entire p4 repository into a python object, then writing it into git. If your repo is more than a couple of gigs, you'll be out of memory faster than you can skim reddit.
<p>
But that problem's fixable. I was able to <a href="git-p4.py.txt">hack up git-p4</a> to do things a file at a time in about an hour. The real problem is:
<h3>Git can't handle large repositories</h3>
<p>
Okay this is subjective because it depends on your definition of large. When I say large, I mean about 6 gigs or so. Because your company's source tree is probably that large. If you have the power, you will use it. Maybe you check in binaries of all your build tools, or maybe for some reason you need to check in the <i>object files of the nightly builds</i>, or something silly like that. P4 can handle this because it runs on a cluster of servers somewhere in the bowels of your company's IT department, administered by an army of drones tending to its every need. It has been developed <a href="http://www.perforce.com/perforce/corp.html">since 1995</a> to handle the strain. Google also uses Perforce, and <a href="http://www.perforce.com/perforce/conferences/eu/2006/presentations/Google.pdf">when it started to show its strain</a>, Larry Page personally went to Perforce's headquarters and threatened to direct large amounts of web traffic up their executives' whazzoos until they did something about it.
<p>
Git has none of that. The typical git user considers the linux kernel to be a "large project". If you've looked at <a href="http://www.youtube.com/watch?v=4XpnKHJAok8">Linus's git rant</a> on Google code, take a listen to see how he sidesteps the question of scalability.
<p>
Don't believe me? Fine. Go ahead and wait a minute after every git command while it scans your entire repo. It's maddening because its long enough to be annoying, but not enough time to skim <a href="http://www.geekologie.com">Geekologie.</a>
<h2>The solution</h2>
<p>
You know what? I don't think many people really use distributed source control. The centralized model is here to stay. Most git users (especially those using Github) use the centralized model anyway.
<p>
Ask yourself this: Is it really that important to duplicate the entire history on every single PC? Do you really need to peruse changelist 1 of KDE from an airplane? In most cases, NO. What you really want is the other stuff: easy branching, clean, and stash, and the ability to transfer changes to another client. The distributed stuff isn't really asked for, or needed. It just makes it <a href="http://blog.aisleten.com/2008/11/23/git-the-fsck-out/">hard to learn</a>.
<p>
Just give me a version control system that lets me do these things and I'll be happy:
<ul>
<li>Let me merge changes into my coworker's repos, without having to check them in first.
<li>Let me "stash" stuff cause it's really handy. Clean is nice to have too.
<li>Make branching easy.
<li>Don't waste 40% of my disk space with a .git folder, when this could be stored on a central server.
</ul>
<p>
Is that really so hard?
<hr>
<ul>
<li><a href="http://www.reddit.com/r/programming/comments/84pfj/why_perforce_is_more_scalable_than_git/">Long discussion of this post on reddit</a>
<li><a href="http://www.reddit.com/r/programming/comments/8stel/why_perforce_is_more_scalable_than_git/
">Another long discussion of this post on reddit</a>
<li><a href="http://news.ycombinator.com/item?id=516276">Long discussion of this post on ycombinator</a>
</ul><ul><li><a href='?id=137'>Asana's shocking pricing practices, and how you can get away with it too</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=13'>Draw waveforms and hear them</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=101'>"Your program is stupid. It doesn't work," my wife told me</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a></ul>
Optimizing Ubuntu to run from a USB key or SD card
tag:smhanovtechblog,2007:id48
2009-01-25T20:17:06-05:00
2009-01-25T20:17:06-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
If you've installed Ubuntu on a USB key or SD card, you are probably experiencing the annoying slowness of Firefox. It freezes up for a couple of seconds every time you click a link. Like many things on Ubuntu, it doesn't work right out of the box and needs some tweaking. Fortunately, by following the tips below, you can make your USB or SD card based linux system fly!
<h3>Tip 1: Stop Firefox from writing to disk</h3>
<p>
Firefox 3 has a hard-to-fix bug that causes linux to write to disk every time you visit a write page. But it doesn't just write Firefox stuff -- it causes your entire system to dump all changes to disk. Unfortunately, your USB key might only have a 6 MB/s write speed, causing everything to freeze up.
<ol>
<li>Under the Privacy setting, uncheck "Keep my history for..."
<li>Under the Advanced Tab, select Network and ensure that you use up to 0 MB of disk space for the cache.
</ol>
<p>
<h3>Tip 2: Use preload</h3>
The preload daemon is a program that constantly looks at the programs you are running and figures out which ones you are most likely to use. When you start your computer, it automatically loads these programs and library from disk in the background, so when you start firefox, for example, it will pop up right away. The background is described in <a href="http://behdad.org/download/preload.pdf">the author's Master's thesis.</a>
<p>
It's kind of like putting magnets under your pillow to improve health. Maybe it's having an effect, but I can't tell. I install it anyway:
<p>
<pre>
sudo aptitude install preload
</pre>
<h3>Tip 3: Compress your files</h3>
<font color="#800000">This tip can wreck your system, and to undo it you will need to be able to use a command line editor like nano, emacs, or vim. At worst you will need to mount the USB key on another linux system to recover (by editing /etc/fstab). If you can't do that, then skip this tip.</font>
<p>
On solid state storage, space is expensive. Ubuntu uses a huge amount of space will all the programs it installs. The /usr folder contains your programs, and it is usually 1.8 GB. Using squashfs, it can be compressed to 0.7 GB. Since read speeds are so slow, you can actually gain performance because there is less data to read. I've adapted these instructions from <a href="http://po-ru.com/diary/linux-liposuction-or-xubuntu-in-under-a-gig-on-the-eee-pc/">here</a>.
<p>
Install squashfs and unionfs:
<pre>
sudo apt-get install squashfs-tools unionfs-tools
</pre>
<p>
Add the following lines to /etc/modules:
<pre>
unionfs
squashfs
loop
</pre>
<p>
Remove apparmor. Otherwise, the cups print server will stop working:
<pre>
sudo apt-get purge apparmor
</pre>
<p>
Make space for the filesystem:
<pre>
sudo mkdir -p /.filesystems/usr/overlay
</pre>
Compress your filesystem:
<pre>
sudo mksquashfs /usr /.filesystems/usr/usr.sqfs
</pre>
Add these lines to /etc/fstab:
<pre>
/.filesystems/usr/usr.sqfs /usr squashfs ro,loop,nodev 0 0
unionfs /usr unionfs nodev,noatime,dirs=/.filesystems/usr/overlay=rw:/usr=ro 0 0
</pre>
Switch to runlevel 1. (Ubuntu will close all open programs, then prompt you what to do. Choose opening a root shell)
<pre>
sudo init 1
</pre>
<p>
Move aside the old /usr directory and create a new mount point:
<pre>
mv /usr /usr.old
mkdir usr
</pre>
<p>
Test whether you previously edited fstab successfully by typing:
<pre>
mount -a
</pre>
<p>
If you get error messages or your /usr directory shows up empty, either fix
your fstab or undo the changes before continuing.
<p>
Now reboot and make sure it all works:
<pre>
reboot
</pre>
<p>
If it works, remove the /usr.old directory to reclaim the space.
<h3>Tip 4: Use memory instead of disk</h3>
<p>
<font colour="#800000">This tip can also lead to data loss. If you do it, you will have to always shut down your computer properly from now on, because unexpected power failures will lead to data loss.</font>
<p>
Linux usually ensures that all changes are written to disk every few seconds. Since disk writes are so slow, you can change your system to keep things in memory longer. All changes will be written to memory, and the excruciatingly slow writes to happen in the background while you continue working. <em>This has an instant, noticeable effect, but it can lead to data loss.</em>
<p>
Add these lines to /etc/sysctl.conf, and reboot.
<pre>
vm.swappiness = 0
vm.dirty_background_ratio = 20
vm.dirty_expire_centisecs = 0
vm.dirty_ratio = 80
vm.dirty_writeback_centisecs = 0
</pre>
<p>
The problem: using this tip means that your system stops writing changes to disk until you shut down or type "sync" at a command line. If your system loses power unexpectedly, you will get bad blocks. I did. You can limit the amount of data loss in the event of a power failure to one minute by setting vm.dirty_writeback_centisecs = 6000.</i>
<p>
A side effect is that shutting down your computer will may take several minutes where it appears to be doing nothing. Don't cut the power until it's done, because it is busy writing all those changes to disk.
<p><ul><li><a href='?id=66'>Test Driven Development without Tears</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a><li><a href='?id=39'>Stock Picking using Python</a><li><a href='?id=37'>Spoke.com scam</a></ul>
UMA Questions Answered
tag:smhanovtechblog,2007:id38
2008-12-20T10:08:32-05:00
2008-12-20T10:08:32-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Geez, I just went through my server logs and it is clear that people have lots of questions on UMA. Whenever someone asks a question in Google, and my web page pops up, and they click on it, I can see what they typed into Google. So in a way, all of you people on the Internet are able to tell me what to write about. So here is my page with what you want to know about UMA.
<p>
Because I work for RIM, there is a general employee directive to not give any technical support online. So this article relates to all cell phones in general. I certainly don't want to get fired. (Ssssh! They don't know about this blog). Oh yeah, almost forgot:
<blockquote>
The postings on this site are my own and don't necessarily represent the position, opinions or strategies of RIM
</blockquote>
<p>
You might also be interested in my <a href="http://stevehanov.ca/blog/?id=2">general description of UMA</a>, and whether you can get <a href="http://stevehanov.ca/blog/?id=4">free long distance over UMA.</a>
<h2>"how does UMA connection work"</h2>
From your perspective, you get one phone number that will work over your Internet connection when you are at home, and over cell towers when you are outside. If you travel to another country, you could <a href="http://gandolf.homelinux.org/~smhanov/blog/?id=4">possibly make calls to your home town and not be charged long distance.</a> However, UMA is not skype. Other than that exact situation, you will normally be charged for long distance calls depending on your plan.
<p>
Normally your phone sends its signals to a cell tower, which forwards it to a server on the carrier's network. With UMA, your phone logs into the carrier's network through your internet connection and sends its signals directly to that server. That means you can access all of the same services over UMA, like voice, data, and SMS. Unfortunately, it means that all of those same services go through your carrier's network, perhaps unnecessarily. Some phones support something called "Internet Offload", in which UMA is only used for voice calls, but all the data goes directly over the Internet. Got it?
<p>
<h2>"connecting a uma phone to the network."</h2>
<p>
Connecting UMA is hard. Normal cell phones are designed so that any idiot with a bank account can use them. But unfortunately, UMA is based on the Wifi network, and Wifi has a lot of lot of things that you can screw up. To connect to UMA, six things have to happen:
<ol>
<li><b>Your cell phone has to be specifically designed for UMA</b>. It has to contain a Wifi radio on it, or it just isn't going to work. You can't take your 10 year old Motorola and sign up for UMA service. It has to be fairly new -- probably made in the last two years. Furthermore, it has to contain the UMA software on it. You pretty much have to have the phone and OS designed to accommodate UMA, because the UMA application needs direct access to secure sockets, audio, and SIM card security functions.
<li><b>You have to sign up for UMA service with your carrier</b>. Their security gateway knows whether you are paying for their service and it won't let you through if you didn't sign up.
<li><b>You have to have the security certificate installed on your phone</b>. Each carrier (for example, T-Mobile, or Rogers in Canada) has a certificate that has to be loaded onto your phone for you to connect. It is kind of like a password, but it is far too long to type in manually, so it has to be loaded with special software. This probably happened when your phone was manufactured and customized for the carrier. Unfortunately, that means that it is locked to a particular carrier, unless you can figure out a way to load a new certificate. This is what gives people <a href="http://www.blackberryforums.com/general-8900-series-discussion-javelin/164976-uma-not-4-6-1-76-how-about-4-6-1-94-a.html">problems when they try to use a phone bought on ebay</a>.
<li><b>You have to successfully connect to Wifi</b>. This will involve scanning for the Wifi network using your phone, and probably entering a WEP or WPA password of some kind. You should try this on your laptop first, to see if your Internet is set up properly. If your laptop can't connect, then there is some other problem with your Wifi router.
<li><b>Your phone has to be configured to use the UMA connection</b>. The mobile phone standards dictate that all phones have to have some way of choosing how UMA is used. You have to be able to choose between cellular-only, cellular-preferred, UMA only, and UMA preferred modes of operation. The only option that makes sense is UMA-preferred. (This is sometimes called "Wifi preferred")
<li><b>Finally, UMA must successfully connect.</b> Turn off your microwave oven. Stop bittorrent. At this point, one missed packet can cause a huge delay. The connection phase retry can take up to 32 minutes, because the <a href="http://www.3gpp.org/ftp/Specs/archive/44_series/44.318/44318-770.zip">mobile standards</a> describe a precise scheme of retries. Specifically, it tries three times, waiting 30 seconds between tries. If those tries fails, it waits 2 minutes, then tries three more times. On failure, it doubles the two minutes to 4 minutes, and so forth, until it eventually waits 32 minutes between retries. Then it stops doubling the timeout. To avoid the 32 minute wait, you can probably make it try again immediately by turning off and on the phone.
</ol>
<h2>"how to use uma phone without sim"</h2>
<p>
It is not possible to use UMA without signing up for an UMA plan with a carrier. That is because the SIM card actually performs the authentication process to login to UMA. No SIM means no UMA.
<h2>"does uma work over different carriers"</h2>
<p>
If you travel, you can use your UMA phone to connect to your home carrier and make calls without roaming charges. Your main problem will be connecting your phone to wireless hotspots, because often you will have to login to the hotspot and click "I agree" to some usage agreement. If your phone browser isn't able to display the button, then you won't be able to connect to Wifi. But if you pass this hurdle, your phone should work fine. Just make sure it is actually using the UMA network when you make your call. In this case, it might be good to set it to UMA only mode.
<h2>"how to make uma server"</h2>
<p>
So you want to make an UMA server so you can run your own phone service. Okay, well, first you have to remember that UMA is simply a wrapper around the regular cell phone messages so they can be transported over the Internet. So all you have to do is implement the entire rest of that phone network. Feel free to download the <a href="http://www.3gpp.org">3GPP specifications</a> and implement your own phone network. Don't let me stop you. Out of the 1,000,000 pages of telephony standards, I think you'd only have to read about 50,000.
<p>
Or you could just go write a SIP server like a normal person.
<p>
<h2>"which carriers have uma"</h2>
<p>
Umm, I don't know. I know T-Mobile US is doing it and Rogers Canada is as well. There is a complete list on <a href="http://en.wikipedia.org/wiki/Unlicensed_Mobile_Access">wikipedia</a>.
<h2>"best uma phone"</h2>
<p>
The <a href="http://www.umatoday.com/awards2009.php">best UMA cell phone</a> is the BlackBerry, which does support Internet Offload. The latest model at this point is the 8900. Go out and buy that one as soon as you can. <a href="http://www.blackberryforums.com/general-8900-series-discussion-javelin/164976-uma-not-4-6-1-76-how-about-4-6-1-94-a.html">But be careful if it is used.</a>
<h2>"UMA skype"</h2>
<p>
Skype shouldn't be used over UMA, because as I have said, on most phones, all data goes through the carriers servers, and is chargeable. Your Skype usage would cost thousands of dollars in data charges. If your phone supports Skype and Wifi, and you can turn of UMA, do so to avoid these charges.
<h2>"is uma service free"</h2>
<p>
That depends on your plan and contract. You did read it, didn't you? Usually you have to pay extra for UMA, but then you get unlimited calling while on UMA. But read your plan very carefully.
<p>
If you make long distance calls, you will probably be charged extra.
<p>
<em>But the number one search topic for UMA is:</em>
<h2>"uma nude"</h2>
<p>
I don't have all the answers about UMA! For that, you'll have to look elsewhere.
<ul><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=25'>Tool for Creating UML Sequence Diagrams</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=37'>Spoke.com scam</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=115'>Compressing dictionaries with a DAWG</a></ul>
See sound without drugs
tag:smhanovtechblog,2007:id41
2008-12-09T19:19:33-05:00
2008-12-09T19:19:33-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
To further my understanding of frequency analysis and the fast fourier transform, I have created an application that just turns on the microphone and continually plots the FFT magnitude of what it records. It allows control over the window size and sampling rate.
<p>
<center>
<img src=soundlab.jpg><br>
<a href="SoundLabInstall.exe">Download SoundLab</a>
</center>
</p>
<p>
Because the FFT of a real valued function is even, we only display the first half.
<p>
It's fun to whistle and see the result. I think it would be cool to make a game out of this. For example, you could make pong and control the paddle with sound. I plan to call it Whistle Hero.
<p>
For some more fun, run soundlab at the same time as <a href="?id=13">WaveStudio</a>, an older application I made that lets you draw a waveform and hear it. That way you can see a waveform that you create in the time domain in the frequency domain, and hear it all at the same time. Its actually quite challenging to try to make a perfect soundwave, and eliminate all of the harmonics.
<ul><li><a href='?id=137'>Asana's shocking pricing practices, and how you can get away with it too</a><li><a href='?id=117'>Why don't web browsers do this?</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=10'>Detecting C++ memory leaks</a><li><a href='?id=9'>What does your phone number spell?</a><li><a href='?id=13'>Draw waveforms and hear them</a></ul>
Stock Picking using Python
tag:smhanovtechblog,2007:id39
2008-10-18T17:16:08-05:00
2008-10-18T17:16:08-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
The stock market is a lot different than it was just a few months ago. Once again, I present my stock selections, as found via python script. Comparing it with <a href="index.php?id=31">last time</a>, you will find most of the same names are on there.
<ol>
<li>Financial data for 1600 public companies listed on the TSX is downloaded from http://finance.google.com.
<li>The annual and quarterly revenue and earnings is scraped from the HTML file using an <a href="http://www.cs.helsinki.fi/u/jjaakkol/sgrepman.html">sgrep</a> query.
<li>Each company is filtered according to the following criteria:
<ul>
<li>Resource, mining, and energy stocks are excluded.
<li>Stocks with PE ratio higher than 50 are excluded.
<li>Stocks which had negative revenue or earnings in the past two years are excluded.
</ul>
<li>The remaining stocks are sorted by growth and displayed here.
</ol>
<p>
<a href="https://github.com/smhanov/pleco-stock-scraper">Source code on Github</a>
<pre>
Data are from October 18, 2008.
SYMBOL COMPANY NAME Revenue Growth EPS Growth Price P/E
(Years positive) (Years Positive)
--------------------------------------------------------------------------------------------
cvg Clairvest Group Inc. 183% (2) 509% (2) 12.50 5
aui AltaGas Utility Group Inc. 165% (2) 127% (2) 5.59 8
chl.un Canadian Helicopters Income Fund 140% (2) 629% (2) 6.36 3
ibg.un IBI Income Fund 94% (3) 58% (2) 14.50 6
gvc Glacier Media Inc. 88% (3) 57% (3) 3.07 7
drx ADF Group, Inc. 82% (2) 785% (3) 1.95 1
rcm RuggedCom Inc. 78% (3) 35% (2) 12.00 22
prk Parkbridge Lifestyle Communities Inc. 70% (3) 119% (2) 4.00 12
rim Research In Motion Limited 65% (3) 86% (3) 70.25 22
mre Martinrea International Inc. 58% (3) 68% (3) 4.67 6
cjt.un Cargojet Income Fund 53% (2) 477% (2) 3.10 44
pbl.un Pollard Banknote Income Fund 52% (2) 68% (2) 7.12 7
cwa.un Coast Wholesale Appliances Income Fund 51% (2) 5% (2) 3.99 5
vnp 5N Plus Inc. 44% (3) 241% (2) 4.08 14
arf.un Armtec Infrastructure Income Fund 43% (3) 46% (3) 15.21 7
wpo World Point Terminals Inc. 43% (2) 43% (2) 12.07 14
et Evertz Technologies Limited 40% (3) 977% (3) 14.00 12
aah Aastra Technologies Limited 38% (3) 34% (2) 10.13 6
etc Equitable Group Inc. 37% (3) 23% (3) 15.00 5
ylo.un Yellow Pages Income Fund 37% (3) 42% (3) 8.40 9
mrd Melcor Developments Ltd. 36% (3) 55% (3) 6.50 4
gmp.un GMP Capital Trust 35% (3) 39% (3) 6.39 4
pki.un Parkland Income Fund 35% (3) 209% (3) 6.78 5
otc Open Text Corporation 33% (2) 232% (2) 31.07 31
wes Western Financial Group 33% (3) 21% (2) 2.90 13
cuq The Churchill Corporation 30% (3) 123% (2) 5.91 3
sz Sceptre Investment Counsel Limited 29% (2) 41% (2) 6.75 12
mlx Marsulex Inc. 28% (3) 302% (2) 8.00 11
sj Stella-Jones Inc. 28% (3) 44% (3) 23.00 10
wja WestJet Airlines Ltd. 26% (3) 215% (2) 10.87 6
sfi Saxon Financial Inc. 25% (3) 19% (3) 20.86 19
gna Gerdau Ameristeel Corporation 24% (3) 30% (2) 7.30 4
sif.un Energy Savings Income Fund 23% (3) 67% (3) 10.95 7
cmg Computer Modelling Group Ltd. 23% (3) 28% (3) 8.50 15
stn Stantec Inc. 23% (3) 24% (3) 18.64 11
pbi.un Premium Brands Income Fund 22% (3) 79% (2) 9.20 7
cca Cogeco Cable Inc. 22% (3) 72% (2) 35.30 12
inn.un InnVest Real Estate Investment Trust 22% (3) 45% (2) 5.66 7
gil Gildan Activewear Inc. 21% (3) 29% (3) 24.94 18
wfi WaterFurnace Renewable Energy, Inc. 20% (3) 27% (3) 24.35 25
x TMX Group, Inc. 20% (3) 30% (3) 30.02 13
gdi.un General Donlee Income Fund 20% (3) 78% (2) 5.75 5
fc.un Firm Capital Mortgage Investment Trust 20% (3) 2% (2) 9.26 8
rba lRitchie Bros. Auctioneers 20% (3) 29% (3) 23.31 26
lgi Logibec Groupe Informatique Ltd. 18% (3) 26% (3) 15.00 20
mda Macdonald Dettwiler & Associates Ltd 17% (3) 18% (3) 16.85 7
eh easyhome Ltd. 16% (3) 42% (2) 12.20 11
dsg Descartes Systems Group Inc. 13% (2) 203% (2) 2.82 6
gcg Guardian Capital Group Ltd. 13% (3) 38% (3) 7.50 12
thi Tim Hortons Inc. 13% (3) 9% (2) 28.55 19
cae CAE, Inc. 13% (3) 54% (2) 6.80 10
igm IGM Financial Inc. 10% (3) 13% (3) 35.52 10
ftt Finning International Inc. 10% (3) 29% (3) 15.20 9
rch Richelieu Hardware Ltd. 10% (3) 9% (3) 17.86 11
fgl The Forzani Group Ltd. 10% (3) 90% (2) 9.50 8
nwf.un North West Company Fund 10% (3) 19% (3) 14.65 10
tih Toromont Industries Ltd. 9% (3) 20% (3) 24.08 12
sc Shoppers Drug Mart Corporation 8% (3) 16% (3) 45.15 18
kos Cossette Communication Group Inc. 8% (3) 17% (2) 4.51 7
lnf Leon's Furniture Ltd. 8% (3) 10% (3) 9.70 11
lnr Linamar Corporation 8% (3) 6% (3) 8.84 5
acc Amica Mature Lifestyles Inc. 7% (2) 33% (3) 4.49 19
alc Algoma Central Corporation 7% (3) 32% (3) 88.00 6
clc.un CML Healthcare Income Fund 7% (2) 5% (3) 12.63 11
cp Canadian Pacific Railway Limited 6% (3) 33% (3) 47.71 9
t TELUS Corporation 6% (3) 35% (3) 39.62 9
idg Indigo Books & Music Inc. 5% (3) 67% (3) 12.50 5
hlf High Liner Foods Incorporated 4% (2) 41% (2) 7.90 13
</pre><ul><li><a href='?id=55'>How wide should you make your web page?</a><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=138'>Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a></ul>
Spoke.com scam
tag:smhanovtechblog,2007:id37
2008-09-24T08:53:48-05:00
2008-09-24T08:53:48-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
I stumbled accross this page about myself on this <a href="http://www.spoke.com">rotten company</a> Spoke.com, who, without my permission, gathered my name and employment history together into <a href="http://center.spoke.com/info/pmdhnz/SteveHanov">one place</a>. I object to it, but there was no obvious way to get it removed. After a lot of searching, I found a contact page and filled it out, but I'm not at all confident that it will be acted on.
<p>
Spoke, if you want me to remove this entry about you, you can opt-out at any time using the contact form below. After I verify your identity I will put your request into a queue for removal.
<ul><li><a href='?id=3'>Cell Phone Secrets</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=135'>How I run my business selling software to Americans</a><li><a href='?id=142'>My thoughts on various programming languages</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=4'>UMA and free long distance</a></ul>
Copy a cairo surface to the windows clipboard
tag:smhanovtechblog,2007:id35
2008-09-19T10:00:00-05:00
2008-09-19T10:00:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
I just spent several hours debugging clipboard copy of a DIB image. I could copy from my application, and paste into Paint. I could paste into Word. But if I pasted into WordPad, nothing showed up. If I pasted into GIMP, it crashed.
<P>
The general procedure is to fill out a BITMAPINFO structure, calculate the size of the image + row padding + the bitmap info structure itself, then allocate a memory handle with GlobalAlloc(). Finally, copy the BITMAPINFO structure into the given memory followed by the image pixel data.
<p>
What they don't tell you in MSDN is that, for maximum compatibility with other applications, you must use a positive value for the BITMAPINFOHEADER biHeight member. That means that you have to <i>create the bitmap <em>upside down</em></i>
<p>
The other thing they don't tell you, unless you're reading <i>really, really <a href="http://msdn.microsoft.com/en-us/library/ms532284(VS.85).aspx">carefully</a></i>, is that you have to insert padding at the end of the rows so that they always end at a DWORD (4 byte) boundary.
<p>
Anyway, here's a code snippet. Hopefully it will help somebody someday. If so, give me $2. Really.
<p>
<form action="https://www.paypal.com/cgi-bin/webscr" method="post">
<input type="hidden" name="cmd" value="_s-xclick">
<input type="image" src="https://www.paypal.com/en_US/i/btn/x-click-but21.gif" border="0" name="submit" alt="Make payments with PayPal - it's fast, free and secure!">
<img alt="" border="0" src="https://www.paypal.com/en_US/i/scr/pixel.gif" width="1" height="1">
<input type="hidden" name="encrypted" value="-----BEGIN PKCS7-----MIIHPwYJKoZIhvcNAQcEoIIHMDCCBywCAQExggEwMIIBLAIBADCBlDCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20CAQAwDQYJKoZIhvcNAQEBBQAEgYB9LnH031yC43SlOkBIxhIzbMGPa8f8i+1sqA6AYlPzvBak+u9NgU7PQ7G84QeBwviTM+lVVDetb5xYvn7qNpkWA2adZW9helcRrXhPx0tVezit65z2s/xTXhZsHYIKtyCwrcvR6MLg78+nMxDauMvxl/+oEinRGml999Lf0bf7iTELMAkGBSsOAwIaBQAwgbwGCSqGSIb3DQEHATAUBggqhkiG9w0DBwQIsI6TkJF3CBuAgZiQrEy1Hg6L8cLEtYKjE8hOcGFhy9adBJ/NxWauijG/j9yiXUHMi/RfTV3ozRiBjPNmJhcryMTI30BHvwLmCtnnQiGmECfrmKJZOzBXP5atJpVGqUZUM/0l5LOtHK4cpIHJfyj0bhUKAPA/vuTzPLSfDnIMQ3HNTYZk7rD5g84xNJkp18JNuDYjnCCStAfICpaQItQx47VpGaCCA4cwggODMIIC7KADAgECAgEAMA0GCSqGSIb3DQEBBQUAMIGOMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxFDASBgNVBAoTC1BheVBhbCBJbmMuMRMwEQYDVQQLFApsaXZlX2NlcnRzMREwDwYDVQQDFAhsaXZlX2FwaTEcMBoGCSqGSIb3DQEJARYNcmVAcGF5cGFsLmNvbTAeFw0wNDAyMTMxMDEzMTVaFw0zNTAyMTMxMDEzMTVaMIGOMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxFDASBgNVBAoTC1BheVBhbCBJbmMuMRMwEQYDVQQLFApsaXZlX2NlcnRzMREwDwYDVQQDFAhsaXZlX2FwaTEcMBoGCSqGSIb3DQEJARYNcmVAcGF5cGFsLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAwUdO3fxEzEtcnI7ZKZL412XvZPugoni7i7D7prCe0AtaHTc97CYgm7NsAtJyxNLixmhLV8pyIEaiHXWAh8fPKW+R017+EmXrr9EaquPmsVvTywAAE1PMNOKqo2kl4Gxiz9zZqIajOm1fZGWcGS0f5JQ2kBqNbvbg2/Za+GJ/qwUCAwEAAaOB7jCB6zAdBgNVHQ4EFgQUlp98u8ZvF71ZP1LXChvsENZklGswgbsGA1UdIwSBszCBsIAUlp98u8ZvF71ZP1LXChvsENZklGuhgZSkgZEwgY4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEUMBIGA1UEChMLUGF5UGFsIEluYy4xEzARBgNVBAsUCmxpdmVfY2VydHMxETAPBgNVBAMUCGxpdmVfYXBpMRwwGgYJKoZIhvcNAQkBFg1yZUBwYXlwYWwuY29tggEAMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAgV86VpqAWuXvX6Oro4qJ1tYVIT5DgWpE692Ag422H7yRIr/9j/iKG4Thia/Oflx4TdL+IFJBAyPK9v6zZNZtBgPBynXb048hsP16l2vi0k5Q2JKiPDsEfBhGI+HnxLXEaUWAcVfCsQFvd2A1sxRr67ip5y2wwBelUecP3AjJ+YcxggGaMIIBlgIBATCBlDCBjjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQKEwtQYXlQYWwgSW5jLjETMBEGA1UECxQKbGl2ZV9jZXJ0czERMA8GA1UEAxQIbGl2ZV9hcGkxHDAaBgkqhkiG9w0BCQEWDXJlQHBheXBhbC5jb20CAQAwCQYFKw4DAhoFAKBdMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA3MDMxOTEzMjY1M1owIwYJKoZIhvcNAQkEMRYEFKz7o9gJ/pTGDrfQZHtsA8R0V3GYMA0GCSqGSIb3DQEBAQUABIGAcK5qQzXg2xudoj9wgxf1MTaSAkqPyVkgRVq68F7ZaCocIUK2aSsTsMEQ+WC8rjDFLdLP10zBWmUddJVVW+H1ncq1JPU3Uns4TQ9QESI0HF1Y4jDZ00nZi395KpAb5bEu5/A1zPPi6npOqbcLKHtIsT98LiWTAO2lR3lZIFQL8BE=-----END PKCS7-----
">
</form>
<pre>
// copy a cairo win32 surface (with dib) to the clipboard.
bool
GraphicsClipboardSurface::copyToClipboard(HWND hwnd)
{
cairo_surface_t* imageSurface = cairo_win32_surface_get_image( _surface );
if ( imageSurface == NULL ) {
assert(false);
return false;
}
unsigned char* bits = cairo_image_surface_get_data( imageSurface );
if ( bits == NULL ) {
assert( false );
return false;
}
assert( cairo_image_surface_get_format( imageSurface ) == CAIRO_FORMAT_RGB24 );
BITMAPINFOHEADER bmi;
unsigned biSizeImage;
memset( &bmi, 0, sizeof(bmi) );
bmi.biSize = sizeof(bmi);
bmi.biWidth = cairo_image_surface_get_width( imageSurface );
bmi.biHeight= cairo_image_surface_get_height( imageSurface );
unsigned rowPad = ( 4 - ( ( bmi.biWidth * 3 ) & 3 ) ) & 3;
bmi.biPlanes = 1;
bmi.biBitCount = 24; // 24 or 32. If 32, high byte is not used.
bmi.biCompression = BI_RGB;
biSizeImage = bmi.biWidth * bmi.biHeight * ( bmi.biBitCount / 8 ) + bmi.biHeight * rowPad;
bmi.biXPelsPerMeter = (LONG)((double)96 * 100 / 2.54 + 0.5) ; // dpix
bmi.biYPelsPerMeter = (LONG)((double)96 * 100 / 2.54 + 0.5); // dpiy
bmi.biClrUsed = 0;
bmi.biClrImportant = 0;
HGLOBAL hMem = NULL;
unsigned char* ptr = 0;
unsigned size;
bool success = false;
// OpenClipboard
if ( !OpenClipboard(hwnd) ) {
return false;
}
// call EmptyClipboard
if ( !EmptyClipboard() ) {
goto error;
}
// calculate size of the data.
size = bmi.biSize + biSizeImage;
// Allocate the data using GlobalAlloc with GMEM_MOVEABLE flag.
hMem = GlobalAlloc( GMEM_MOVEABLE, size );
if ( hMem == NULL ) {
goto error;
}
ptr = (unsigned char*)GlobalLock( hMem );
if ( ptr == 0 ) {
goto error;
}
// copy data to clipboard
memcpy( ptr, &bmi, bmi.biSize );
// copy each row of the bitmap in reverse order, adding padding after each
// row.
unsigned char* src = bits + bmi.biWidth * (bmi.biHeight-1) * 4;
unsigned char* dest = ptr + bmi.biSize;
for ( int i = 0; i < bmi.biHeight; i++ ) {
for ( int j = 0; j < bmi.biWidth; j++ ) {
*dest++ = *src++;
*dest++ = *src++;
*dest++ = *src++;
src++;
}
dest += rowPad;
src -= bmi.biWidth * 4 * 2;
}
GlobalUnlock( hMem );
// Call SetClipboardData
if ( !SetClipboardData( CF_DIB, hMem ) ) {
goto error;
}
hMem = NULL;
success = true;
error:
if ( hMem ) {
GlobalFree( hMem );
}
CloseClipboard();
return success;
</pre>
<p>
On the plus side, <a href="http://www.websequencediagrams.com">websequencediagrams.com</a> Desktop Edition is coming along very nicely. I implemented Print yesterday. It should be ready soon... Hopefully some companies will buy it. Here's a screenshot.
<center><p>
<a href="wsd-screenshot-1.png"><img src="wsd-screenshot-1.png" width=640></a>
</center>
<p><b>Update</b>: Of course, by 2011 desktop applications are gone. I made the right call by not finishing the desktop version, and just licensing the whole webserver. More information on this strategy is in <a href="http://stevehanov.ca/blog/index.php?id=95">C++: A language for next generation web apps</a><ul><li><a href='?id=76'>Does Android team with eccentric geeks? (comic)</a><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=48'>Optimizing Ubuntu to run from a USB key or SD card </a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=146'>O(n) Delta Compression With a Suffix Array</a></ul>
Simulating freehand drawing with Cairo
tag:smhanovtechblog,2007:id33
2008-07-30T20:49:28-05:00
2008-07-30T20:49:28-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>Have a look at this image. You might think I scrawled it on a napkin and scanned it in. Wrong! It was completely automatically generated by an upcoming release of <a href="http://www.websequencediagrams.com">www.websequencediagrams.com</a>, with the new "napkin" style. Getting it to render this way was easy, simply with a tiny bit of math and a change to my line drawing function. The handwriting font <a href="http://www.fontco.com/font-info/fgvirgil.php">FG Virgil</a>.
</p>
<center><p><img src=cairo-freehand.png></p></center>
<p>
Here's the same diagram in a different style:
</p>
<center><p><img src="http://www.websequencediagrams.com/sample5.png"></p></center>
<p>Here's the C code that I use for my line drawing function, using the <a href="http://www.cairographics.org/manual">cairo</a> API. </p>
<pre>
void
crazyLine( cairo_t* ctx, double fromX, double fromY, double toX, double toY)
{
// Crazyline. By Steve Hanov, 2008
// Released to the public domain.
// The idea is to draw a curve, setting two control points at random
// close to each side of the line. The longer the line, the sloppier it's drawn.
double control1x, control1y;
double control2x, control2y;
// calculate the length of the line.
double length = sqrt( (toX-fromX)*(toX-fromX) + (toY-fromY)*(toY-fromY));
// This offset determines how sloppy the line is drawn. It depends on the
// length, but maxes out at 20.
double offset = length/20;
if ( offset > 20 ) offset = 20;
// Overshoot the destination a little, as one might if drawing with a pen.
toX += ((double)rand()/RAND_MAX)*offset/4;
toY += ((double)rand()/RAND_MAX)*offset/4;
double t1X = fromX, t1Y = fromY;
double t2X = toX, t2Y = toY;
// t1 and t2 are coordinates of a line shifted under or to the right of
// our original.
t1X += offset;
t2X += offset;
t1Y += offset;
t2Y += offset;
// create a control point at random along our shifted line.
double r = (double)rand()/RAND_MAX;
control1X = t1Y + r * (t2X-t1X);
control1Y = t1Y + r * (t2Y-t1Y);
// now make t1 and t2 the coordinates of our line shifted above
// and to the left of the original.
t1X = fromX - offset;
t2X = toX - offset;
t1Y = fromY - offset;
t2Y = toY - offset;
// create a second control point at random along the shifted line.
r = (double)rand()/RAND_MAX;
control2X = t1X + r * (t2X-t1X);
control2Y = t1Y + r * (t2Y-t1Y);
// draw the line!
cairo_move_to( _ctx, fromX, fromY );
cairo_curve_to( _ctx, control1X, control1Y, control2X, control2Y, toX, toY );
}
</pre>
<ul><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=58'>Email Etiquette</a><li><a href='?id=105'>Finding awesome developers in programming interviews</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=26'>A simple command line calculator</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=117'>Why don't web browsers do this?</a></ul>
Free, Raw Stock Data
tag:smhanovtechblog,2007:id31
2008-06-05T21:23:37-05:00
2008-06-05T21:23:37-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Why can't anybody write a decent stock screener? Google <a href="http://finance.google.com/finance/stockscreener">did</a>, but they left out my favourite exchange, the TSX. The best indicator of whether a stock is going to go up in the medium term is growth in earnings, but it is near impossible to find this information for Canadian stocks. I have tried the one at <a href="http://www.globeinvestor.com">GlobeInvestor.com</a>, but it seems to be written by an imbecile, and its results are quite random.
<p>
Frustrated, I wrote my own tool to pull this information from publicly available sources (Only took about 5 hours). Here, at last, is a text file containing the fundamentals for about 1100 securities on the TSX, as of June, 2008.
<p>
Right now I list only Revenue and EPS, because that is what I use to screen stocks. My plan is to analyse this data in the near future, and find the next Research in Motion.
<center>
<a href="fundamentals.txt">Download the Database</a>
</center>
<h2>Example and format</h2>
<p>
<pre>
svc,q,2008-02-29,Revenue,8290000.00
svc,q,2008-02-29,EPS,-0.05
svc,q,2007-11-30,Revenue,17110000.00
svc,q,2007-11-30,EPS,0.00
svc,q,2007-08-31,Revenue,21180000.00
svc,q,2007-08-31,EPS,0.02
svc,q,2007-05-31,Revenue,20020000.00
svc,q,2007-05-31,EPS,0.08
svc,q,2007-02-28,Revenue,15380000.00
svc,q,2007-02-28,EPS,0.05
svc,a,2007-11-30,Revenue,73680000.00
svc,a,2007-11-30,EPS,0.14
svc,a,2006-11-30,Revenue,31660000.00
svc,a,2006-11-30,EPS,0.00
svc,a,2005-11-30,Revenue,15810000.00
svc,a,2005-11-30,EPS,-0.04
svc,a,2004-11-30,Revenue,3260000.00
svc,a,2004-11-30,EPS,-0.11
svc,a,2003-11-30,Revenue,2040000.00
svc,a,2003-11-30,EPS,-0.07
</pre>
<p>
Each line is a comma separated list of the following fields:
<ul>
<li>Symbol
<li>q (Quarterly) or a (Annual)
<li>Date of the report
<li>Type of data (Revenue or EPS)
<li>Revenue or EPS, both in dollars.
</ul>
<h2>Hottest non-energy/non-mining stocks</h2>
So here are the hottest stocks, filtered with the following criteria:
<ul>
<li>Both Revenue and EPS are increasing for at least two years
<li>P/E ratio is positive and less than 50.
<li>Industry is not resources or energy.
</ul>
<i>All data is from Friday, June 6, 2008</i>
<table border=1>
<tr><th>Stock</th><th>Average Revenue Growth (# years positive)</th><th>Average EPS Growth (# years positive)</th><th>Industry</th><th>Price</th><th>Price/Earnings Ratio</th></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aylo.un">Yellow Pages Income Fund (ylo.un)</a></td>
<td>629% (4)</td><td>298% (4)</td><td>Communications & Media (Publishing & Printing)</td><td>$9.80</td><td>10</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aaui">AltaGas Utility Group Inc. (aui)</a></td>
<td>165% (2)</td><td>127% (2)</td><td>Utilities (Gas Utilities)</td><td>$6.79</td><td>10</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Achl.un">Canadian Helicopters Income Fund (chl.un)</a></td>
<td>140% (2)</td><td>629% (2)</td><td>Transportation and Environmental Services (Transportation)</td><td>$13.00</td><td>6</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aibg.un">IBI Income Fund (ibg.un)</a></td>
<td>94% (3)</td><td>58% (2)</td><td>Business Services (Consulting)</td><td>$22.90</td><td>7</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Adrx">ADF Group, Inc. (drx)</a></td>
<td>82% (2)</td><td>785% (3)</td><td>Industrial Products (Metal Fabricators)</td><td>$4.95</td><td>5</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Arcm">RuggedCom Inc. (rcm)</a></td>
<td>78% (3)</td><td>35% (2)</td><td>Industrial Products (Electrical & Electronic)</td><td>$14.25</td><td>32</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aprk">Parkbridge Lifestyle Communities Inc. (prk)</a></td>
<td>70% (3)</td><td>119% (2)</td><td>Business Services (Computer Software & Processing)</td><td>$5.45</td><td>18</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agvc">Glacier Ventures International Corp. (gvc)</a></td>
<td>62% (5)</td><td>57% (3)</td><td>Communications & Media (Publishing & Printing)</td><td>$4.10</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Asfi">Saxon Financial Inc. (sfi)</a></td>
<td>62% (5)</td><td>31% (5)</td><td>Financial Services (Investment Companies and Funds)</td><td>$14.20</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Amre">Martinrea International Inc. (mre)</a></td>
<td>58% (3)</td><td>68% (3)</td><td>Industrial Products (Metal Fabricators)</td><td>$8.08</td><td>9</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acjt.un">Cargojet Income Fund (cjt.un)</a></td>
<td>53% (2)</td><td>477% (2)</td><td>Transportation and Environmental Services (Transportation)</td><td>$11.15</td><td>24</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Apbl.un">Pollard Banknote Income Fund (pbl.un)</a></td>
<td>52% (2)</td><td>68% (2)</td><td>Industrial Products (Misc. Industrial Products)</td><td>$7.59</td><td>6</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acwa.un">Coast Wholesale Appliances Income Fund (cwa.un)</a></td>
<td>51% (2)</td><td>5% (2)</td><td>Consumer Products (Household Goods)</td><td>$7.72</td><td>8</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Apki.un">Parkland Income Fund (pki.un)</a></td>
<td>49% (5)</td><td>209% (3)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$11.45</td><td>7</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aarf.un">Armtec Infrastructure Income Fund (arf.un)</a></td>
<td>43% (3)</td><td>88% (4)</td><td>Industrial Products (Misc. Industrial Products)</td><td>$24.00</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Awpo">World Point Terminals Inc. (wpo)</a></td>
<td>43% (2)</td><td>43% (2)</td><td>Transportation and Environmental Services (Transportation)</td><td>$14.00</td><td>17</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agna">Gerdau Ameristeel Corporation (gna)</a></td>
<td>43% (5)</td><td>30% (2)</td><td>Industrial Products (Steel)</td><td>$18.20</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Algi">Logibec Groupe Informatique Ltd. (lgi)</a></td>
<td>41% (5)</td><td>40% (5)</td><td>Business Services (Computer Software & Processing)</td><td>$20.00</td><td>27</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agmp.un">GMP Capital Trust (gmp.un)</a></td>
<td>38% (5)</td><td>39% (3)</td><td>Financial Services (Investment Houses)</td><td>$16.31</td><td>8</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aaah">Aastra Technologies Limited (aah)</a></td>
<td>37% (4)</td><td>34% (2)</td><td>Communications & Media (Telecommunications)</td><td>$25.68</td><td>12</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Az.un">Sleep Country Canada Income Fund (z.un)</a></td>
<td>31% (4)</td><td>31% (4)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$20.00</td><td>10</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agcm">Gemcom Software International Inc. (gcm)</a></td>
<td>31% (5)</td><td>37% (3)</td><td>Business Services (Computer Software & Processing)</td><td>$2.99</td><td>23</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Amrd">Melcor Developments Ltd. (mrd)</a></td>
<td>30% (4)</td><td>42% (4)</td><td>Real Estate (Developers)</td><td>$14.73</td><td>7</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aetc">Equitable Group Inc. (etc)</a></td>
<td>29% (5)</td><td>28% (4)</td><td>Financial Services (Finance and Leasing)</td><td>$21.35</td><td>8</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Asj">Stella-Jones Inc. (sj)</a></td>
<td>29% (4)</td><td>53% (4)</td><td>Industrial Products (Misc. Industrial Products)</td><td>$35.26</td><td>17</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acci">Canaccord Capital Inc. (cci)</a></td>
<td>29% (2)</td><td>34% (2)</td><td>Financial Services (Investment Houses)</td><td>$9.97</td><td>3</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Awes">Western Financial Group (wes)</a></td>
<td>29% (5)</td><td>21% (2)</td><td>Financial Services (Insurance)</td><td>$4.17</td><td>20</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Asz">Sceptre Investment Counsel Limited (sz)</a></td>
<td>29% (2)</td><td>41% (2)</td><td>Financial Services (Investment Companies and Funds)</td><td>$9.21</td><td>17</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Arc">RDM Corporation (rc)</a></td>
<td>28% (4)</td><td>203% (3)</td><td>Business Services (Computer Software & Processing)</td><td>$1.60</td><td>9</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Asif.un">Energy Savings Income Fund (sif.un)</a></td>
<td>26% (5)</td><td>73% (4)</td><td>Utilities (Gas Utilities)</td><td>$14.79</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aalc">Algoma Central Corporation (alc)</a></td>
<td>22% (4)</td><td>50% (4)</td><td>Transportation and Environmental Services (Transportation)</td><td>$138.00</td><td>9</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Amlx">Marsulex Inc. (mlx)</a></td>
<td>22% (4)</td><td>302% (2)</td><td>Transportation and Environmental Services (Environmental)</td><td>$13.50</td><td>16</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agil">Gildan Activewear Inc. (gil)</a></td>
<td>21% (3)</td><td>29% (3)</td><td>Consumer Products (Household Goods)</td><td>$29.40</td><td>21</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Ax">TSX Group, Inc. (x)</a></td>
<td>21% (4)</td><td>22% (5)</td><td>Other Services (Other Services)</td><td>$43.82</td><td>20</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Apbi.un">Premium Brands Income Fund (pbi.un)</a></td>
<td>20% (4)</td><td>79% (2)</td><td>Consumer Products (Food Processing)</td><td>$13.10</td><td>8</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agdi.un">General Donlee Income Fund (gdi.un)</a></td>
<td>20% (3)</td><td>78% (2)</td><td>Industrial Products (Metal Fabricators)</td><td>$8.75</td><td>8</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acmg">Computer Modelling Group Ltd. (cmg)</a></td>
<td>20% (5)</td><td>24% (5)</td><td>Business Services (Computer Software & Processing)</td><td>$18.81</td><td>21</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acuq">The Churchill Corporation (cuq)</a></td>
<td>19% (5)</td><td>123% (2)</td><td>Real Estate (Contractors)</td><td>$21.38</td><td>16</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Arba">Ritchie Bros. Auctioneers (rba)</a></td>
<td>18% (5)</td><td>29% (3)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$26.45</td><td>37</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Astn">Stantec Inc. (stn)</a></td>
<td>18% (5)</td><td>22% (5)</td><td>Business Services (Consulting)</td><td>$29.14</td><td>18</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acca">Cogeco Cable Inc. (cca)</a></td>
<td>17% (5)</td><td>72% (2)</td><td>Communications & Media (Cable)</td><td>$39.95</td><td>14</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Afc.un">Firm Capital Mortgage Investment Trust (fc.un)</a></td>
<td>17% (5)</td><td>2% (2)</td><td>Financial Services (Finance and Leasing)</td><td>$10.58</td><td>10</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Asc">Shoppers Drug Mart Corporation (sc)</a></td>
<td>17% (5)</td><td>18% (5)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$56.17</td><td>23</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Avln">Velan Inc. (vln)</a></td>
<td>16% (3)</td><td>707% (2)</td><td>Industrial Products (Misc. Industrial Products)</td><td>$12.21</td><td>22</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Agcg">Guardian Capital Group Ltd. (gcg)</a></td>
<td>16% (5)</td><td>42% (4)</td><td>Financial Services (Investment Companies and Funds)</td><td>$8.46</td><td>12</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Amda">Macdonald Dettwiler & Associates Ltd (mda)</a></td>
<td>16% (5)</td><td>17% (5)</td><td>Business Services (Computer Software & Processing)</td><td>$42.23</td><td>18</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Awfi">WFI Industries Ltd. (wfi)</a></td>
<td>15% (5)</td><td>21% (5)</td><td>Consumer Products (Misc. Consumer Products)</td><td>$25.95</td><td>30</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aeh">easyhome Ltd. (eh)</a></td>
<td>13% (5)</td><td>42% (2)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$16.49</td><td>14</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Adsg">Descartes Systems Group Inc. (dsg)</a></td>
<td>13% (2)</td><td>203% (2)</td><td>Business Services (Computer Software & Processing)</td><td>$3.85</td><td>9</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Atih">Toromont Industries Ltd. (tih)</a></td>
<td>12% (5)</td><td>24% (5)</td><td>Merchandising and Lodging (Wholesale Distributors)</td><td>$30.61</td><td>16</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aftt">Finning International Inc. (ftt)</a></td>
<td>12% (5)</td><td>29% (3)</td><td>Merchandising and Lodging (Wholesale Distributors)</td><td>$28.00</td><td>17</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Alnr">Linamar Corporation (lnr)</a></td>
<td>11% (5)</td><td>27% (4)</td><td>Industrial Products (Transportation Equip. & Compnts)</td><td>$16.30</td><td>10</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aigm">IGM Financial Inc. (igm)</a></td>
<td>11% (4)</td><td>12% (5)</td><td>Financial Services (Investment Companies and Funds)</td><td>$45.84</td><td>13</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acae">CAE, Inc. (cae)</a></td>
<td>11% (4)</td><td>54% (2)</td><td>Industrial Products (Transportation Equip. & Compnts)</td><td>$13.34</td><td>20</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Arch">Richelieu Hardware Ltd. (rch)</a></td>
<td>10% (5)</td><td>11% (5)</td><td>Merchandising and Lodging (Wholesale Distributors)</td><td>$21.25</td><td>14</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Anwf.un">North West Company Fund (nwf.un)</a></td>
<td>10% (3)</td><td>13% (5)</td><td>Merchandising and Lodging (Department Stores)</td><td>$18.27</td><td>14</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Atoc">Thomson Reuters Corporation (toc)</a></td>
<td>8% (2)</td><td>30% (2)</td><td> ()</td><td>$37.48</td><td>22</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Akos">Cossette Communication Group Inc. (kos)</a></td>
<td>8% (5)</td><td>17% (2)</td><td>Business Services (Advertising Agencies)</td><td>$6.15</td><td>7</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Afgl">The Forzani Group Ltd. (fgl)</a></td>
<td>7% (5)</td><td>90% (2)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$17.02</td><td>12</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Alnf">Leon's Furniture Ltd. (lnf)</a></td>
<td>7% (5)</td><td>13% (4)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$12.00</td><td>14</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aclc.un">CML Healthcare Income Fund (clc.un)</a></td>
<td>7% (2)</td><td>18% (4)</td><td>Other Services (Medical Services)</td><td>$15.42</td><td>13</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Acp">Canadian Pacific Railway Limited (cp)</a></td>
<td>6% (4)</td><td>25% (4)</td><td>Transportation and Environmental Services (Transportation)</td><td>$69.03</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Aidg">Indigo Books & Music Inc. (idg)</a></td>
<td>5% (3)</td><td>116% (5)</td><td>Merchandising and Lodging (Specialty Stores)</td><td>$14.49</td><td>6</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3At">TELUS Corporation (t)</a></td>
<td>5% (5)</td><td>45% (4)</td><td>Utilities (Telephone Utilities)</td><td>$46.07</td><td>11</td></tr>
<tr><td><a href="http://finance.google.com/finance?q=tse%3Ahlf">High Liner Foods Incorporated (hlf)</a></td>
<td>4% (2)</td><td>41% (2)</td><td>Consumer Products (Food Processing)</td><td>$8.85</td><td>20</td></tr>
</table>
<ul><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=75'>Pitching to VCs (comic)</a><li><a href='?id=68'>When programmers design web sites (comic)</a><li><a href='?id=13'>Draw waveforms and hear them</a><li><a href='?id=74'>Blame the extensions (comic)</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=19'>Installing the Latest Debian on an Ancient Laptop</a></ul>
Why are all my lines fuzzy in cairo?
tag:smhanovtechblog,2007:id28
2008-04-04T13:50:07-05:00
2008-04-04T13:50:07-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<a href="http://www.cairographics.org">Cairo</a> is the hot new cross platform graphics library. It is becoming very
popular, because it solves two outstanding problems in a portable way:
<p>
<ol>
<li>Path based drawing
<li>Antialiasing
</ol>
<p>
Both of these problems are astoundingly hard. You would have to read a whole
graphics textbook in order to implement basic drawing, and antialising. Before
cairo, your choices were Win32 GDI based drawing, or whatever GTK uses. In
addition, cairo is supported in Python.
<p>
The problem is that cairo has something that's not obvious for some people. A
lot of users might write a program to draw a line and get this:
<pre>
#!/usr/bin/python
import cairo
def drawLine( ctx, x1, y1, x2, y2 ):
ctx.move_to( x1, y1 )
ctx.line_to( x2, y2 )
ctx.set_line_width( 1.0 )
ctx.stroke()
surface = cairo.ImageSurface(cairo.FORMAT_RGB24, 32, 32)
ctx = cairo.Context( surface )
ctx.set_source_rgb( 1.0, 1.0, 1.0 )
drawLine( ctx, 2, 16, 30, 16 )
drawLine( ctx, 16, 2, 16, 30 )
surface.write_to_png( "out.png" )
</pre>
<center>
<p><img src="nosnap.png">
<br>(Magnified 4 times)
</center>
<p>
The lines are all fuzzy! Even <a href="http://www.inkscape.org">Inkscape</a>, an otherwise well-polished
graphics program, has this naive implementation, and it
<a href="http://www.google.com/search?q=inkscape+fuzzy+lines&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-GB:official&client=firefox-a">frustrates users to no end</a>, because all of their lines are fuzzy.
<p>
The reason is because cairo's coordinates are centered on the pixel boundaries,
instead of in the middle of a pixel. So when you draw the line at
coordinate (2, 16), it is really beginning half way in between pixel 2 and
3, and pixels 16 and 17.
<p>
The immediate solution is to add 0.5 to all your coordinates. If you are doing
more complicated drawing, with varying pen widths and scales, you will have to
modify it somewhat. Also, this system breaks down as soon as you scale the
image smaller, as adding 0.5 starts to make huge errors in where things are.
But for an image that is not scaled smaller, please snap the coordinates to
avoid the fuzzy lines, and the eyesight of your users!
<pre>
#!/usr/bin/python
import cairo
def snapCoords( ctx, x, y ):
(xd, yd) = ctx.user_to_device(x, y)
return ( round(x) + 0.5, round(y) + 0.5 )
def drawLine( ctx, x1, y1, x2, y2 ):
point1 = snapCoords( ctx, x1, y1 )
point2 = snapCoords( ctx, x2, y2 )
ctx.move_to( point1[0], point1[1] )
ctx.line_to( point2[0], point2[1] )
ctx.set_line_width( 1.0 )
ctx.stroke()
surface = cairo.ImageSurface(cairo.FORMAT_RGB24, 32, 32)
ctx = cairo.Context( surface )
ctx.set_source_rgb( 1.0, 1.0, 1.0 )
drawLine( ctx, 2, 16, 30, 16 )
drawLine( ctx, 16, 2, 16, 30 )
surface.write_to_png( "out.png" )
</pre>
<center>
<p><img src="snap.png">
</center>
<ul><li><a href='?id=96'>Bending over: How to sell your software to large companies</a><li><a href='?id=123'>Zero load time file formats</a><li><a href='?id=71'>Using the Acer Aspire One as a web server</a><li><a href='?id=61'>Experiment: Deleting a post from the Internet</a><li><a href='?id=14'>Experiments in making money online</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=59'>When a reporter mangles your elevator pitch</a></ul>
A simple command line calculator
tag:smhanovtechblog,2007:id26
2008-03-23T15:28:15-05:00
2008-03-23T15:28:15-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
How many times have you needed to calculate something, for example the value of 0x398A3BB, so you pop up windows calculator to convert it? I did lots of times. The problem is I hate to use the mouse. It takes precious deciseconds away from software development time to remove my hands from the keyboard and use the mouse.
That's why I created calc.exe. Its a simple command line calculator (and its also an example of recursive decent parsing).
<ul>
<li>Download <a href="calc.exe">calc.exe</a>
<li>Download <a href="calc.c">calc.c</a>
</ul>
<h3>Examples</h3>
<pre>
C:>calc 5+5*5
30.000000
c:>calc 0x30
48.000000
c:>calc (123456 % 51)/12
3.000000
</pre>
<ul><li><a href='?id=79'>How QBASIC almost got me killed</a><li><a href='?id=140'>A little VIM hacking</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=10'>Detecting C++ memory leaks</a></ul>
Tool for Creating UML Sequence Diagrams
tag:smhanovtechblog,2007:id25
2008-03-03T19:51:21-05:00
2008-03-03T19:51:21-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>If you have to draw something called "UML Sequence Diagrams" for work or school, you already know that it can take hours to get a diagram to look right. Here's a web site that will save you some time:
</p>
<p align=center><a href="http://www.websequencediagrams.com">www.websequencediagams.com</a></p>
<p>
You can just write the diagram out in text, click "Draw", and the web site will spit out an image. Then you can tell your boss that you slaved for hours in MS Visio perfecting every line...
</p>
<h2>Example</h2>
<p>Here's an example of what you'd write. Notice that the syntax is very natural.</p>
<pre>
Alice->Bob: Authentication Request
alt successful case
Bob->Alice: Authentication Accepted
else some kind of failure
Bob->Alice: Authentication Failure
note right of Bob: Bob clears key cache
end
</pre>
<p>
... And here's the resulting image:
</p>
<p align=center>
<a href="http://www.websequencediagrams.com"><img border=0 src=msc.png /></a>
</p>
<h2>Hey, wait a minute...</h2>
<p>
Astute readers will notice that I am the author of websequencediagrams.com. Yesterday I added a note pleading with people to blog about it or at least link to it, and I figured that I should practice what I preach. So this article is nothing but a shameless plug for my other web site.
</p>
<p>
The fact is, the page doesn't have much text on it, so its hard for people to come across it by searching alone. I added it to Wikipedia and so far that's where most people find it. The more links I have, the more Googler's will find it.
</p>
<h2>Alternatives</h2>
<table border=1>
<tr><td><a href="http://sdedit.sourceforge.net">Quick Sequence Diagram Editor</a></td><td>Java program with confusing syntax.</td></tr>
<tr><td><a href="http://www.mcternan.me.uk/mscgen/">mscgen</a></td><td>Unix command line program. I was inspired by its syntax, but found it overly verbose.</td></tr>
<tr><td><a href="http://www.sequencediagrameditor.com">Sequence Diagram Editor</a></td><td>Nice editor, if you <a href="http://www.codinghorror.com/blog/archives/000532.html">love filling in text boxes</a>. $99</td></tr>
<tr><td><a href="http://www.tracemodeler.com">Tracemodeler</a></td><td>A worthy competitor, though we have different views on the use of text. Its author, Yanic, and I try to see is first to mention our tool on web forums.</td></tr>
</table><ul><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=33'>Simulating freehand drawing with Cairo</a><li><a href='?id=39'>Stock Picking using Python</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=19'>Installing the Latest Debian on an Ancient Laptop</a><li><a href='?id=11'>Cell Phones on Airplanes</a><li><a href='?id=123'>Zero load time file formats</a></ul>
Exploring sound with Wavelets
tag:smhanovtechblog,2007:id22
2007-12-27T22:39:00-05:00
2007-12-27T22:39:00-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Here's a program to create scalograms of sound files. Pictured below is the "windows xp startup sound". See how the individual frequencies have been isolated visually.
<p>
<font color=red>I have created a separate web page for this project... please <a href="../wavelet">go there</a>.</font>
<p>
<center>
<img src="wavelet1.png">
<br>
<a href="WaveletSoundExplorer.exe">Download Installer</a>
</center>
<p>
I've been curious about wavelets since I did a <a href="http://gandolf.homelinux.org/~smhanov/cs698_wavelet_project.pdf">course project</a> on them.
<p>
The wavelet transform is similar to Fourier analysis, in that it figures out which frequencies exist in a given signal. The difference is that it adds another dimension to the data. From a 1-D waveform, you will get a 2-D picture. Each row is a frequency, and the columns are times. So you get a picture of how the frequency changes with time.
<p>
The DWT does speedup the wavelet transform greatly, and mathematically, no information is lost. However it is not a very good way to look at the data visually. From 1024 samples, you only get 10 frequency bands. There's no way to, for instance, distinguish individual notes in song. Here's an example of what you'd get from the DWT. Compare it to the result from the first image, and you see how much information is hidden!
<p>
<center>
<img src=wavelet2.png>
<br>
<i><b>Figure 1: Ten frequency bands from 512 samples. Where did all the information go???</b></i>
</center>
</p>
<p>
Because of the DWT, very few people give the CWT (continuous wavelet transform) a second glance. The library is filled with books on wavelets that spend two pages on the CWT, and then talk for the rest of the book about applying the DWT. As a result, people think the DWT is all there is.
<p>
Another technique, called the the wavelet packet transform, gives you a little more detail. But at the end of it, if you have 1024 sound samples, you will have 1024 transformed points. The more times you perform the algorithm, the more detail you loose in time (and the image looks like a pixellated mess).
<p>
<h2>Continuous Wavelet Transform</h2>
My program applies the continuous wavelet transform to a wave file that you load in, and lets you zoom into see the individual frequencies that make up a sound. Give it a try!
<p>
One problem with it is that it generates a <i>lot</i> of data. Analyzing that sound took 170 MB of memory, and a couple of minutes on my computer. If you tried it on a 5 minute MP3 file, that's 5 times 60 seconds times 44100 samples per second * 44100/60 frequency bands = 9.7 billion data points, or about 38 GB of floating point data, <i>if you don't use stereo!</i>.
<p>
But it does produce some pretty pictures for short files. Here's a closeup view of the famous <i>tada.wav</i>:
<p>
<center>
<img src=wavelet3.png>
<br><b><i>Closeup on a small section of tada.wav</b></i>
</center>
<p>
<center>
<a href="wavelet4-big.jpg"><img src=wavelet4.jpg></a>
<br><b><i>The majestic noise of the <a href="recycle.wav">"c:windowsmediarecycle.wav"</a> paper crumpling makes a great <a href="wavelet4-big.jpg">wallpaper.</a></b></i>
</center>
<p>
<h2>How it works</h2>
<ol>
<li>The program loads in a wave file using libsnd. If it is stereo or multichannel, the other channels are ignored and only the first channel is used.
<li>
When you see "Rendering... 1%" on the screen, the program is busy calculating the wavelet transform. It first calculates some frequency scales, from 2 samples to sampleRate divided by 60 samples long, and goes through them logarithmically (eg. 2, 4, 8, 16 samples long).
<li>For each scaling factor, it creates a "real" and "complex" wavelet whose period is that many samples long. The wavelet we use is the cosine function multiplied by a gaussian (For the real part) and the imaginary part is the same thing, but with a sine function. This is known as the <i>Morlet</i> wavelet, and it is exceptionally good for sound analysis due to the sine and cosine basis.
<li>Once it has created the wavelets, it <i>convolves</i> the wavelet with the signal. Convolution is kind of like smearing one signal with another. To speed up the algorithm, I perform convolution by multiplying the fourier transforms of the signal and the wavelet. After the convolution, we end up with the strength of the wavelet in the signal at each point in time.
<li>The process is repeated for each scale level.
<li>Now we have real and complex data samples. The magnitude of the data samples are converted into a huge device independent bitmap in memory, so it can be displayed to the screen. I hope you have lots of RAM.
</ol>
<h2>Future Work</h2>
<p>
If I have time in the new year, I'm going to add some fun stuff:
<ul>
<li>Drag and drop pitch shifting - This is not as easy as moving pixels on the image... first you have to do something called "phase unwrapping". My first cut at a phase unwrapping algorithm didn't work, so I'm trying to translate some fortran code from a 1981 paper I found. Does anybody have some C code for this???
<li>Boost/Reduce -- Draw a square with the mouse and boost or reduce the strength of that region. This could be great for manual noise elimination and sound retouching, or restoring that old copy of <a href="http://www-ccrma.stanford.edu/~brg/brahms2.html">Brahms playing the piano</a>.
</ul>
<p><ul><li><a href='?id=19'>Installing the Latest Debian on an Ancient Laptop</a><li><a href='?id=3'>Cell Phone Secrets</a><li><a href='?id=72'>Microsoft's generosity knows no end for a year (comic)</a><li><a href='?id=140'>A little VIM hacking</a><li><a href='?id=122'>Finding the top K items in a list efficiently</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a></ul>
UMA and free long distance
tag:smhanovtechblog,2007:id4
2007-07-29T21:47:40-05:00
2007-07-29T21:47:40-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<h2>UMA and free long distance</h2>
Last time, I talked about the UMA technology used in some newer cell phones. Some of you might be thinking, these new cell phones work over the Internet. What's to stop me from travelling to another continent, and then making free long distance calls to local numbers back home?
<p>
Technically, nothing's stopping you. But in theory, carrier policy might get in the way. UMA technology makes it possible for the carrier's to decide what should happen.
<p>
You see, when an UMA phone starts up, it is required to scan the cellular network first, before it does anything else. Then, it will try to get UMA service. This will happen even if you've configured the device to only use WiFi.
<p>
As part of the registration process towards the UNC (this is the carrier's server ,the one that acts like a cell tower, only over the Internet), the mobile will report the identity of the surrounding network. Part of this report is the MCC, or mobile country code of the network. Using this information, the carrier can easily figure out what country you are in. If they have a database of the exact cells in the area, they could figure out where you are to within a 30 km radius too.
<p>
If that doesn't work, devices equipped with GPS will generally report your coordinates as well. This is all happening as soon as you power up the phone.
<p>
So if there is cellular coverage, your home carrier will be able to figure out where in the world you are. They can then comply with existing roaming agreements with the carrier in the country you are in, or they could just be evil and charge you more, keeping all the money for themselves, since they don't even need the other carrier.
<p>
I suppose you could make sure the phone is out of cellular coverage somehow. Maybe you could wrap your hotel room in aluminum foil. But then the mobile will report that too. It will say that it's out of coverage, and your carrier will know that you are not at home from your IP address, and they may refuse you service.
<h2>Reality</h2>
In reality, carrier's aren't all that concerned about this yet. It seems like, for the time being, you can get free long distance using this method. It makes sense for the carriers to extend their UMA service abroad, because otherwise you would simply be benefiting a foreign network.
<p>
Here's an ideal scheme: Suppose you have a lot of family in the US, but you live in the UK. So you go to the US, sign up for UMA service from T-Mobile, and they give you a handset and an access point. Say thanks, and then go back home. Plug that AP into your existing broadband internet, and you can now make calls to the US at local rates. This assumes that your Internet bandwidth is cheap, however.
<p>
So right now, you can beat the system. But when travelling, I'd take an extra roll of aluminum foil, just in case...
<ul><li><a href='?id=117'>Why don't web browsers do this?</a><li><a href='?id=146'>O(n) Delta Compression With a Suffix Array</a><li><a href='?id=10'>Detecting C++ memory leaks</a><li><a href='?id=88'>How IE <canvas> tag emulation works</a><li><a href='?id=13'>Draw waveforms and hear them</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=139'>The strange man reading a novel in the meeting room</a></ul>
UMA's dirty secrets
tag:smhanovtechblog,2007:id2
2007-07-24T19:57:10-05:00
2007-07-24T19:57:10-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
For more UMA answers, see my <a href="?id=38">more recent article.</a>
<h2>What's UMA?</h2>
Recently, many carriers have started offering UMA, or WiFi phones. These are cell phones with WiFi capabilites. Don't be fooled -- you won't be able to get free calls and run skype on them. The UMA technology is meant to extend the carrier's cellular network into your home using your broadband internet connection.
<h2>How does UMA work?</h2>
An UMA phone operates just like a regular cell phone. It can talk to cellular base stations. But it is dual mode, and it also has a WiFi radio on board. When it finds a WiFi access point, it will attempt to connect to your carrier's servers over the Internet. If the connection is successful, it will "Rove in" and begin sending everything over the Internet.
<p>
The carrier's server is called an UMA Network Controller, (UNC). From the perspective of the phone, the UNC looks just like a regular cell tower, and it talks to it in the same way as a cellular base station, except that everything is wrapped up and forwarded over the Internet. Communicating in this way has some important differences from the way your laptop accesses the Internet
<h2>Hands in your pocket</h2>
When you browse the web from your laptop, the data flows from your laptop to the web site you are visiting, with nobody in between. It is different when you are browsing with a cell phone, however. With a cell phone, you get assigned an IP address in your carrier's core network. The IP address is how your handset is identified on the network. For example, when you browse a web site, the IP address lets the web server know who to send the web page back to.
<p>
When you are using a cell phone, the idea is that your IP address will stay the same no matter which cellular tower you are at. So, if you are loading a web page and driving down the highway at 120 km/h, you might switch from cell tower to cell tower, but your IP address will remain the same, and your web page will still load. The carriers accomplish this by giving you an IP address in their core network. When you ask for a web page, your request is forwarded through your cell phone company's servers. Your cell phone company actually downloads the web page for you, and then sends it to your phone.
<p>
The same thing happens with UMA. You might rove-in to your WiFi connection, but your IP address will remain the same. Your device is still directly connected to the carrier's core network, and the web page loads through your carrier's servers.
<p>
So if you wanted to load Skype on your phone to try and make free phone calls, forget it. It would cost you more in data usage charges than you'd save. Also, it's probably technically impossible, due to the amount of extra work your phone has to do.
<h2>UMA efficiency</h2>
Another important difference between browsing using your laptop and the cell phone is efficiency. Because your laptop is directly connected to the internet, it has a much greater advantage in terms of speed. Your tiny cell phone, however, is burdened with extra protocols that make loading web pages a very costly operation.
<p>
When your laptop is transmitting data, the data is broken up into small chunks, called IP packets. These IP packets can then be transmitted directly over the Internet.
<p>
Over UMA, however, the situation is very different. IP packets over UMA are transmitted using the same techniques as if they were going over a cell tower. That means that after your web browser forms an IP packet, it has to be transformed into a form that is recognizable by carrier's servers. The packet will first be broken up into smaller chunks, called frames. Each frame will then have extra information added to it, called headers, that is needed to be understood by your carrier's network. The extra information is not so much, but what is really costly is the security.
<h2>Security</h2>
Your UMA phone has a direct pipe into your carrier's core network. This requires a lot of security, because your carrier doesn't want just anyone to have this kind of access. So your phone communicates using a special protocol known as IPSec.
<p>
You many be familiar with IPSec already. It's used by a lot of companies that issue their employees laptops. If you have to work from home, you might have some kind of security key, and to log in, you'll start up an application called a VPN Client, and then boom, it's as if you were sitting in your cubical at work, except that you're at home in your underwear.
<p>
UMA phones use the same technology. To connect, they form an IPSec tunnel into the carrier's network. Instead of a password, the phone checks that your SIM card is valid and up to date before letting you on.
<p>
IPSec provides great security. The packets are encrypted, and it's pretty much impossible to figure out what they mean, what web pages you're browsing, or what you are saying in your phone calls. However, it has a huge cost in terms of overhead. Each packet has to have extra headers added, and then it's encrypted. This encryption can expand the packets by as much as 30%. This means that your web pages will take 30% longer to load vs. using your laptop, even under the best of conditions.
<p>
I filed <a href="http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=/netahtml/PTO/srchnum.html&r=1&f=G&l=50&s1=20080310448.PGNR.">this patent</a> to try to mitigate the problem.
<h2>UMA Advantages</h2>
If UMA is so inefficient, why use it at all? I am a strong supporter of UMA, despite its flaws. It's great for the consumer, because it gives you better coverage when you are at home. Also, it affects the pricing of your phone. Many carriers have special discounts, or even unlimited calling when you are on UMA.
<p>
You see, it's all part of the strategy to get you to use your cell phone at home. Carriers would much prefer you to use your cell phone all the time, so they can squeeze more revenue out of you. This would be beneficial for the consumer too, because rather than paying for a cell phone plus a landline, you ditch your landline and just pay a little more for your cell phone.
<p>
But if everybody did this, without UMA, it wouldn't work, because cell towers can only support a few dozen calls at the same time. UMA is a relatively cheap add-on to a carrier's infrastructure, so it makes sense to add it. Adding a new base station to cover dead spots in a neighbourhood costs a quarter million dollars. A WiFi access point, at wholesale rates, costs maybe $30.
<h2>Handsets</h2>
Early generation handsets, like the Samsung and Nokia, have a few problems. I have read reviews on the Internet and apparently they were horrible and people are asking for their money back.
<p>
There are a few reasons behind this. UMA specifications were only finalized as recently as 2005, and unlike the mature GSM specifications, they leave much open to interpretation. They don't address things like when your phone is supposed to rove in and rove out. There isn't an easy way to figure out if your Wifi connection is stronger than your cell tower. Your typical tower transmits at several watts of power because it has to reach tiny cell phones up to 30 km away, but your typical access point transmits at only a tiny fraction of that power. Your cell phone can't just choose the stronger one. There just isn't an easy way to decide which one to use. If your phone chooses to use the Wifi access point, but it's too weak to be used, then quality will suffer.
<p>
Quality of service is also a problem. If your laptop is downloading movies using bittorent, and you're trying to make a phone call, it just isn't going to work. Theoretically, a technology called Quality of Service is supposed to fix problems like this, but the technology just isn't there today in 2007. Most access point deployed don't support it at all, or they say they support it but it is completely inadequate. So if you are planning on making internet calls while watching videos online, plan on getting an up-to-date AP with decent QoS.
<p>
Finally, most people's access points use the default settings set at the factory. That means that they will be using the same WiFi channel, and two of them placed close together (as in an apartment building) will cause interference. Other appliances like Microwaves will also cause a degradation of the signal.
<h2>But wait...</h2>
Some cell phones can <a href="http://umatoday.blogspot.com/2008/04/internet-offload-and-uma.html">send data natively over WiFi</a>, without going over carrier's servers. The carrier has no way to track this data, so you can send as much as you want. But in practice, you really have no way of knowing if it is using Wifi directly, or the UMA connection.
<p><ul><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=101'>"Your program is stupid. It doesn't work," my wife told me</a><li><a href='?id=50'>Why Perforce is more scalable than Git</a><li><a href='?id=116'>Fun with Colour Difference</a><li><a href='?id=70'>Finding great ideas for your startup</a><li><a href='?id=140'>A little VIM hacking</a><li><a href='?id=57'>Coding tips they don't teach you in school</a></ul>
Installing the Latest Debian on an Ancient Laptop
tag:smhanovtechblog,2007:id19
2007-05-19T16:09:36-05:00
2007-05-19T16:09:36-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<img align=right src=old_laptop.jpg>
<em>The challenge:</em> Install Linux on a really old laptop. The catch: It has only <em>32 MB of RAM, no network ports, no CD-ROM</em>, and the floppy drive makes creaking noises. Is it possible? Yes. Is it easy? No. Is is useful? Maybe...
<h2>Motivation</h2>
Why? Like mountain climbers say: because it's there. As an environmental nut, I don't like to throw away things that still work. But I have a PCMCIA network card and I would rather not have to hunt down and install 10 year old drivers to get it to work with Windows 95. The latest Linux definitely supports more hardware out of the box than Windows 95.
<h2>The Laptop</h2>
I don't know very much about this laptop. I recall that the system information utility in Windows 95 is "MSD" or "MSDIAG", but either didn't exist in this installation, or my memory is faulty and I was typing the wrong command.
<p>
The only thing I do know is that it has 32 MB of RAM, integrated stereo sound and modem, no network card, and no CDROM drive. When it boots, there is no obvious way to enter the BIOS utility. I tried DEL, F8, etc but I figure it doesn't have one.
<p>
<h2>Ubuntu's Tragic Failure</h2>
I tried really hard to install Ubuntu, which I am familiar with on my other machines. The problem is that Ubuntu has removed the ability to install it from floppies. You have to a) use the CDROM or b) do a network boot.
<p>
I wasted hours trying to do the network boot, which I have done before for the machine running this web server (The DELL DVD drive died long ago). I followed the instructions to put the network boot CDROM onto another Linux server, and installed TFTP and a DHCP server. But the ancient Compaq laptop presented a problem.
<p>
Normally you could go to <a href="http://rom-o-matic.net/">http://rom-o-matic.net/</a> and make a boot floppy, which will boot up, detect the network interface, and then do a network boot (which would start the Ubuntu installer). However, the only network card I had was a PCMCIA 3com card that is supposed to be supported by rom-o-matic. But no matter what I did, etherboot would not detect it and would just sit there dumbly.
<p>
It may have been possible to use the PLIP (IP over parallel port networking) option and boot from over the network using a special cable. But I don't have the cable, and such an install is so uncommon, it would be a miracle if it worked at all.
<p>
<h2>Debian saves the day</h2>
After some research I found that Debian still supports the floppy install option. All you need are 4 floppies for <i>boot.img, root.img</i>, and two additional disks of network drivers. Of course, I only own one floppy disk so I had to keep re-imaging it during the install.
<p>
The minimum memory required for installation is 32 MB, so we are in luck. The problem is that the installer enters a "low memory mode" and doesn't load any kernel modules on its own. Instead, it pops up a list and you have to guess what drivers you are going to need. If you are wrong, you can always click "go back" to back up. I went through the four floppy disks that it asked for, and selected anything that looked like IDE (for the hard drive), 3COM, and PCMCIA (for the network card). Actually, at first I didn't select the IDE components. As a result, the installer offered to partition my floppy disk. I went back to add in the hard disk drivers.
<p>
Finally, the installer was working. It connected to the network and downloaded and installed the minimum debian distribution. The only changes I made were to the partitioning. Initially, it offered a 90 MB swap partition. That seems small, so I increased it to 400 MB, leaving 1.3 GB of disk space left over for the install.
<p>
<h3>Hiccups</h3>
The installer seemed to freeze at one stage, while "preparing installation report." I rebooted the machine and it worked the second time.
<p>
When the laptop boots up, I get lots of kernel messages about failed I/O operations. However, once it starts everything is okay. I did a surface check using <code>e2fsck -c</code> but the errors persist.
<h2>Running Programs</h2>
Once everything was set up, I installed gvim, xdm, Xorg, and icewm, (which is a great window manager that doesn't take up too much space). When I started X for the first time, the screen was red and didn't look right. It turns out that I had to limit it to 16-bit and reduce the resolution to 800x600, the native resolution of the screen. Then everything worked.
<h3>Battle of the Browsers</h3>
Once the graphics were set up, I used good old lynx, the text based browser, to download Firefox. But once I started my favourite browser, I waited, and waited, and waited...
<p>
It turns out that Firefox is a memory pig. It took 15 minutes to start, and it takes up over 100 MB of memory to show a blank window. Ugh!
<p>
There isn't a lot of choice of browsers out there. Galeon is part of Gnome, and I definitely didn't want any bloated Gnome packages on my lean but slow machine. Instead, I downloaded the latest version of Opera.
<p>
Opera starts in only 10 seconds or so. It's usable, if you don't mind waiting a few seconds between clicks. So Opera wins the browser wars for low-resource machines.
<h3>vncviewer</h3>
Since the laptop is so slow to use, I mostly use it to connect to other machines using xtightvncviewer. For this purpose, it works very well.
<h2>Conclusion</h2>
<ul>
<li>For old machines with no CD-ROM drive, you are better off installing Debian than Ubuntu.
<li>Because of Debian's low-memory mode installer, you'd better be a computer expert to pick the right drivers during installation.
<li>The best package manager is "aptitude", but only when run from the command line, because even the text-based GUI is too slow. It keeps stopping everytime you do something to do housekeeping.
<li>The best browser for low-resource machines is Opera.
<li>Aside from a minor hiccup with X.org's graphics detection, all hardware works flawlessly.
</ul><ul><li><a href='?id=39'>Stock Picking using Python</a><li><a href='?id=26'>A simple command line calculator</a><li><a href='?id=8'>A Rhyming Engine</a><li><a href='?id=145'>A Quick Measure of Sortedness</a><li><a href='?id=41'>See sound without drugs</a><li><a href='?id=123'>Zero load time file formats</a><li><a href='?id=85'>barcamp (comic)</a></ul>
Experiments in making money online
tag:smhanovtechblog,2007:id14
2007-04-18T19:04:30-05:00
2007-04-18T19:04:30-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Is it possible to make money on the internet, if you try really hard? I want to find out.
I have always been interested in getting money for doing nothing. In an ideal business, you would do some initial work to get a system set up, and then wait for cash to come in. Here are some results, including revenue earned, from:
<ul>
<li>Shareware
<li>Adware
<li>Adsense
<li>Donations
</ul>
<h2>Shareware</h2>
My experiements in shareware have been a dismal failure. I created <a href="http://www.hanovsolutions.com/?prod=hj">Hotkey Jumpstart</a>, a utility program that lets you start any program or music file by typing a few letters of its name, in 2004. After posting it on dozens of sites, I do have a hard time getting downloads, and hardly anybody registers. In two years, I made a total of $25.12 US.
<p>
The association of shareware professionals, which I joined for a year, has a few examples of success. Winzip apparently made lots of money, and its creator could earn a living off it it. A few others worked well too.
<p>
Apparently, software utilities are a bad category for shareware, and they don't do well at all. I think games would work better, because my wife has bought several flash games online. But to create games, you have to use Adobe Flash creator, and it costs $699 to download. That's pretty hard to justify.
<p>
It is also possible that Hotkey Jumpstart doesn't even work for most computers. It has when I tested it, but if it weren't working at all, I doubt that anybody would bother to email. Having a shareware product makes beta testing difficult.
<p>
So shareware hasn't worked for me. I think it would work better in these areas:
<ul>
<li><em>BlackBerry applications</em> Web sites like <a href="http://www.handango.com">Handango</a> have gotten people used to having to pay to download something, without trying it out first.
<li><em>Apple shareware</em> Because there is so little software for Apple, people are still willing to pay for good applications.
<li><em>Games</em> - Games are easy to monetize. You just have to make extra levels, or put in a time limit.
</ul>
<h2>Adware</h2>
In the year 1999, the term "spyware" didn't exist. We had trojans and viruses and if I heard the word spyware I would assume it is some kind of trojan that steals passwords (this is not what spyware does today). After I released <a href="http://www.hanovsolutions.com/?prod=alarm">Banshee Screamer Alarm</a>, it was wildly successful because it was a free, and I had consciously made it better than other alarm clocks at the time. It was getting thousands of downloads a month.
<p>
I got an email from the marketing director of a company called Onflow. Onflow was trying to compete with Macromedia Flash. Their product was better because it allowed smaller downloads. If they could get their browser plugin installed on a lot of browsers, then they could (like Adobe today) charge advertisers hundreds of dollars for their program to create ads. According to this marketing guy, if I included the Onflow installer with Banshee Screamer Alarm, they would pay me 14 cents a download. I accepted, and I included their installer in my program.
<p>
A few months later, I got a check for about $1014 US, which I used to buy much needed clothing (my wardrobe at the time consisted of T-shirts that I got by signing up for things online). Then the checks stopped coming. Apparently Onflow went defunct in the tech crash.
<p>
So at one time adware was a very successful model. But what about today? I recently researched this topic. We all remember when the Opera browser had banner ads. At one point, pkzip for windows had banner ads too. I searched for ways of including ads in my programs, but all the companies that do this have apparently gone out of business.
<p>
The most successful company is Zango Cash, which apparently pays a huge rate for installs (if their web site is true). I refused to work with them, however. After some research, I found that they are the creators of the CoolWebSearch toolbar, which crippled my grandmother's computer. I spent a couple of hours trying to remove it, so I will not inflict this on people even for .40 cents a download.
<h2>Web Ads</h2>
When I was first promoting Hotkey Jumpstart, I dropped $60 into the Google adsense program for zero return. I read horrible stories about sweat shops that get paid to sit there all day clicking on Google ads. So the entire adsense program stunk to me. However, when I released <a href="http://www.hanovsolutions.com/?prod=PhotoWipe">PhotoWipe</a> I found that my web site was getting thousands of hits a day, so I signed up for adsense.
<p>
My main problem was that people didn't have to visit my web site in order to download PhotoWipe. So I modified the installer to open up a "thank-you for downloading PhotoWipe" web page after you install it. (This is also how I track how many downloads vs installs I have).
<p>
On that web page, I put in the google ad for Picassa, which is actually very relevant. It says "Organize your Photos with Google Picassa". So problem solved! Every install gets exposure to the ad. One important remark: Google claims their "referral" program pays "up to" $2 per install. This is a blatant lie. Actually, I get 10-20 cents per install.
<P>
One problem was that (as far as I can tell) google referral ads don't change their language according to the user, but most of my installs were coming from Japan and Spain. My php code takes care of that, my choosing the ad based on the HTTP_ACCEPT_LANGUAGE code.
<p>
<h3>Results for ads</h3>
If all you want to do is pay for bandwidth, it's okay. Right now, people downloading PhotoWipe consume about 1 GB/day, which costs me $1 from my internet provider. I get about 4-5 installs of Google Pack per day, which is just over $1. So I'm just scraping by with a few cents a day of profit.
<p>
Once a week or so, somebody clicks on a $1 ad and my profits skyrocket for that day. Also, I seem to get a few dollars more, for a couple of days, whenever PhotoWipe makes it to the front page of a major web site (usually in Japan). But such earnings are short-lived, and the bandwidth costs make up the difference.
<h2>Donations</h2>
Since my shareware business is failing so badly, I wondered if donation works. At <a href="http://www.donationcoder.com">Donation Coder</a> there is a discussion of it. Overall, it doesn't work. <p>
The "Thank-you for installing PhotoWipe" page also has a paypal "donate" button, as does the Help menu. In one month, I have had three donations that total to $18. At 15,000 installs (that's installs, not downloads), that's pretty dismal.
<h2>Conclusions</h2>
I will continue this experiment, and updating this entry as new facts come in. Right now, it looks like the Google Adsense is a clear winner for keeping up with bandwidth costs, but there is not enough to make a profit. Donations come in second, but there is not enough data, since they come in so sporadically. Shareware fails for the Windows platform, because people won't download it. If I were unscrupulous, I could probably make a few thousand dollars a month with spyware / trojans.
<ul><li><a href='?id=81'>Building a better rhyming dictionary</a><li><a href='?id=2'>UMA's dirty secrets</a><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=86'>Boring Date (comic)</a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a><li><a href='?id=93'>Zwibbler: A simple drawing program using Javascript and Canvas</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a></ul>
Draw waveforms and hear them
tag:smhanovtechblog,2007:id13
2007-04-11T19:18:09-05:00
2007-04-11T19:18:09-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
A while back I thought it would be interesting to be able to draw arbitrary waveforms and then listen to how they sound. I had an audio engine just laying around, so I whipped up a quick application to do that.
<p>
<center>
<img src=wavestudio.png><br>
<a href="WaveStudio.exe">download WaveStudio.exe</a>
</center>
<p>
<h2>Results</h2>
In theory, you can make any sound that you want. The results aren't very interesting. You can draw a sine wave and it sounds muffled. Add some jagged edges and the sound starts to sound more raw and high pitched. But it's okay to demonstrate what a sawtooth vs. sine vs. square wave sound like.
<h2>Future work</h2>
It would be a better to be able to draw in the time vs. frequency domain, using standard brush painting tools. Thay way you could come up with more interesting waveforms.
<ul><li><a href='?id=149'>I found Security Vulnerability in your web application</a><li><a href='?id=76'>Does Android team with eccentric geeks? (comic)</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=67'>Game Theory, Salary Negotiation, and Programmers</a><li><a href='?id=121'>An instant rhyming dictionary for any web site</a><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=120'>Succinct Data Structures: Cramming 80,000 words into a Javascript file.</a></ul>
Cell Phones on Airplanes
tag:smhanovtechblog,2007:id11
2007-04-08T20:22:03-05:00
2007-04-08T20:22:03-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
Much ink has been spilled about the use of cell phones on airplanes. Here's the truth, which will be disappointing to conspiracy theorists: Cell phone signals most definately have an effect on other electronic equipment. Read on for more.
<p>
Want proof? hold you cell phone near your computer speakers. Turn up the volume. Then make a call. You will be able to hear out loud the GSM signal. Although the microwave frequency isn't in the audible range, the envelope of the radio bursts is, so you are hearing a buzz formed by the radio packets going over the air.
<p>
You will only here these bursts when using the older 2G technology. 3G/HSDPA/LTE still interferes with equipment, but the designers specifically took the speaker interference into account. The radio bursts of newer phones are spread out in frequency and time, so that even though they are there, you can't hear them.
<h2>Interference</h2>
<p>
Sure, cell phones will probably work on airplanes. The range of a cell tower is a maximum of 30 km. How fast is the plane traveling? About 885 km/h, which works out to 14.75 km/minute. Cell phones only take about 5-10s to arrange a handover to another cell during a call, so it's certainly possible.
<p>
The problem is that since you are so far off the ground, and because the airplane is made of metal (impervious to radio signals), your phone will have to use the maximum transmit power. Such powers could easily interfere with radio equipment.
<h2>The real reason</h2><p>
There is a theory on the Internet that, while in the air, your cell phone can see many too many different base stations at once, and it can't handle it. In my tests (done without a SIM card, so that no transmission can occur) I have not had good results. As a protocol stack developer on the BlackBerry, I can make it go into a special mode where it shows all of the cell towers that it can see. What I see in commercial airlines is that it may be able to see one or two cells, but they will very quickly disappear. There won't even be enough time to register.
<p>
The real dealbreaker is that airlines tend to fly over open space most of the time. Cell towers cover only moderately populated areas, and most of the time, you won't be in range of a tower <ul><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=139'>The strange man reading a novel in the meeting room</a><li><a href='?id=137'>Asana's shocking pricing practices, and how you can get away with it too</a><li><a href='?id=135'>How I run my business selling software to Americans</a><li><a href='?id=84'>I didn't know you could mix and match (comic)</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=114'>Fast and Easy Levenshtein distance using a Trie</a></ul>
Detecting C++ memory leaks
tag:smhanovtechblog,2007:id10
2007-04-07T19:38:11-05:00
2007-04-07T19:38:11-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
A while ago I had the problem of detecting memory leaks in my code, and I didn't want to spend lots of money on a brittle software package to do that. It's fairly simple to redefine malloc() and free() to your own functions, to track the file and line number of memory leaks. But what about the new() and delete() operators? It's a little more difficult with C++, if you want to figure out the exact line number of a resource leak.
<p>
In this article, I'll explain how you can get a stack trace for where your resource leaks occur. This method is for Microsoft Windows. Linux developers are better served with <a href="http://valgrind.org/">Valgrind</a>.
<p>Download the source code:
<ul>
<li><a href="debug.h">debug.h</a>
<li><a href="debug.cpp">debug.cpp</a>
<li><a href="MapFile.h">MapFile.h</a>
<li><a href="MapFile.cpp">MapFile.cpp</a><br>
</ul>
<ul>
<li><a href="MemExample.zip">MemExample.zip</a> (Sample project)
</ul>
<h2>Overview</h2>
<ul>
<li>We will use #define to replace the standard implementation of malloc() and free() with ones that record the file and line numbers where they are called. That way, we can track where memory leaks occur for allocations made using the standard C allocation functions.
<li>We will overload the new() and delete() operators to track the address of the functions that they are called, by walking backwards up the stack.
<li>Finally, we will parse the .map file generated by the linker. This will let us figure out where new() and delete() were called based on the return address information.
</ul>
<h2>The header file</h2>
<p>
The first thing we'll do is have an #ifdef, because memory tracking is inefficient. You'll want to cut it out in release versions of your code.
<p>
debug.h:<br>
<pre>
#ifdef DEBUG_MEM
#include <new>
#define malloc(A) _dbgmalloc(__FILE,__LINE, (A) )
#define free(A) _dbgfree( __FILE__, __LINE__, (A) )
// ... continue with calloc, realloc, strdup, etc.
#endif
</pre>
<p>
Every *.cpp source file in your program should include this file. It's optional, of course. But if you allocate something in a memory-tracked module, and free it in another that doesn't, your program will crash, since it was allocated with _dbgmalloc() and free'd with free() instead of _dbgfree().
<h2>The implementation for malloc</h2>
<pre>
void*
_dbgmalloc( const char* file, int line, size_t size )
{
void* ptr;
if ( !_init ) {
return malloc( size );
}
ptr = add_record( file, line, size );
if ( ptr == 0 ) {
dbgprint(( DMEMORY, "Out of memory." ));
return 0;
}
dbgprint(( DMEMORY, "%s:%d: malloc( %d ) [%p]", file, line, size, ptr ));
return ptr;
}
void _dbgfree( const char* file, int line, void* ptr )
{
if ( ptr == 0 ) {
return;
}
if ( !_init ) {
free( ptr );
return;
}
MemBlock* block = (MemBlock*)ptr - 1;
int size = block->size;
del_record( file, line, ptr );
dbgprint(( DMEMORY, "%s:%d: free( [%p], %d )", file, line, ptr, size ));
}
</pre>
<p>
The add_record() and del_record() functions perform the real work of memory tracking. They will allocate the requested amount of memory, but they will add space for extra tracking information. The tracking information is stored in the first few bytes of the memory block, and then the returned pointer offset by this amount. We will also reserve extra space at the end of the memory block, so we will be able to detect writes past the end of the array. We will write a specific sequence of bytes (Here, 0x12345678) at this location, and if when the block is free'd, the bytes have been modified, then your program has done something it shouldn't have, and the del_record() function will complain.
<pre>
void*
add_record( const char* file, int line, size_t size )
{
MemBlock* block;
assert(_init);
block = (MemBlock*)malloc( sizeof( MemBlock ) + size + 4 );
if ( block == 0 ) {
dbgprint(( DMEMORY, "Out of memory." ));
return 0;
}
block->sentry = SENTRY;
block->size = size;
block->line = line;
block->file = _strdup( file );
if ( 0 == block->file && file ) {
free( block );
dbgprint(( DMEMORY, "Out of memory." ));
return 0;
}
memcpy( (char*)block + sizeof(*block) + size, &SENTRY,
sizeof( SENTRY ) );
EnterCriticalSection(&_cs);
list_add_tail( &_blockList, &block->list );
LeaveCriticalSection(&_cs);
return block + 1;
}
</pre>
<h2>What about new?</h2>
That's all fine and good for malloc() and free(), and strdup() and _tcsdup() and calloc() and realloc(), but what about C++? When you call malloc() above, you see that the macro puts in the file and line number information, but this is not possible for the new operator. Instead, we will do it the hard way. We'll redefine the new operator and then search up the stack for the caller's address and store that. Later, we'll parse the linker's map file to figure out which function it was from the address.
<p>
Here's the implementation for new() and delete(). They are almost the same as malloc() and free() above, except that they record the return address instead of the file and line information.
<pre>
void* operator new( size_t size ) throw ( std::bad_alloc )
{
static bool recurse = false;
void* ret;
CrashPosition_t pos;
if ( recurse || !_init) {
return malloc( size );
}
EnterCriticalSection(&_cs);
pos = getFileLine(1);
if ( pos.file == 0 ) {
pos.file = pos.function;
}
ret = add_record( pos.file, pos.line, size );
if ( ret == 0 ) {
dbgprint(( DMEMORY, "Out of memory." ));
LeaveCriticalSection(&_cs);
return 0;
}
dbgprint(( DMEMORY, "%s:%d: new( %d ) [%p]", pos.file, pos.line, size, ret ));
LeaveCriticalSection(&_cs);
return ret;
}
/******************************************************************************
*****************************************************************************/
void operator delete( void* ptr ) throw ()
{
CrashPosition_t pos;
if ( !_init ) {
free( ptr );
return;
}
if ( ptr == 0 ) {
return;
}
EnterCriticalSection(&_cs);
pos = getFileLine(2);
LeaveCriticalSection(&_cs);
dbgprint(( DMEMORY, "%s:%d: delete [%p]", pos.file, pos.line, ptr ));
del_record( pos.file, pos.line, ptr );
}
</pre>
<h2>Walking the stack</h2>
Here's where the magic happens. Because file and line number information is not available to the new operator, we will walk the stack in order to record the return address. Later on, we'll figure out the function name where they were called from.
<pre>
static int
GetCallStack( unsigned* stack, int max )
{
unsigned* my_ebp = 0;
int i;
__asm {
mov eax, ebp
mov dword ptr [my_ebp], eax;
}
// It is not safe to use this function in a WIN32 standard exception handler!
if ( IsBadReadPtr( my_ebp + 1, 4 ) ) {
return 0;
}
stack[0] = *(my_ebp + 1);
for ( i = 1; i < max; i++ ) {
unsigned addr;
if ( IsBadReadPtr( my_ebp, 4 ) ) {
break;
}
my_ebp = (unsigned*)(*my_ebp);
if ( IsBadReadPtr( my_ebp + 1, 4 ) ) {
break;
}
addr = *(my_ebp + 1);
if ( addr ) {
stack[i] = addr;
} else {
break;
}
}
return i;
}
</pre>
<h2>Making the map file</h2>
So far, for malloc() and free() calls, we have recorded the file and line number information, but for new() and delete() we have only the return address. How do we figure out which function called new() and delete()?
<p>
We will induce the linker to create a .map file. Add these options to your makefile when calling link.exe. Replace <i>example</i> with the name of your executable output file. (The debug.cpp code will assume that the map file has the same base name as the executable).
<pre>
/MAP:example.map /MAPINFO:LINES
</pre>
<h3>Compiler differences</h3>
Note: For Microsoft Visual Studio 2005, Microsoft has removed the MAPINFO:LINES option. So you should either use an earlier version of the compiler, or be content without line numbers. You will still have function names.
<h2>The Map File</h2>
The Map file contains a list of every function in your program, and the exact addresses to which they are loaded. So, using a binary search, we are able to look up a function given an address. I have implemented this process in Mapfile.cpp, which is called diretly from debug.cpp.
<h2>Putting it together</h2>
When your program exits, the debug.cpp module will automatically execute this cleanup code. The cleanup code will dump out any unfree'd memory chunks.
<pre>
static void
dump_blocks()
{
list_entry_t* entry = list_head( &_blockList );
while( entry != &_blockList ) {
MemBlock* block = list_entry( entry, MemBlock, list );
dbgprint(( DMEMLEAK, "Leaked %d bytes from %s:%d [%08x]",
block->size, block->file, block->line, block + 1
));
entry = entry->next;
}
if ( list_empty(&_blockList ) ) {
dbgprint(( DMEMLEAK, "No memory leaks detected." ));
}
}
</pre>
<h2>dbgprintf</h2>
To see the memory leaks, you will have to implement a debug message handler. I don't have time to explain this right now, but it should be pretty obvious from the source code. Or, you can replace dbgprint() with OutputDebugString(), or printf(), or MessageBox(), or whatever you want.
<h2>Enjoy!</h2> <ul><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=35'>Copy a cairo surface to the windows clipboard</a><li><a href='?id=128'>Why you should go to the Business of Software Conference Next Year</a><li><a href='?id=111'>The Curious Complexity of Being Turned On</a><li><a href='?id=74'>Blame the extensions (comic)</a><li><a href='?id=95'>C++: A language for next generation web apps</a><li><a href='?id=119'>Throw away the keys: Easy, Minimal Perfect Hashing</a></ul>
What does your phone number spell?
tag:smhanovtechblog,2007:id9
2007-04-07T18:35:22-05:00
2007-04-07T18:35:22-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
<form action="http://gandolf.homelinux.org/~smhanov/cgi-bin/spellophone.cgi" method=get>
Type it in here and see.
<input maxLength=15 size=20 name="hello"><input value=submit type=submit value="">
</form>
<em>You might want to visit <a href="http://dialabc.com/words/search">DialAbc.com</a>, which has more results.</em> Stay here if you are interested in the theory behind it.
<p>
This article was actually written in 2002. Here, I explain a technique for figuring out which words are in which phone numbers. Full C source code is included.
</p>
<h2>How did the computer do that?</h2>
<h3>Quick answer:</h3>
<ul>
<li><a href="TrieStore.h">TrieStore.h</a>
<li><a href="TrieStore.c">TrieStore.c</a>
<li><a href="main.c">main.c</a>
<li>Compile using gcc -o spellophone *.c on unix/linux. A competent C
programmaer can adapt it to run on Windows in about 5 minutes.
</ul>
<h2>Long answer:</h2>
<p>
Computers are wonderful things. We take for granted that they can do stuff
like the above in microseconds, but to non-computer scientists it seems
like magic. If you've ever wondered what computer scientists do all day,
this will help.
<p>
I developed the algorithm for spellophone over Christmas break 2001.
It wasn't easy -- I went through three different drafts until I found
the right one. Would you believe that the first one took over ten minutes
to run on a ten-digit number on my old Pentium 166?
<p>
The first algorithm I tried is known as the "brute force" approach. I
had the computer go through every possible combination of letters and
check it for words. I thought I had a clever way of doing that because
I used a <i>trie</i> to store all of the words in the dictionary.
A trie is like a very smart parrot. You can shout letters at it, and it
will squawk if anything you say forms a word. For example, if you say
"D-O-G" at it, it will squawk because DOG is a word. If you then say
"M-A" it will squawk again because DOGMA is also a word.
<p>
Despite the clever use of the trie, the program still ran too slowly.
Can you guess why? Think about how many possible letters are in a
telephone number.
<p>
If you look at a phone keypad, you'll see that each digit has three letters
on it. Some of them have four on newer phones. So you can only make
three one-letter words with one digit. But if you add a second digit, say
"22", you can spell "AA, AB, AC, BA, BB, BC, CA, CB, CC", or 9 words.
That's quite a bit more than one! It turns out that if you add 10 digits,
there are 1049760 possible combinations of letters. And for each possible
combination that spells actual words, there are lots of different ways
those words can be placed together between dashes. So it turns out that
the computer might have to go through millions of combinations, trying
to pick out the ones that make sense.
Unfortunately, even today's computers can't process all of those words
fast enough. So I had to find a better way.
<p>
<h2>Dynamic Programming</h2>
I looked at what the computer was doing, and it seemed to me that it
was wasting a lot of its time asking the trie about the same words
over and over again. For example, if the last three digits in phone number
didn't actually spell anything, it would still check them thousands of
times in combination with the first part of the phone number.
<p>
Then something struck me. Last year, I had learned of something that
was meant to deal with just this type of problem in one of my
computer science courses. It's called "Dynamic Programming", or DP for
short.
Dynamic Programming is a way of programming that solves problems
a little bit at a time. It's best for problems where each piece of the
the problem is built on the previous one, so once you solve all of these
little pieces you can put them together and solve the entire puzzle.
If I could figure out a way to make DP work, then the computer would
only need to check each word once, and the program might work in seconds
instead of minutes!
<p>
I racked my brain, trying to remember what Professor Chan had said in his
thick accent. With DP, you have to make a grid of squares, and each
square represents a small part of the solution. After a lot of thinking,
I drew a grid on paper. Across the top, I wrote <i>Starting Position</i>,
and along the side, I wrote <i>Length of word</i>. Each square would
contain all of the words that you could spell if you started on a certain
digit and used a certain number of letters.
<p>
My hands trembled as I stepped through the algorithm on paper. I used the
phone number "78225" because I knew it spelled "QUACK." (I used to work
for a company called Quack.com and that was part of their number). Here's
what I came up with. The partial words are in normal printing, and the
finished words in each square are in bold:
<p>
<table align=center cellspacing=5 cellpadding=2 border=3>
<tr>
<th colspan=6>Starting digit</th>
</tr>
<tr>
<th rowspan=6>Length</th>
<th>7 (PQRS)</th><th>8 (TUV)</th><th>2 (ABC)</th><th>2 (ABC)</th>
<th>5 (JKL)</th>
</tr>
<tr>
<td>
P, Q, R, S
</td>
<td>
T, U, V
</td>
<td>
A, B, C
</td>
<td>
A, B, C
</td>
<td>
J, K, L
</td>
</tr>
<tr>
<td>
PU, QU, RU, ST, SU
</td>
<td>
TA, UB, VA
</td>
<td>
AA, <b>AB</b>, AC, BA, CA
</td>
<td>
AL, BL, CL
</td>
<td>
.
</td>
</tr>
<tr>
<td>
<b>PUB</b>, PUC, QUA, RUB, STA, SUC
</td>
<td>
<b>TAB</b>, TAC, VAC
</td>
<td>
ACK, BAL, CAJ, CAK, CAL
</td>
<td>
.
</td>
<td>
.
</td>
</tr>
<tr>
<td>
<b>PUBA</b>, QUAC, RUBA, RUBB, <b>STAB</b>, STAC, SUCC
</td>
<td>
<b>TACK</b>
</td>
<td>
.
</td>
<td>
.
</td>
<td>
.
</td>
</tr>
<tr>
<td>
<b>QUACK</b>, STABL, <b>STACK</b>
</td>
<td>
.
</td>
<td>
.
</td>
<td>
.
</td>
<td>
.
</td>
</tr>
</table>
<p>
What is good about this is the table could be calculated very quickly.
Each square builds on the data that was already processed in the square
above.
I had done it in five minutes on paper -- the computer could do it
in the blink of an eye. Also, it is pretty easy to string all of the
words together so that the longest ones appear first.
<p>
Dynamic Programming always involves two steps -- first creating the
solution table and then analyzing it to piece the solution together.
The way I chose to piece the solution together is pretty simple:
<ol>
<li>Begin at the leftmost column that you haven't worked on yet.
<li>Get a word from the bottom-most square.
<li>Now put that word in your phone number, and go to the digits
that are now left-over at the end. Start at step 1.
<li>Once you run out of words, try the next one from the square you
chose in step 2 before you went off to the left-over digits. If there
are no more words left in the square, continue up-wards. Repeat
step 3.
<li>Once you have exhausted the left-most column, get rid of it,
add the number instead of a letter, and start over until there
are no more columns left in the table.
</ol>
<p>It's harder to explain than to do. Using the above data, you get
the following words in this order:
QUACK, STACK, STAB-5, PUBA-5, PUB-25, 7-TACK, 78-AB-5
<h2>Other work</h2>
There are quite a few web sites that do this kind of thing. <a href="http://www.phonespell.org">Phonespell.org</a> has a good description of how their algorithm works, in the F.A.Q. section.
<p><a href="http://dialabc.com">dialabc.com</a> is the nicest web site that I have seen to search for words in phone numbers.
<ul><li><a href='?id=85'>barcamp (comic)</a><li><a href='?id=130'>VP trees: A data structure for finding stuff fast</a><li><a href='?id=99'>The simple and obvious way to walk through a graph</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=106'>Five essential steps to prepare for your next programming interview</a><li><a href='?id=53'>cairo blur image surface</a><li><a href='?id=57'>Coding tips they don't teach you in school</a></ul>
A Rhyming Engine
tag:smhanovtechblog,2007:id8
2007-04-06T10:31:11-05:00
2007-04-06T10:31:11-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
<p>
Here's a rhyming engine, written in 1000 lines of C++ code. It uses the freely available Moby dictionary, and full source code is provided. Give it a try. Read on for technical information.
<p>
<em>Please try the <a href="http://rhymebrain.com">updated rhyming web site</a>.</em>
<p>
<script language="Javascript">
var haveImage = false;
var style = "default";
function xmlhttpPost() {
var form = document.forms['f1'];
updatePage("");
strURL = "http://stevehanov.ca/cgi-bin/poet.cgi?" + getquerystring();
var xmlHttpReq = false;
var self = this;
// Mozilla/Safari
if (window.XMLHttpRequest) {
self.xmlHttpReq = new XMLHttpRequest();
}
// IE
else if (window.ActiveXObject) {
self.xmlHttpReq = new ActiveXObject("Microsoft.XMLHTTP");
}
self.xmlHttpReq.open('POST', strURL, true);
self.xmlHttpReq.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
self.xmlHttpReq.onreadystatechange = function() {
if (self.xmlHttpReq.readyState == 4) {
updatePage(self.xmlHttpReq.responseText);
}
}
self.xmlHttpReq.send("");
document.getElementById("status").innerHTML = "Please wait...";
}
function getquerystring() {
var form = document.forms['f1'];
var word = escape(form.word.value);
return word;
}
function updatePage(str) {
var form = document.forms['f1'];
document.getElementById("status").innerHTML = "";
form.result.value = str;
}
</script>
<form name=f1 action="index.php" method="POST">
<input type=text value="shine" name="word"/>
<input type=button value="Find Rhymes" onclick='xmlhttpPost()'/>
<div style="color:red; text-decoration: blink;" id=status> </div>
<div id=debug> </div>
<br><br>Results:<br>
<textarea cols=60 name="result" rows=10 ></textarea><br>
</form>
<blockquote>
<h2>Relationship to Rhymebrain.com</h2>
I originally wrote this post in 2007. For the past five years I have been researching linguistics and machine learning techniques and created <a href="http://rhymebrain.com">rhymebrain.com</a>. Rhymebrain, at its core, still uses the same techniques, but it does it much faster and supports any human language. Rhymebrain is propreitary. However, the source code presented with this article is released to the public domain.
</blockquote>
<h2>How it works</h2>
Programs involving the english language are quite interesting, because there really are no rules. English is very complicated. A great resource is the <a href="http://www.dcs.shef.ac.uk/research/ilash/Moby/">Moby project</a>, which is a public domain dictionary. It includes a text file providing word lists, including pronunciation and parts of speech.
<p>
Although I demonstrate only the rhyming part, my WordDatabase object uses both the part of speech and pronunciation information. This is for future expansion.
<a href="poet.zip">Download source code here</a>
<h2>Overview</h2>
<ul>
<li>As part of compilation, a perl script combines the Parts Of Speech and Pronunciation dictionary into a single file, dict.txt.
<li>The rhyming engine reads dict.txt and, for every word, creates a Word object.
<li>The Word object has the part of speech. It also contains a representation of the pronunciation of the word.
<li>To figure out if two words rhyme, we compare the last part of their pronunciation. The more syllables that rhyme, the better the rhyme is.
<li>The output is sorted so that better rhymes are listed first.
</ul>
<p>
<h2>Similar projects</h2>
I'm not aware of many similar projects. Somebody named "tuffy" made a rhyming dictionary on sourceforge: <a href="http://rhyme.sourceforge.net/">rhyme.sourceforge.net</a>. However, for the life of me, I can't figure out why it needs to use an external database package. My technique does not need to pre-compute the rhymes, and it is half as many lines of code.
<p>
Personally, I use the <a href="http://www.rhymezone.com">rhymezone.com</a> rhyming dictionary for my rhyming needs.
<h2>The Rhyming API</h2>
<pre>
class WordDatabase
{
public:
WordDatabase();
~WordDatabase();
bool load(const char* filename);
bool findRhymes( DynamicArray<Word*>& results, const char* word, WordFilter* filter,
WordArray* wordList = 0);
Word* lookup( const char* whichword );
void filter( WordArray& results, WordFilter* filter );
bool makeWords( WordArray& results, const char* text );
bool loadThesaurus( const char* thesaurus );
void addSynonym( Word* base, Word* synonym );
unsigned getSynonyms( WordArray& results, const char* wordtext );
private:
StringMap<Word*> _wordMap;
DynamicArray<Word*> _wordArray;
};
</pre>
<p>
In this introduction, I use the word phoneme to mean a part of a word, as pronounced. Using a sequence of phonemes, and whether each phoneme is emphasized, you can completely describe how to pronounce a word, and hence derive its rhymes and syllables.
<h2>Preprocessing</h2>
The Moby project has separate files for parts of speech and pronunciation. So I wrote a perl script to combine the two files. The resulting word list only includes words that have both part of speech and pronunciation information.
<p>
From the readme file, I believe that the Moby pronunciation was derived by a British person. Fortunately, for the purposes of rhyming, it doesn't really matter. "Potato" will rhyme with "Tomato" no matter if you say "Po-tay-to" or "po-tah-toh".
<h2>A standard set of phonemes</h2>
For this project, I also leave it open to include the <a href="http://www.speech.cs.cmu.edu/cgi-bin/cmudict">CMU dictionary</a>, another free online dictinary. The problem is that these two dictionaries use a different set of phonemes. I had to figure out a mapping so that I could possible combine both dictionaries later on. The mapping is below. The fields are:
<ul>
<li>An enumeration for the phoneme
<li>How the phoneme is displayed. This is for debugging purposes only, and is different from what is read in from the dictionary file.
<li>Whether it represents a syllable.
</ul>
<pre>
PhoneSet_t PhoneSet[] = {
{ a_in_dab, "ae", 1 },
{ a_in_air, "ey", 1 },
{ a_in_far, "ao", 1 },
{ a_in_day, "ay", 1 },
{ a_in_ado, "ah", 1 },
{ ir_in_tire, "ire", 1 },
{ b_in_nab, "b", 0 },
{ ch_in_ouch, "ch", 0 },
{ d_in_pod, "d", 0 },
{ e_in_red, "e", 1 },
{ e_in_see, "ee", 1 },
{ f_in_elf, "f", 0 },
{ g_in_fig, "g", 0 },
{ h_in_had, "h", 0 },
{ w_in_white, "wh", 0 },
{ i_in_hid, "i", 1 },
{ i_in_ice, "eye", 1 },
{ g_in_vegetably, "g", 0 },
{ c_in_act, "k", 0 },
{ l_in_ail, "l", 0 },
{ m_in_aim, "m", 0 },
{ ng_in_bang, "ng", 0 },
{ n_in_and, "n", 0 },
{ oi_in_oil, "oy", 1 },
{ o_in_bob, "aa", 1 },
{ ow_in_how, "ow", 1 },
{ o_in_dog, "ah", 1 },
{ o_in_boat, "oh", 1 },
{ oo_in_too, "oo", 1 },
{ oo_in_book, "ooh", 1 },
{ p_in_imp, "p", 0 },
{ r_in_ire, "er", 0 },
{ sh_in_she, "sh", 0 },
{ s_in_sip, "s", 0 },
{ th_in_bath, "dth", 0 },
{ th_in_the, "th", 0 },
{ t_in_tap, "t", 0 },
{ u_in_cup, "uh", 1 },
{ u_in_burn, "u", 1 },
{ v_in_average, "v", 0 },
{ w_in_win, "w", 0 },
{ y_in_you, "y", 0 },
{ s_in_vision, "zh", 0 },
{ z_in_zoo, "z", 0 },
{ a_in_ami, "a", 1 },
{ n_in_francoise, "n", 0 },
{ r_in_der, "r", 0 },
{ ch_in_bach, "chh", 0 },
{ eu_in_bleu, "eu", 1 },
{ u_in_duboise, "u", 1 },
{ wa_in_noire, "WA", 1 }
};
</pre>
<h2>The combined dictionary</h2>
Here's the first few lines of the combined dictionary, which includes 110000 words. Each line has three parts. The first is the word, the second is the part of speech (N is noun, etc) and the third is the pronunciation. Right now, I use only the moby dictionary, so the pronunciation keys are weird characters as described in the moby readme file. Each phoneme is separated by a slash character.
<pre>
A N /eI/
AWOL A '/eI/w/A/l
Aachen N '/A/k/@/n
Aalborg N '/O/lb/O/rg
Aalesund N '/O/l/@/,s/U/n
Aalst N /A/lst
Aalto N '/A/lt/O/
Aar N /A/r
</pre>
<h2>Syllables</h2>
It took a bit of thinking to figure out how to count the syllables in the word. Believe it or not, it can be derived from the phoneme alone. The first approach I tried is to split the word into syllables based on consonents. This didn't work. In fact, you get the best results if you split based on vowels. (Linguists reading this are now shouting at the monitor, <i>"Of course, you fool!"</i>) In the PhoneSet table above, I mark a phoneme with a 1 if it represents a new syllable. The number of syllables in a words can be derived by adding up the value of the phoneme.
<h2>Representing words</h2>
On startup, the program takes a second to read the dictionary file. It parses the phoneme (separated by slash characters) into a word structure, where each phoneme is represented by a 16 bit number. If the phoneme begins a new syllable, the upper 2 bits are:
<ul>
<li>0 if the phoneme doesn't begin a new syllable
<li>1 if the phoneme is a secondary stress
<li>2 if the phoneme is of primary stress
</ul>
<h2>Rhyming</h2>
Rhyming is not very complex once the words are in the phoneme format. To determine whether two words rhyme, you simple compare the suffixes of their pronunciation. Stresses are included, as well. Amateur song writers often try to rhyme things like "hello" and "yellow" and they come up with horrible lyrics, since the words are stressed differently. Since the stresses are stored with the phonemes, a simple string comparison will take care of this automatically, and these words will not be found to rhyme.
<h2>Other features</h2>
<p>
Although I didn't include it here, the word database can optionally load in the moby thesaurus, "mobythes.aur". Thus you can figure out if any word has the same meaning as any other.
<p>
You can use a "WordFilter" object to filter the results to have a certain number of syllables, or match a set of stresses. For example, you can request a noun phrase of 5 syllables to match the stresses "0101010101", which is iambic pentameter.
<h2>Future work</h2>
Lots of stuff can be done. I have noticed in hip hop and rap music, the rhyming rules are relaxed greatly. For example, eminem might consider <i>time</i> and <i>spine</i> to be good rhymes. By redefining more phonemes to be identical, you can emulate this type of rhyming.
<ul><li><a href='?id=22'>Exploring sound with Wavelets</a><li><a href='?id=4'>UMA and free long distance</a><li><a href='?id=104'>Compress your JSON with automatic type extraction</a><li><a href='?id=147'>My favourite Google Cardboard Apps</a><li><a href='?id=28'>Why are all my lines fuzzy in cairo?</a><li><a href='?id=62'>Exploiting perceptual colour difference for edge detection</a><li><a href='?id=119'>Throw away the keys: Easy, Minimal Perfect Hashing</a></ul>
Rules for Effective C++
tag:smhanovtechblog,2007:id7
2007-04-06T10:25:28-05:00
2007-04-06T10:25:28-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
I used to be a strong supporter of C++. It was the perfect language. In C++, if you want to influence how the hardware instructions are generated, you can do that. If you want to program without pointers and without caring about how memory is allocated, you can do that.
<p>
Recently, however, my views have changed after reading Scott Meyer's book, <u>Effective C++</u>. In
Meyer's book, he goes through every feature of C++ and shows you how you have to program with extreme care to avoid undefined behaviour. It seems like every modern feature that C++ has was specifically designed to help you shoot yourself in the foot.
<p>
I never realized this before, because I simply never use these dangerous features. In this article, I'll show you how to program in C++ safely.
<p>
<h2>C++'s Broken Exceptions</h2>
Take exceptions, for example. Many programmers will tell you that they are a great idea. They let you
indicate errors when creating object, and avoid making you check return codes. You can handle errors in
the areas of the code that is prepared to handle them.
<p>
What you may not know is that using exceptions in C++ makes a lot of code unsafe. In effect, it means
that you cannot use pointers. Take this code, for example:
<p>
<pre>
void foo()
{
MyObject* obj = new MyObject();
bar();
delete obj;
}
</pre>
<p>
If you are a C++ programmer and you use exceptions, you should see the obvious memory leak. If bar()
throws an exception, or calls any function that throws an exception, then obj will not be deleted.
<h2>Steve's rules for effective C++</h2>
I have been programming in C++ for a decade, and I never realized these flaws until I read Meyer's
books. I find C++ to be just fine, and the reason for that is because I program in a style that doesn't
involve these pitfals. Here's how you can program in this way too:
<h3>Avoid exceptions</h3>
Exceptions will only leave you open to the memory and resource leaks. Don't use them. The exception to this rule is if you are programming in a style that doesn't use pointers, and everything is encapsulated into smart pointers.
<h3>Constructors should do nothing</h3>
Constructors have no way of returning an error code (unless you use exceptions, which are bad). That means that your constructors shouldn't do any real work. Don't try to open up a database connection, or call any functions that could fail.
Constructors should be used only to initialize data members.
<h3>Use copy constructors sparingly</h3>
Copy constructors are very error prone, because they are another thing that you have to remember to change if you add a data member. You're much better off if you don't allow copying at all. Just pass pointers around. If you must pass by value, then don't put anything in your object, like pointers, that will require special handling. That way, you can use the automatically generated copy constructor. Unlike you, the compiler will never forget anything.
<h3>Use malloc or new without checking the result</h3>
I used to write programs that checked every call to malloc() and new() for failure. In Microsoft C++, the new() operator will actually through an exception if it fails, so checking for NULL is useless anyway. Today's machines have gigabyes of memory, and you don't need to verify every call to malloc() or new().
<p>
It's actually quite hard to induce these functions to fail, so even if you did handle their failure, you probably wouldn't test it. Do you really want to be releasing code that you haven't tested? There are cases where it would be better for your program to crash, then to continue to operate in an undefined state.
<p>
However, there are times where I would check whether a memory allocation failed:
<p>
<ul>
<li><em>When you are allocating something that is several megabytes</em>, like space for images or files. In this case, it is quite possible for the allocation to fail if the user has opened up too many files in your program.
<li><em>When you are programming for a nuclear reactor, or space shuttle.</em> Also, a missile guidance system would be acceptable.
</ul>
When you are programming for an embedded device, however, it might be beneficial to not check the return code of malloc. This is when there should be enough memory in the heap for all operations. If your process silently fails when memory allocation fails, you might never catch a memory leak that is exhausting your heap. It is much better to fail catastrophically by trying to use the NULL pointer than silently failing.
<ul><li><a href='?id=134'>0, 1, Many, a Zillion</a><li><a href='?id=92'>qb.js: An implementation of QBASIC in Javascript </a><li><a href='?id=136'>5 Ways PowToon Made Me Want to Buy Their Software</a><li><a href='?id=143'>Let's read a Truetype font file from scratch</a><li><a href='?id=141'>You can cheat so your web site seems faster than it is</a><li><a href='?id=131'>[comic] Appreciation of xkcd comics vs. technical ability</a><li><a href='?id=135'>How I run my business selling software to Americans</a></ul>
Cell Phone Secrets
tag:smhanovtechblog,2007:id3
2007-04-03T17:00:27-05:00
2007-04-03T17:00:27-05:00
Steve Hanov
steve.hanov@gmail.com
steve.hanov@gmail.com
I am a mobile telecommunications "engineer", and I thought I'd explain what to look for in a cell phone. Most guides will review phones on their user interface, but pay little attention to one of the most important pieces: the radio. The radio on GSM cell phones is very mysterious to most people, so here is a guide on how to decode the features of cell phones.
<h2>Bands</h2>
The bands your mobile phone supports will depend on the region in which you live. In the year 2006, four bands are in common use: 850, 1900, 900, and 1800 MHz. The 850/1900 bands are used in North America, while the rest of the world (with a few exceptions) uses the 900/1800 bands. Your carrier will likely deploy either 850/1900 or 900/1800 in your area, so for maximum coverage your phone should support the two bands in use in your region.
<p>
If you are traveling, you will want a Tri-Band or Quad-Band phone. A Tri-Band phone supports the two bands in your country, plus one other one. It may also be advertised as a "World Band" phone. A Quad-Band phone will support all four bands.
<p>
All phones sold today should automatically detect and switch between the bands.
<h2>GPRS/EDGE</h2>
GPRS refers to the ability of the phone to transfer packet data (for example, emails, ring tones, and web pages). The speed of the connection is related to the multislot class. EDGE is an enhancement to GPRS that allows faster data rates, comparable to broadband connections. If all you want is phone calls, you should ask your carrier to disable your packet access, to avoid incurring usage charges by accident.
<h2>UMTS/3G</h2>
UMTS handsets, also called "3G", support a new technology that will theoretically give you faster data access than EDGE. However, because they are a newer technology, their battery life will be much less than a GSM-only phone. In addition, UMTS base stations are deployed only in major metropolitan centres. Recently, in the Blackjack phone, users discovered if they turned off the 3G feature, their battery life doubled!
<p>
If you are only making phone calls, you don't need this. It helps the carriers because it moves phone calls off of their congested GSM cells, onto their UMTS cells that hardly anybody uses at the moment. If you do want a data modem, you should consider it, because the higher data speeds will be noticeable.
<h2>Multislot class</h2>
The multislot class of your phone determines how quickly it will transmit or receive packet data. (For example, emails and web pages, but not voice or SMS messages.) Most phones will be Multislot Class 10, which means it can receive on four different channels simultaneously, or send on two different channels. Higher multislot classes allow more channels, and thus it will be faster. However, if the cell tower is being used by more than a few mobiles at the same time, this won't make any difference, because it will run out of channels.
<p>
Multislot class only applies to packet data, like web pages or picture messages. Phone calls only use one slot, anyway.
<h2>Dual Mode</h2>
A dual-mode handset will support two different radio technologies and switch between them when appropriate. For example, because UMTS base stations are deployed only in major cities, a UMTS handheld will probably be able to fall back on GPRS technology when you roam away from the city.
<h2>Dual transfer mode</h2>
Dual transfer mode handsets are expected to be deployed to some networks in 2007. With dual transfer mode, you will be able to transfer packet data during a phone call. This is something already supported by UMTS but not GSM, so there is a greater push for it in GSM phones.
<h2>Flip-Phone</h2>
Audio engineers love flip phones, because it brings the microphone closer to the mouth. Companies spend millions of dollars, and hire lots of Ph.D's to try to get the microphone to work when it is on your cheek, but a handset designer will tell you that you can get the best audio quality with a flip phone.
<h2>Data Modem</h2>
If your phone has data modem capabilities, you will be able to attach it to your computer and use it as a modem. However, be aware that your transfer speeds will be limited. GPRS/EDGE phones have an inherent limitation: The very first packet that you send (after a break of about 5 seconds) will take up to 1.5 seconds to start the transfer, although subsequent packets will be faster. This makes GPRS modems inefficient for the TCP protocol used by all networked PCs today. However, you should still be able to browse at speeds similar to dial-up.
<p>
Make sure to read the fine print: Your "unlimited" plan may actually only include a few MB/month, with steep charges if you go over the limit.
The limit is especially troubling, because Microsoft Windows will typically send several megabytes of data in a few minutes, because a lot of the software that you have will constantly be checking for updates.
<h2>Talk and Standby Time</h2>
Talk time and standby time are tested in a standardized way. Because they are not real conditions, handset manufacturers can employ certain tricks to get a better rating. Look for a talk time of at least 4 hours, and a standby time of at least 10 days, whether you need it or not.
<p>
All rechargeable batteries have a limited life. They are killed by both heat and time. You should get a lithium battery, which will last for several years. Nickle-Metal-Hydride (NiMH) batteries are good too, but exhibit a "memory-effect", which means that you should charge them only when they are close to empty. NEVER leave a battery the car in hot weather, because this shaves months off of its life. When you are replacing it, buy only a newly manufactured battery, because they will slowly die even if left in the package. For this reason, don't bother getting a spare when you buy the phone. If you do want to store your battery for a period of time, discharge it to 40%, and keep it in the refrigerator inside a sealed plastic bag. Make sure it is dry, or the contacts will rust.
<p>
The more gadgets your phone has, the more the battery will run out. External memory slots, GPS, and Wi-Fi are all things that will suck the juice out of your battery. Also, if you live on the fringes of coverage and get only 1 to 3 bars of service, your battery will only last a couple of days because the phone will have to transmit at maximum power. GSM phones have to update with the network every 5-15 minutes, so they will consume power even if you are not using it.
<h2>Wi-fi</h2>
The industry is working on a new feature, called GAN or UMA, which is already available in some areas. In an UMA phone, the carrier will also sell you Internet access and give you a wireless access point. While in your home, your calls will go through the Wi-Fi connection. When you leave your house, the calls will be handed over to the cellular (even if you are talking). This means that you will always get full service in your home, but it will also reduce the standby time. You should get it only if you want to buy Internet service from your carrier.
<ul><li><a href='?id=128'>Why you should go to the Business of Software Conference Next Year</a><li><a href='?id=56'>How a programmer reads your resume (comic)</a><li><a href='?id=19'>Installing the Latest Debian on an Ancient Laptop</a><li><a href='?id=83'>Sign here (comic)</a><li><a href='?id=80'>Comment spam defeated at last</a><li><a href='?id=86'>Boring Date (comic)</a><li><a href='?id=71'>Using the Acer Aspire One as a web server</a></ul>