Hate UML?

Draw sequence diagrams in seconds.
http://www.websequencediagrams.com

Creating portable binaries on Linux
Posted on: 2010-04-29 18:55:31

Distributing applications on Linux is hard. Sure, with modern package management, installing software is easy. But if you are distributing an application, you probably need one Windows version, plus umpteen different versions for Linux. In this article, we'll create a dummy application that targets the following operating systems, which are commonly used in business environments:

  • Windows Server 2003
  • Windows XP
  • Red Hat Enterprise Linux 3
  • Red Hat Enterprise Linux 4
  • Ubuntu 6.06.2
  • Ubuntu 8.04
  • Ubuntu 9.10

As evidence that the problem is hard, try downloading Firefox. It fails to start on many of the above platforms, due to missing libraries.

The sample application

The sample application is called plookup (download source and all binaries). It runs from the command line and takes a hostname, looks it up and prints out the IP address. Ignoring the security flaws, it has several monkey wrenches thrown in that make it hard to port to different versions of linux:
  • It uses C++, which causes headaches when dynamically linking
  • It uses socket functions, which cause migraines when statically linking

Will distributing source code solve the problem?

In theory, distributing source code seems to be an easy way to get around the problem, assuming your end user 1) has administrative access, 2) can install a compiler 2) knows how to run a configure script, 3) has the technical knowledge to interpret the output of a configure script and download the appropriate dependencies, 4) has technical expertise to resolve conflicts in library versions.

In other words, it's a completely unreasonable solution for software written for normal human beings.

Building on Windows

With the appropriate build flags, you can produce software on Windows 7 that will run on all versions of windows since Windows 95. By defining WINVER and some other macros, you'll be warned at compile time if you're using a feature that will break your program on earlier versions.

CL /EHsc /Feplookup.exe /DWINVER=0x0400 /D_WIN32 /D_WIN32_WINDOWS=0 /D_WIN32_IE=0x0400 wsock32.lib *.cpp
We're done with Windows. On to Linux!

Static linking on Linux FAIL

Static linking has gained an undeserved reputation for being portable. But see what happens when you try to create a statically linked version of plookup:

steve@ubuntu:~/plookup$ g++ -static -static-libgcc -o plookup main.cpp
/tmp/ccMhUffR.o: In function `hostlookup(std::basic_string, std::allocator > const&, std::basic_string, std::allocator >&)':
main.cpp:(.text+0x15): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

It defeats the purpose. The warning isn't kidding, either. Your app might run fine for months, then all it takes is one update and it will crash. Even if you didn't use any socket functions, you might have other problems.

Note: If you statically link using the GNU compiler, then according to the L-GPL you have to also distribute your object files so the end-user can possibly re-link them to another version of the C libraries that they could have modified. Because your customers love hacking on strcat in their spare time.

Dynamic Linking

I built and tested the sample application on many different platforms. This table summarizes the results of the experiment.

Build on Red Hat EL 3 Build on Red Hat EL 4 Build on Ubuntu 6.06.2 g++3.3 Build on Ubuntu 6.06.2 g++4.0 Build on Ubuntu 8.04 Build on Ubuntu 9.10
Runs on Red Hat EL 3? Yes No Yes No No No
Runs on Red Hat EL 4? No Yes No Yes Yes Yes
Runs on Ubuntu 6.06.2? No Yes No Yes Yes Yes
Runs on Ubuntu 8.04? No Yes No Yes Yes Yes
Runs on Ubuntu 9.10? No Yes No Yes Yes Yes

Analysis

There are two classes of systems. Those with libstdc++5.0 (Red Hat EL 3), and those with libstdc++6.0 (All others). If you build on a system with one version, your application will only run on systems which have that library.

Ubuntu 6.06.2 is a special case. It does not have libstdc++5.0 by default, but you can add it by installing and building with g++3.3. That makes Ubuntu 6.06.2 a great build environment for portable binaries.

Linux Standards Base

Linux Standards Base (LSB) has an excellent utility that will predict which versions of Linux your application won't run on. But I had some problems using the rest of their toolchain:

  1. It's not clear what you have to download to use their toolchain.
  2. It's not clear how to use their toolchain (Hint: It's a wrapper around gcc)
  3. Their toolchain doesn't work with recent versions of gcc
  4. Once I finally found a distribution that would work with the lsb tools, it did not produce a portable application.

I spent six hours on LSB one weekend and gave up.

Summary

To distribute binaries on Linux,
  • Dynamically link the standard libraries
  • Produce one binary using g++3.3 for older systems
  • Produce another binary using g++4.0 for newer systems
Hopefully in time the situation will improve. According to the FAQ on libstdc++ 3:
The GNU C/C++/FORTRAN/ compiler (gcc, g++, etc) is widely considered to be one of the leading compilers in the world. Its development has recently been taken over by the GCC team. All of the rapid development and near-legendary portability that are the hallmarks of an open-source project are being applied to libstdc++.

If you liked this you'll love:

Want more programming tech talk?
Add to Circles on Google Plus
Subscribe to posts

Post comment

Real Name:
Your Email (Not displayed):

Text only. No HTML. If you write "http:" your message will be ignored.
Choose an edit password if you want to be able to edit or delete your comment later.
Editing Password (Optional):

acc

2010-04-30 14:20:04
I've also tried to use dietlibc (it also wraps gcc, you invoke it like diet gcc) but it won't work with all sources, just with the ones which didn't use too much of glibc. Yes shared libraries are real *pain* for GNU/Linux. I'd like to hear more experiences...

Steve Hanov

2010-05-01 14:04:54
I'd love to use dietlibc, but it's GPL and "other licensing terms (e.g. for commercial projects) can be negotiated for substantial contributors or project sponsors." It doesn't sound cheap.

Phil

2010-09-14 04:31:07
An alternative: build wherever you like & set the RPATH in the binary to $ORIGIN. Then you can ship the extra libraries you need with the binary & the end user can drop them if they want to & use the system ones.

This doesn't work with libc (IIRC), but works well with libstdc++ in my experience.

Tim

2011-02-13 12:25:39
The autopackage project had solutions for some of these problems. Google for apbuild. They did things like have a wrapper around gcc that included a special header to avoid depending on newer glibc symbols.

Laura

2011-05-06 10:11:20
I've been doing a lot of research on this subject, more for reasons of backward compatibilty than portabilty. You have very succinctly described some of the difficulties in this area. Just wish there were more solutions. I've considered building with another compiler that doesn't have as many issues with distributing the runtime environment (c libraries). Some of the compilers I looked at were tcc and OpenWatcom. Thanks for the information on lsb. That was also on my list of possible options too. One could use other C libraries with gcc that are better at statically linking, possibly eglibc. The Stali project mentions static linking and using C libraries like bionic (BSD licensed), but it doesn't sound ready for general use yet. As someone wrote below, there's a technique of using $ORIGIN and RPATH to localize some of the shared libraries. PC-BSD uses a similar method but creates packages that change the LD_LIBRARY_PATH instead of building each program to look for libraries locally. Portablelinuxapps uses something like a virtual file system and includes all the dependencies in a package.

If you come up with any further information or ideas on this topic, I would love to hear about it. I can build applications on Windows and use them several years later. It would be nice to be able to do the same on Linux and not continually have to recompile multiple programs from source code after major libraries change. Thanks.

Pádraig Brady

2011-09-12 08:36:03
My binary compat notes:

www.pixelbeat.org/programming/linux_binary_compatibility.html

oiaohm

2012-01-23 04:19:17
You have missed a solution patchelf and rpath(loader feature mostly forgotten about).

Basically package and ship libstdc++6.0 for the older distrobutions.

I have been using lsb 4.1 works with modern compliers. patchelf allows you to correct loader if LSB loaders are not on the system and set rpath/rpaths for the .so files you shipped with.

Lsb 4.0 and 4.1 has to be installed straight from the Linuxstandard base sites.

ld.so the loader of all things has a --list feature. Run the loader on your application with list and it will tell you if you need to install libstdc++6.0 to make your program work or not. Of course this means also installing all the other C++ libraries you used.

The loader --list feature is very good thing to do in all install scripts this tells you straight away about a case of missing .so file that is straight opened by the binary. dlopen stuff it does not check.

Also you missed on older copies of windows having to install the complier runtime or the program don't run. Basically my install detect the run-time is missing then has a embedded correction.

"Windows Server 2003 , Windows XP" These are not really that different versions of windows. Windows 2000 to 2003 you would notice more of a issue with runtime being required.

Yes it a common error on windows person built with a newer MS complier and expects the MS runtime installed on every machine for that complier when it might not be.

For closed source programs I get slightly evil and download a package containing libstdc++6.0 and extract the library into the rpath I have set. So the program was not truly shipped with libstdc++6.0 instead informed user and asked user to install it by downloading it.

Basically a little bit of installer script intelligence and you have a portable application built with gcc 4.x and up. So keeping the updates to the complier.

Phil

"This doesn't work with libc (IIRC), but works well with libstdc++ in my experience."

I can tell you why. You would have only replaced the libc part of the glibc package. If you need to ship libc you must ship matching loader. ld.so is annoying part of the glibc package and miss match causes glibc symbols not to be correctly findable. Yes you can get around this with patchelf to point the executable to where you installed the glibc compatible loader to the glibc you wish to use. Remember this is just like replacing C++ now everything C based using the different glibc might or might not work.

Laura LD_LIBRARY_PATH can be used but it has a habit of chaining on to other programs the application runs. rpath inserted into your binaries is the most stable. Since it don't replicate on to effect other programs.

I guess you did not know that the loader could inform you about missing libraries.

I call it the biggest mistake about making a portable binary not having your install system check if all the dependencies are there by loader.

All the tech I am talking about is pre package management tech. Ways to solve this problem is old. Problem is no one has written good documentation using the old techs. Its a step by step method.

Use a static binary to check.

Loader location.

Existence of libraries by loader for install gui(if you have a install gui)

Existence of require libraries by loader.

Have patchelf or equal static built to correct loaders and set rpath on binaries.

Use patchelf or equal to correct install information as required.

static binary can also download any missing parts or alter the rpath to use the shipped fix up platform error parts.

Finally don't be afraid to error out with a down right clear message about any missing critical libraries.

Fairly straight forwards universal install script really.

Email
steve.hanov@gmail.com

Other posts by Steve

Yes, You Absolutely Might Possibly Need an EIN to Sell Software to the US How Asana Breaks the Rules About Per-Seat Pricing 5 Ways PowToon Made Me Want to Buy Their Software How I run my business selling software to Americans 0, 1, Many, a Zillion Give your Commodore 64 new life with an SD card reader 20 lines of code that will beat A/B testing every time [comic] Appreciation of xkcd comics vs. technical ability VP trees: A data structure for finding stuff fast Why you should go to the Business of Software Conference Next Year Four ways of handling asynchronous operations in node.js Type-checked CoffeeScript with jzbuild Zero load time file formats Finding the top K items in a list efficiently An instant rhyming dictionary for any web site Succinct Data Structures: Cramming 80,000 words into a Javascript file. Throw away the keys: Easy, Minimal Perfect Hashing Why don't web browsers do this? Fun with Colour Difference Compressing dictionaries with a DAWG Fast and Easy Levenshtein distance using a Trie The Curious Complexity of Being Turned On Cross-domain communication the HTML5 way Five essential steps to prepare for your next programming interview Minimal usable Ubuntu with one command Finding awesome developers in programming interviews Compress your JSON with automatic type extraction JZBUILD - An Easy Javascript Build System Pssst! Want to stream your videos to your iPod? "This is stupid. Your program doesn't work," my wife told me The simple and obvious way to walk through a graph Asking users for steps to reproduce bugs, and other dumb ideas Creating portable binaries on Linux Bending over: How to sell your software to large companies Regular Expression Matching can be Ugly and Slow C++: A language for next generation web apps qb.js: An implementation of QBASIC in Javascript Zwibbler: A simple drawing program using Javascript and Canvas You don't need a project/solution to use the VC++ debugger Boring Date (comic) barcamp (comic) How IE <canvas> tag emulation works I didn't know you could mix and match (comic) Sign here (comic) It's a dirty job... (comic) The PenIsland Problem: Text-to-speech for domain names Pitching to VCs #2 (comic) Building a better rhyming dictionary Does Android team with eccentric geeks? (comic) Comment spam defeated at last Pitching to VCs (comic) How QBASIC almost got me killed Blame the extensions (comic) How to run a linux based home web server Microsoft's generosity knows no end for a year (comic) Using the Acer Aspire One as a web server When programmers design web sites (comic) Finding great ideas for your startup Game Theory, Salary Negotiation, and Programmers Coding tips they don't teach you in school When a reporter mangles your elevator pitch Test Driven Development without Tears Drawing Graphs with Physics Free up disk space in Ubuntu Keeping Abreast of Pornographic Research in Computer Science Exploiting perceptual colour difference for edge detection Experiment: Deleting a post from the Internet Is 2009 the year of Linux malware? Email Etiquette How a programmer reads your resume (comic) How wide should you make your web page? Usability Nightmare: Xfce Settings Manager cairo blur image surface Automatically remove wordiness from your writing Why Perforce is more scalable than Git Optimizing Ubuntu to run from a USB key or SD card UMA Questions Answered Make Windows XP look like Ubuntu, with Spinning Cube Effect See sound without drugs Standby Preventer Stock Picking using Python Spoke.com scam Stackoverflow.com Copy a cairo surface to the windows clipboard Simulating freehand drawing with Cairo Free, Raw Stock Data Installing Ubuntu on the Via Artigo Why are all my lines fuzzy in cairo? A simple command line calculator Tool for Creating UML Sequence Diagrams Exploring sound with Wavelets UMA and free long distance UMA's dirty secrets Installing the Latest Debian on an Ancient Laptop Dissecting Adsense HTML/ Javascript/ CSS Pretty Printer Web Comic Aggregator Experiments in making money online How much cash do celebrities make? Draw waveforms and hear them Cell Phones on Airplanes Detecting C++ memory leaks What does your phone number spell? A Rhyming Engine Rules for Effective C++ Cell Phone Secrets