< >

Creating portable binaries on Linux

Posted 14 years ago

Distributing applications on Linux is hard. Sure, with modern package management, installing software is easy. But if you are distributing an application, you probably need one Windows version, plus umpteen different versions for Linux. In this article, we'll create a dummy application that targets the following operating systems, which are commonly used in business environments:

Windows Server 2003
Windows XP
Red Hat Enterprise Linux 3
Red Hat Enterprise Linux 4
Ubuntu 6.06.2
Ubuntu 8.04
Ubuntu 9.10

As evidence that the problem is hard, try downloading Firefox. It fails to start on many of the above platforms, due to missing libraries.

The sample application

The sample application is called plookup (download source and all binaries). It runs from the command line and takes a hostname, looks it up and prints out the IP address. Ignoring the security flaws, it has several monkey wrenches thrown in that make it hard to port to different versions of linux:

It uses C++, which causes headaches when dynamically linking
It uses socket functions, which cause migraines when statically linking

Will distributing source code solve the problem?

In theory, distributing source code seems to be an easy way to get around the problem, assuming your end user 1) has administrative access, 2) can install a compiler 2) knows how to run a configure script, 3) has the technical knowledge to interpret the output of a configure script and download the appropriate dependencies, 4) has technical expertise to resolve conflicts in library versions.

In other words, it's a completely unreasonable solution for software written for normal human beings.

Building on Windows

With the appropriate build flags, you can produce software on Windows 7 that will run on all versions of windows since Windows 95. By defining WINVER and some other macros, you'll be warned at compile time if you're using a feature that will break your program on earlier versions.

CL /EHsc /Feplookup.exe /DWINVER=0x0400 /D_WIN32 /D_WIN32_WINDOWS=0 /D_WIN32_IE=0x0400 wsock32.lib *.cpp

We're done with Windows. On to Linux!

Static linking on Linux FAIL

Static linking has gained an undeserved reputation for being portable. But see what happens when you try to create a statically linked version of plookup:

steve@ubuntu:~/plookup$ g++ -static -static-libgcc -o plookup main.cpp
/tmp/ccMhUffR.o: In function `hostlookup(std::basic_string, std::allocator > const&, std::basic_string, std::allocator >&)':
main.cpp:(.text+0x15): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

It defeats the purpose. The warning isn't kidding, either. Your app might run fine for months, then all it takes is one update and it will crash. Even if you didn't use any socket functions, you might have other problems.

Note: If you statically link using the GNU compiler, then according to the L-GPL you have to also distribute your object files so the end-user can possibly re-link them to another version of the C libraries that they could have modified. Because your customers love hacking on strcat in their spare time.

Dynamic Linking

I built and tested the sample application on many different platforms. This table summarizes the results of the experiment.

	Build on Red Hat EL 3	Build on Red Hat EL 4	Build on Ubuntu 6.06.2 g++3.3	Build on Ubuntu 6.06.2 g++4.0	Build on Ubuntu 8.04	Build on Ubuntu 9.10
Runs on Red Hat EL 3?	Yes	No	Yes	No	No	No
Runs on Red Hat EL 4?	No	Yes	No	Yes	Yes	Yes
Runs on Ubuntu 6.06.2?	No	Yes	No	Yes	Yes	Yes
Runs on Ubuntu 8.04?	No	Yes	No	Yes	Yes	Yes
Runs on Ubuntu 9.10?	No	Yes	No	Yes	Yes	Yes

Analysis

There are two classes of systems. Those with libstdc++5.0 (Red Hat EL 3), and those with libstdc++6.0 (All others). If you build on a system with one version, your application will only run on systems which have that library.

Ubuntu 6.06.2 is a special case. It does not have libstdc++5.0 by default, but you can add it by installing and building with g++3.3. That makes Ubuntu 6.06.2 a great build environment for portable binaries.

Linux Standards Base

Linux Standards Base (LSB) has an excellent utility that will predict which versions of Linux your application won't run on. But I had some problems using the rest of their toolchain:

It's not clear what you have to download to use their toolchain.
It's not clear how to use their toolchain (Hint: It's a wrapper around gcc)
Their toolchain doesn't work with recent versions of gcc
Once I finally found a distribution that would work with the lsb tools, it did not produce a portable application.

I spent six hours on LSB one weekend and gave up.

Summary

To distribute binaries on Linux,

Dynamically link the standard libraries
Produce one binary using g++3.3 for older systems
Produce another binary using g++4.0 for newer systems

Hopefully in time the situation will improve. According to the FAQ on libstdc++ 3:

The GNU C/C++/FORTRAN/ compiler (gcc, g++, etc) is widely considered to be one of the leading compilers in the world. Its development has recently been taken over by the GCC team. All of the rapid development and near-legendary portability that are the hallmarks of an open-source project are being applied to libstdc++.

If you liked this you'll love:

Steve Hanov makes a living working on Rhymebrain.com, rapt.ink, www.websequencediagrams.com, and Zwibbler.com. He lives in Waterloo, Canada.

Post comment

edit

oiaohm

twelve years ago

You have missed a solution patchelf and rpath(loader feature mostly forgotten about).

Basically package and ship libstdc++6.0 for the older distrobutions.

I have been using lsb 4.1 works with modern compliers. patchelf allows you to correct loader if LSB loaders are not on the system and set rpath/rpaths for the .so files you shipped with.

Lsb 4.0 and 4.1 has to be installed straight from the Linuxstandard base sites.

ld.so the loader of all things has a --list feature. Run the loader on your application with list and it will tell you if you need to install libstdc++6.0 to make your program work or not. Of course this means also installing all the other C++ libraries you used.

The loader --list feature is very good thing to do in all install scripts this tells you straight away about a case of missing .so file that is straight opened by the binary. dlopen stuff it does not check.

Also you missed on older copies of windows having to install the complier runtime or the program don't run. Basically my install detect the run-time is missing then has a embedded correction.

"Windows Server 2003 , Windows XP" These are not really that different versions of windows. Windows 2000 to 2003 you would notice more of a issue with runtime being required.

Yes it a common error on windows person built with a newer MS complier and expects the MS runtime installed on every machine for that complier when it might not be.

For closed source programs I get slightly evil and download a package containing libstdc++6.0 and extract the library into the rpath I have set. So the program was not truly shipped with libstdc++6.0 instead informed user and asked user to install it by downloading it.

Basically a little bit of installer script intelligence and you have a portable application built with gcc 4.x and up. So keeping the updates to the complier.

Phil

"This doesn't work with libc (IIRC), but works well with libstdc++ in my experience."

I can tell you why. You would have only replaced the libc part of the glibc package. If you need to ship libc you must ship matching loader. ld.so is annoying part of the glibc package and miss match causes glibc symbols not to be correctly findable. Yes you can get around this with patchelf to point the executable to where you installed the glibc compatible loader to the glibc you wish to use. Remember this is just like replacing C++ now everything C based using the different glibc might or might not work.

Laura LD_LIBRARY_PATH can be used but it has a habit of chaining on to other programs the application runs. rpath inserted into your binaries is the most stable. Since it don't replicate on to effect other programs.

I guess you did not know that the loader could inform you about missing libraries.

I call it the biggest mistake about making a portable binary not having your install system check if all the dependencies are there by loader.

All the tech I am talking about is pre package management tech. Ways to solve this problem is old. Problem is no one has written good documentation using the old techs. Its a step by step method.

Use a static binary to check.

Loader location.

Existence of libraries by loader for install gui(if you have a install gui)

Existence of require libraries by loader.

Have patchelf or equal static built to correct loaders and set rpath on binaries.

Use patchelf or equal to correct install information as required.

static binary can also download any missing parts or alter the rpath to use the shipped fix up platform error parts.

Finally don't be afraid to error out with a down right clear message about any missing critical libraries.

Fairly straight forwards universal install script really.

edit

PÃ¡draig Brady

13 years ago

My binary compat notes:

www.pixelbeat.org/programming/linux_binary_compatibility.html

edit

Laura

13 years ago

I've been doing a lot of research on this subject, more for reasons of backward compatibilty than portabilty. You have very succinctly described some of the difficulties in this area. Just wish there were more solutions. I've considered building with another compiler that doesn't have as many issues with distributing the runtime environment (c libraries). Some of the compilers I looked at were tcc and OpenWatcom. Thanks for the information on lsb. That was also on my list of possible options too. One could use other C libraries with gcc that are better at statically linking, possibly eglibc. The Stali project mentions static linking and using C libraries like bionic (BSD licensed), but it doesn't sound ready for general use yet. As someone wrote below, there's a technique of using $ORIGIN and RPATH to localize some of the shared libraries. PC-BSD uses a similar method but creates packages that change the LD_LIBRARY_PATH instead of building each program to look for libraries locally. Portablelinuxapps uses something like a virtual file system and includes all the dependencies in a package.

If you come up with any further information or ideas on this topic, I would love to hear about it. I can build applications on Windows and use them several years later. It would be nice to be able to do the same on Linux and not continually have to recompile multiple programs from source code after major libraries change. Thanks.

edit

Tim

13 years ago

The autopackage project had solutions for some of these problems. Google for apbuild. They did things like have a wrapper around gcc that included a special header to avoid depending on newer glibc symbols.

edit

Phil

14 years ago

An alternative: build wherever you like & set the RPATH in the binary to $ORIGIN. Then you can ship the extra libraries you need with the binary & the end user can drop them if they want to & use the system ones.

This doesn't work with libc (IIRC), but works well with libstdc++ in my experience.

edit

Steve Hanov

14 years ago

I'd love to use dietlibc, but it's GPL and "other licensing terms (e.g. for commercial projects) can be negotiated for substantial contributors or project sponsors." It doesn't sound cheap.

edit

acc

14 years ago

I've also tried to use dietlibc (it also wraps gcc, you invoke it like diet gcc) but it won't work with all sources, just with the ones which didn't use too much of glibc. Yes shared libraries are real *pain* for GNU/Linux. I'd like to hear more experiences...

UMA Questions Answered

A bunch of questions answered about UMA wireless technology.

Finding great ideas for your startup

"I just don't have any ideas." This is the #1 stumbling block for budding entrepreneurs. Here are a few techniques to get the creative juices flowing.

barcamp (comic)

Spoke.com scam

Rant: Why do companies think they can make money by posting false information about you on the Internet?

Cell Phones on Airplanes

Much ink has been spilled about the use of cell phones on airplanes. Here's the truth, which will be disappointing to conspiracy theorists: Cell phone signals most definately have an effect on other electronic equipment. Read on for more.

UMA's dirty secrets

Recently, many carriers have started offering UMA, or WiFi phones. These are cell phones with WiFi capabilites. Don't be fooled -- you won't be able to get free calls and run skype on them. The UMA technology is meant to extend the carrier's cellular network into your home using your broadband internet connection.

Blame the extensions (comic)

VP trees: A data structure for finding stuff fast

Let's say you have millions of pictures of faces tagged with names. Given a new photo, how do you find the name of person that the photo most resembles?

In the cases I mentioned, each record has hundreds or thousands of elements: the pixels in a photo, or patterns in a sound snippet, or web usage data. These records can be regarded as points in high dimensional space. When you look at a points in space, they tend to form clusters, and you can infer a lot by looking at ones nearby.

Finding the top K items in a list efficiently

Do you use sort() to find the top results? Here's a simple trick that will make your software run much faster.

Cross-domain communication the HTML5 way

Making a web application mashable -- useable in another web page -- has some challenges in the area of cross-domain communications. Here is how I solved those problems for Zwibbler.com, using HTML5 cross domain communication.