Creating portable binaries on Linux
Distributing applications on Linux is hard. Sure, with modern package management, installing software is easy. But if you are distributing an application, you probably need one Windows version, plus umpteen different versions for Linux. In this article, we'll create a dummy application that targets the following operating systems, which are commonly used in business environments:
- Windows Server 2003
- Windows XP
- Red Hat Enterprise Linux 3
- Red Hat Enterprise Linux 4
- Ubuntu 6.06.2
- Ubuntu 8.04
- Ubuntu 9.10
As evidence that the problem is hard, try downloading Firefox. It fails to start on many of the above platforms, due to missing libraries.
The sample application
The sample application is called plookup (download source and all binaries). It runs from the command line and takes a hostname, looks it up and prints out the IP address. Ignoring the security flaws, it has several monkey wrenches thrown in that make it hard to port to different versions of linux:- It uses C++, which causes headaches when dynamically linking
- It uses socket functions, which cause migraines when statically linking
Will distributing source code solve the problem?
In theory, distributing source code seems to be an easy way to get around the problem, assuming your end user 1) has administrative access, 2) can install a compiler 2) knows how to run a configure script, 3) has the technical knowledge to interpret the output of a configure script and download the appropriate dependencies, 4) has technical expertise to resolve conflicts in library versions.
In other words, it's a completely unreasonable solution for software written for normal human beings.
Building on Windows
With the appropriate build flags, you can produce software on Windows 7 that will run on all versions of windows since Windows 95. By defining WINVER and some other macros, you'll be warned at compile time if you're using a feature that will break your program on earlier versions.
CL /EHsc /Feplookup.exe /DWINVER=0x0400 /D_WIN32 /D_WIN32_WINDOWS=0 /D_WIN32_IE=0x0400 wsock32.lib *.cppWe're done with Windows. On to Linux!
Static linking on Linux FAIL
Static linking has gained an undeserved reputation for being portable. But see what happens when you try to create a statically linked version of plookup:
steve@ubuntu:~/plookup$ g++ -static -static-libgcc -o plookup main.cpp /tmp/ccMhUffR.o: In function `hostlookup(std::basic_string, std::allocator > const&, std::basic_string , std::allocator >&)': main.cpp:(.text+0x15): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
It defeats the purpose. The warning isn't kidding, either. Your app might run fine for months, then all it takes is one update and it will crash. Even if you didn't use any socket functions, you might have other problems.
Note: If you statically link using the GNU compiler, then according to the L-GPL you have to also distribute your object files so the end-user can possibly re-link them to another version of the C libraries that they could have modified. Because your customers love hacking on strcat in their spare time.
Dynamic Linking
I built and tested the sample application on many different platforms. This table summarizes the results of the experiment.
Build on Red Hat EL 3 | Build on Red Hat EL 4 | Build on Ubuntu 6.06.2 g++3.3 | Build on Ubuntu 6.06.2 g++4.0 | Build on Ubuntu 8.04 | Build on Ubuntu 9.10 | |
---|---|---|---|---|---|---|
Runs on Red Hat EL 3? | Yes | No | Yes | No | No | No |
Runs on Red Hat EL 4? | No | Yes | No | Yes | Yes | Yes |
Runs on Ubuntu 6.06.2? | No | Yes | No | Yes | Yes | Yes |
Runs on Ubuntu 8.04? | No | Yes | No | Yes | Yes | Yes |
Runs on Ubuntu 9.10? | No | Yes | No | Yes | Yes | Yes |
Analysis
There are two classes of systems. Those with libstdc++5.0 (Red Hat EL 3), and those with libstdc++6.0 (All others). If you build on a system with one version, your application will only run on systems which have that library.
Ubuntu 6.06.2 is a special case. It does not have libstdc++5.0 by default, but you can add it by installing and building with g++3.3. That makes Ubuntu 6.06.2 a great build environment for portable binaries.
Linux Standards Base
Linux Standards Base (LSB) has an excellent utility that will predict which versions of Linux your application won't run on. But I had some problems using the rest of their toolchain:
- It's not clear what you have to download to use their toolchain.
- It's not clear how to use their toolchain (Hint: It's a wrapper around gcc)
- Their toolchain doesn't work with recent versions of gcc
- Once I finally found a distribution that would work with the lsb tools, it did not produce a portable application.
I spent six hours on LSB one weekend and gave up.
Summary
To distribute binaries on Linux,- Dynamically link the standard libraries
- Produce one binary using g++3.3 for older systems
- Produce another binary using g++4.0 for newer systems
The GNU C/C++/FORTRAN/compiler (gcc, g++, etc) is widely considered to be one of the leading compilers in the world. Its development has recently been taken over by the GCC team. All of the rapid development and near-legendary portability that are the hallmarks of an open-source project are being applied to libstdc++.
If you liked this you'll love:
Basically package and ship libstdc++6.0 for the older distrobutions.
I have been using lsb 4.1 works with modern compliers. patchelf allows you to correct loader if LSB loaders are not on the system and set rpath/rpaths for the .so files you shipped with.
Lsb 4.0 and 4.1 has to be installed straight from the Linuxstandard base sites.
ld.so the loader of all things has a --list feature. Run the loader on your application with list and it will tell you if you need to install libstdc++6.0 to make your program work or not. Of course this means also installing all the other C++ libraries you used.
The loader --list feature is very good thing to do in all install scripts this tells you straight away about a case of missing .so file that is straight opened by the binary. dlopen stuff it does not check.
Also you missed on older copies of windows having to install the complier runtime or the program don't run. Basically my install detect the run-time is missing then has a embedded correction.
"Windows Server 2003 , Windows XP" These are not really that different versions of windows. Windows 2000 to 2003 you would notice more of a issue with runtime being required.
Yes it a common error on windows person built with a newer MS complier and expects the MS runtime installed on every machine for that complier when it might not be.
For closed source programs I get slightly evil and download a package containing libstdc++6.0 and extract the library into the rpath I have set. So the program was not truly shipped with libstdc++6.0 instead informed user and asked user to install it by downloading it.
Basically a little bit of installer script intelligence and you have a portable application built with gcc 4.x and up. So keeping the updates to the complier.
Phil
"This doesn't work with libc (IIRC), but works well with libstdc++ in my experience."
I can tell you why. You would have only replaced the libc part of the glibc package. If you need to ship libc you must ship matching loader. ld.so is annoying part of the glibc package and miss match causes glibc symbols not to be correctly findable. Yes you can get around this with patchelf to point the executable to where you installed the glibc compatible loader to the glibc you wish to use. Remember this is just like replacing C++ now everything C based using the different glibc might or might not work.
Laura LD_LIBRARY_PATH can be used but it has a habit of chaining on to other programs the application runs. rpath inserted into your binaries is the most stable. Since it don't replicate on to effect other programs.
I guess you did not know that the loader could inform you about missing libraries.
I call it the biggest mistake about making a portable binary not having your install system check if all the dependencies are there by loader.
All the tech I am talking about is pre package management tech. Ways to solve this problem is old. Problem is no one has written good documentation using the old techs. Its a step by step method.
Use a static binary to check.
Loader location.
Existence of libraries by loader for install gui(if you have a install gui)
Existence of require libraries by loader.
Have patchelf or equal static built to correct loaders and set rpath on binaries.
Use patchelf or equal to correct install information as required.
static binary can also download any missing parts or alter the rpath to use the shipped fix up platform error parts.
Finally don't be afraid to error out with a down right clear message about any missing critical libraries.
Fairly straight forwards universal install script really.
www.pixelbeat.org/programming/linux_binary_compatibility.html
If you come up with any further information or ideas on this topic, I would love to hear about it. I can build applications on Windows and use them several years later. It would be nice to be able to do the same on Linux and not continually have to recompile multiple programs from source code after major libraries change. Thanks.
This doesn't work with libc (IIRC), but works well with libstdc++ in my experience.