I've been installing Ubuntu exclusively on USB memory sticks, because then if Canonical suddenly decides I don't need frivolous things like sound or wifi to work properly, I can just pop in another installation and keep going. A 4GB USB drive is plenty of space, if you remove the all the junk that comes with the default Ubuntu Desktop installation.
I went through the entire package listing in Ubuntu 8.10 and figured out which packages were useless space-hogs. It is utterly astounding that for every package installed by default, someone at Canonical decided it would be more useful than VLC Media Player.
Here is a magic command that will immediately save 500 MB of disk space. This is customized to my own usage, and your's may be different. In particular, I don't use openoffice or bluetooth, and I don't read Thai or Chinese.
sudo apt-get autoremove gnome-games-data thunderbird openoffice.org-help-en-us ttf-arphic-uming pidgin-data ttf-unfonts-core seahorse seahorse-plugins libsane ttf-kochi-gothic brasero totem vinagre ttf-thai-tlwg yelp listen bluez catfish xchat bluez-gnome xsane-common xchat-common
The "autoremove" part says that it will remove these packages, and all others that depend on them as well. For example, by listing "gnome-games-data", it will automatically remove gnome-games as well.
Start synaptic package manager. Select "Filter" from the "Settings Menu". Create a new filter, and call it "installed". Deselect all options, and then put a check mark beside "Installed". Click OK
Select "Preferences" from the "Settings Menu". Under the "Columns and Fonts" tab, make sure "Installed size" has a checkmark beside it.
Now you can get a listing of the most bloated packages, sorted by size. Go through them and "Mark for Complete Removal" any applications that you don't use. If in doubt, leave it in. If it prompts you, don't be afraid of removing "ubuntu-desktop". This is just an empty "meta-package", and its absence won't affect your system. However, you should avoid removing anything that would also remove "gnome-panel".
Or you can record their name and make a command that you can use on other systems, as I have above.