Stubborn Tech Problem Solving

Diary and notebook of whatever tech problems are irritating me at the moment.


Steam client missing images on Linux

If your Steam client is missing images (broken image icon) in the store or other pages then try this fix.  First exit the Steam client.  Then delete the HTML cache directory at ~/.steam/config/htmlcache.  Then start the client and browse to the problem page again.  The missing images should be reloaded from the server and display correctly.

The tilde "~" is terminal shorthand for your home directory but you can use any file manager instead of a terminal.  A file or directory that starts with a period like ".steam" makes it a hidden directory so you'll have to configure your file manager to show them (Ctrl-H toggles the setting in most file managers).


Workarounds for some game bit-rot on [A-Z]*Ubuntu 14.04

After upgrading to Xubuntu 14.04 x86_64 several games failed to work including Torchlight (from Humble Indie Bundle 6), Doom3, Quake 4, and Aquaria.  I had expected this because bit-rot is a common problem with closed-source games when upgrading.

Aquaria and Doom3 segfaulted.  Quake 4 failed to load textures and its log reported problems.  These all had the same cause - the libs they included were incompatible with the newer libs being loaded automatically from the OS.  They also had the same solution - rename the included libs. On Quake 4 that included,, and  To determine where a dynamic binary is getting libraries from use the ldd command but only change those necessary to get it working else other compatibility problems may occur.

Torchlight would load but the cursor had a black box around it and none of the menu buttons could be clicked.  The only way out of it was Alt-F4.  Past experience had taught me that libSDL is a common culprit.  It was using version 2, the same that Ubuntu 12.04/Linux Mint 13 supplied, but the game worked without problems on them.  I suspected that changes to the lib since (or its dependencies) then had created an incompatibility so I simply copied the lib from 12.04 to the 14.04 system and put it in /usr/local/games/Torchlight/lib64.  The game started without problems.

Using different library versions works if they haven't changed much.  The better solution is recompiling the game with newer libraries but getting developers to do that with old closed-source games is difficult.  While fixing library problems is not n00b-friendly, it's trivial compared to what is required for getting old Loki Software games working.


Wgetting to the correct file name

One of the annoyances of using wget in scripts is the problem of files from some web sites being named "index.html" instead of the actual file name.  This is due to the use of the content-disposition header field (part of MIME) to indicate the file name instead of the URL.  Support for this field by wget is incomplete and disabled by default.

There are a few ways around this.  The easiest is to tell wget to look for it with the --content-disposition option.

Another option is to use curl --remote-header-name or aria2 instead of wget.

To easily create commands for more complicated file retrieval (with cookies, etc.) the Firefox add-on cliget can provide a command line for curl, wget, and aria2 in the opening download window.  It only shows the curl command line by default.  You can enable the others in cliget preferences (available in the cliget entry in about:addons).


The Heartbleed Bug and You

TL;DR: Most web sites have been leaking passwords due to a bug for the past two years.  Immediately change your Internet passwords for Email and financial accounts, even if you changed your passwords April 7, 2014 or in the days preceding.  Don't use the same password for all sites because if one is compromised, they all are even if some sites were not affected by the bug.  Use a password manager such as KeePass/KeePassX or Password Safe to keep track of them.  Don't click on links in emails, especially any that seem to come from your bank asking to change your password.  Go to your bank's web site directly by entering the address in your web browser manually.

Verbose Version
You may have seen reports about the Heartbleed Internet bug in the news lately.  Note that the term "bug" doesn't mean malware or a virus, just an unintentional flaw.  This particular bug is a vulnerability in the encryption system used by about 2/3rds of the Internet web sites.  Ignoring the hype, on a scale of 1 (ignore) to 10 (critical) this is an 11.  Read this through entirely before panicking.

The bug allows an attacker to potentially obtain passwords from web sites or the credentials of the web site itself (possibly for creating a fake web site that validates as authentic).  Most web server operators can detect if they have the vulnerablity but not if someone exploited it.  While there are lists of affected sites and test services that can tell if a web site is affected, there are so many that it's best to assume most sites you have passwords on were exploited.  Microsoft's web sites were NOT affected since they have their own encryption system, nor were web sites operating with their software (not that they haven't had many other security problems).  But it's difficult for web users to determine what software a site is using now or what was used previously.  The software you are using on your computer, tablet, or phone doesn't matter since the vulnerability affects the servers you connect to.

What to do?  Immediately change your passwords.  You should change them periodically anyways, at least once a year depending on how complicated and long they are.  Log into your account there and look for a password change option.  Sometimes this option is on the account's "profile" or "preferences" page.

If you change your password on a site that is still affected the benefit will be temporary since an attacker can just obtain it again.  Many of the affected web sites have been fixed in the past two days but even those that were safe at the time the bug was found may have already been exploited if they were vulnerable any time in the past two years when the bug was introduced.

You can expect to start receiving notices from major companies about the need to change your passwords.  Don't wait for them to contact you.  Some web site operators may underestimate the risk or be reluctant to announce it due to liability concerns.

Because of the size of the problem you can also expect fake emails (phishing) prompting you to change your passwords.  These are attempts to get your current passwords.  To avoid being caught by these never click on a link in an email.  Always go to the web site directly by entering the web site address into the web browser's address bar or by searching for the company by name using Google.

I suggest prioritizing password changes as follows.  Some web sites and vendor names are listed as examples but not all of them had the Heartbleed vulnerability.

1. Email (Google/Gmail, Yahoo!).  Do these first since they can be used to reset passwords on most other web site accounts.  If your email provider has other services (Google+, Google Docs, Yahoo! Groups) then changing your email password usually changes it for everything else from them.
Google password change:
Yahoo! password change:

2. Financial (banks, credit unions, investments, IRAs, pensions, PayPal)
3. Government (, passports, social security, taxes)
4. Password services (a.k.a. "Single Sign-On").  If you are using a service that controls access to multiple web sites (OpenID, Facebook, Google), then change that too since a breach with it can affect multiple sites.  Usually these aren't used for financial and government sites so the potential damage is less but they affect many other web sites.
5. Any site that stores your credit card or financial information ( and accounting sites like
6. Vendors (insurance, medical) and utilities (power, phone, Internet) that have your Social Security number or you have set up for automatic bill payments.
7. Any site that makes payments to you (eBay, Etsy,
8. Computer remote backup services.  These usually don't have the encryption key for reading the contents of your backup data but someone with your password may be able to delete your backups.  If your encryption key was weak (short and simple), an attacker that obtains your backup data via your password may be able to break the encryption key by trying every possible key combination.
9. File hosting services (Dropbox, MediaFire, Google Drive).  An attacker can install malware within the hosted files and spread it to everyone that accesses them.
10. Social networking sites (Twitter, Linked-In, Reddit, dating) and picture hosting sites (Imgur, Panaramio, Photobucket) are less important but worth changing if you have information on them that you want to keep private.
11. Passwords for blogs and web sites you have made should be changed to prevent them from being used to host malware.

Vendors that don't keep your credit card info (i.e., you have to enter it every time you buy something) are less of a risk but you should consider changing those of vendors you depend on the most and keep an eye on your credit card statements.

Passwords used for starting your computer or unlocking your phone are probably not affected.

Some wireless Internet sharing boxes (routers) you may have in your home or business have built-in web servers for configring them and many are affected by the Heartbleed vulnerability.  Fortunately the configuration web site ususally can't be accessed by people on the Internet unless the device is intentionally configured to allow remote access.  Unfortunately manufacturers aren't likely to fix any that have the problem.

Some security systems and home automation systems also have web access and can be affected.  If you have a system that you can control from a web browser (on your computer or phone), then contact the manufacturer or installer to verify it and get it fixed if necessary.

You should not use the same password for multiple sites.  If one is compromised, then they all are regardless of which has the Heartbleed bug. Create unique passwords for each and use a password manager program to keep track of them.  With a password manager you create a single (preferably long and complicated) master password you can remember.  This is used to encrypt its password list.  You then use the manager to create complicated random passwords for each web site, preferably random letters and numbers at least 12 characters long.  Many web sites accept much longer passwords and typographical characters (@$%*&...) but not all do.  You don't have to remember these, just copy and paste them from the password manager as needed.

Many web browsers also have password managers built-in and prompt you to store passwords when you enter them.  But these password lists are often not well encrypted which makes it easier for an attacker (or other users of your computer) to obtain them.  In addition,  if you decide to change web browsers, these password lists can be difficult to transfer.  Standalone password managers avoid this problem and can also be used for other codes and non-Internet passwords.

Two popular free password managers for desktop and laptop computers are KeePass and Password Safe.   I use KeePassX on Linux and it's installed on every system I build (in the menu: Applications > Accessories > KeePassX).  It's compatible with KeePass on Windows.  It's available in the repositories of most Linux distributions (Ubuntu, Linux Mint, etc.)

Obviously the password manager will now be the center of your Internet life so it's important to make a backup of its password file, perhaps to a Flash "thumb" drive, and keep it in a safe place.  To limit loss in case your computer is burned up in a house fire, keep the backup in a different location like in the house of a relative, your car or a safe deposit box.  And don't forget its master password!

Changing many passwords is a pain but don't ignore this critical security problem.  The damage is already done and all you can do is prevent further damage to your Internet accounts.

You can panic now.


Cleaning up the Ubuntu and Linux Mint game menus

Most kids love games.  Thanks to F/OSS developers, Humble Bundle, Desura, Steam, and a variety of cross-platform Kickstarter and Indiegogo projects they now have many to choose from with more on the way.

But when I set up a system and start loading dozens of games the desktop menus become quite a mess.  Some games are not listed, or are listed in only in the main "Games" menu but not the correct sub-menus, or are listed in multiple sub-menus, or missing icons, or fail to function.

This is on top of the usual problems with broken packages, bitrot with old closed-source games (Loki titles in particular), conflicts with PulseAudio, and a lack of documentation combined with unusual keyboard key assignments.  The latter is really annoying when a game launches full screen and the user is unable to figure out how to exit it.

Obviously some problems are much more difficult to resolve than others but the desktop menus are fairly easy to fix.  I have fixed dozens and you can do the same.  The link to the archive with my changes is at the bottom of this post but before you get into that you need to understand how the menu system works.

Originally all window managers and desktop environments used their own menu configuration files.  One legacy menu structure is located in /usr/share/applnk.  Debian created a standard for their distro where a menu file was created once in /usr/share/menu and then exported dynamically as needed for each menu system.  Later, (formerly named the X Desktop Group) developed a new standard that has been adopted by Gnome, KDE, Xfce, and others.

With the fdo/XDG standard the menu entries are created as INI files according to the Desktop Menu Specification (*.menu) and Desktop Entry Specification (*.desktop).  The latter standard also defines how to specify the names and icons of the menus themselves (*.directory).  Most desktop environments follow v1.0 of the standard but a v1.1 draft is in development.  The *.menu files are located in /etc/xdg/menus.  The *.desktop files are located in /usr/share/applications or /usr/local/share/applications.  The *.directory files are located in /usr/share/desktop-directories or /usr/local/share/desktop-directories.

A note about /usr, /usr/local, and /opt.  Their structure is defined by the Filesystem Hierarchy Standard. The first two follow the traditional *nix structure where binaries, libraries, documentation, and support files are organized into specific subdirectories by type and may be shared with other programs (especially libraries).  Files for programs in /opt are organized by provider and are generally contained entirely within their own subdirectory and not shared with other programs.  In practice, /usr is where files "managed" by a package manager (apt) are located.  Unmanaged files are in /usr/local and are usually not registered with the package manger.  Files in /opt may be managed or unmanaged depending on their source (an install script or a deb package).

Unfortunately some games don't follow the FHS properly.  Some managed packages put their files in /usr/local and some installers put their unmanaged files in  /usr.  Other installers add their files and then ask to "register" with the package management system for easier removal, resulting in a mix of managed and unmanaged files.  Managed files can be identified using dpkg-query:

dpkg-query -S /path/to/filename

If dpkg-query doesn't find the file then it's not managed.

Desktop entries are organized according to rules in the *.menu files.  While a *.desktop file can be specified directly, most are arranged by matching "Categories" values in the files.  According to the standard there are both "Main" and "Additional" categories.  A desktop entry file should have one Main category except as required (for example, "Audio" or "Video" mandates "AudioVideo") and an Additional category for more precise organizing.  Most of the Additional categories are reserved for specific Main categories.  Obviously sub-menus have have to be defined before they can be used else everything lands in the primary menus.  Linux Mint's Xfce menu configuration doesn't define the game secondary sub-menus by default (mint-artwork-xfce package) so the games menu gets very crowded.  My menu configuration defines sub-menus for all standard game categories.

Desktop entry files with multiple Main and Additional categories specified is an occasional problem.  While some programs can arguably fit into more than one category, doing this in the desktop entry file often causes duplicates in the menus.  Duplication can be suppressed with menu rules but it's much easier to avoid the problem in the first place.  Some desktop entry files don't specify an Additional category and as a result the menu entry lands in the primary sub-menu, not in the secondaries with similar programs.  Occasionally there will be a standards-compliant combination of Main and Additional categories that are not handled properly in the menu definition files which results in an entry erroneously stuck in the main menu or under an "Other" sub-menu.  But errors in the desktop entry files are the root cause most of the time.

Another problem is the categories themselves.  The video game genres are somewhat ambiguous and many games overlap multiple genres.  Even the main category "Game" can be a problem with some.  Emulators, for example, can be specific to a game console (PCSX) or general-purpose (QEMU).  The fdo/XDG standards only cover a few game genres, some of which are very narrow while others are overly broad.  These are my interpretations:

ActionGame: A catch-all for anything that requires reflexes that doesn't fit in a more specific category.

AdventureGame: Point-and-click adventures and interactive stories.  If it requires reflexes then it doesn't belong here.  Even so, there are some games which have a mixture of both so it becomes a judgment call as to developer intent (Full Throttle).

ArcadeGame: An ambiguous genre unless you restrict it to games that were found in coin-operated arcades at some point.  I included games with simplistic gameplay and a minimalistic storyline, if any.  If it has a point-based "high score" then it's probably an arcade game.

BoardGame: Board games and games that could reasonably be board games or were derived from one.

BlocksGame: Tile-matching and Tetris types, as per the Wikipedia article.

CardGame: Probably the easiest to figure out although gtali is a bit of an oddball.

KidsGame: Another non-genre genre.  I included any game with a theme that is heavily slanted towards young children.  However, some also fall into the Educational main category (GCompris).

LogicGame: Slightly odd name as it includes puzzles.  Again there are some platformers that combine this with action (EDGE, Stealth Bastard).

RolePlaying: If it doesn't have a character development system then it's not a role-playing game.  Having a fantasy theme is not good enough.  Waste's Edge is an example of a game that says it's a role-playing game when it obviously is an adventure game.  It's nothing more than a long dialog puzzle.  It's not a bad puzzle and has a good story, but it's not a role-playing game.

Simulation: Vehicle, economic, and environment simulations including cars, cities, and space.  Car racing games are often marketed as sports games but that never seems to include airplane racing games.  Some vehicle simulations have cinematic physics and may belong elsewhere (ArcadeGame, ActionGame).

SportsGame: Traditional sports not involving vehicles.  Includes "fantasy" sports management applications.  Real sports management applications probably belong elsewhere, perhaps Office/ProjectManagement.

StrategyGame: Some are obvious, some not.  The problematic ones are those that require tactics but not quite long-term strategic thinking (Atom Zombie Smasher).  I tend to follow the developer's opinion but not always.

Amusement: Snowing desktops (xsnow), eye-candy (xdesktopwaves), and the like.  Doesn't have a defined Main category but I set all the ones I found to Games since that is the most logical option.  Creating a new primary sub-menu didn't make sense.

Shooters: This is new in the v1.1 draft specification.  Again it is a bit too broad of category but because the ActionGame category is so crowded I decided to support it, just not by default.  I wrote the "set-shooter-categories" script which changes some of my desktop entries to this category (mostly first-person shooters) but the targets are all hard-coded.  It doesn't have a reversion option so keep a backup copy of the original *.desktop files.  I defined a Shooters sub-menu in my replacement menu definitions (, and included an icon and the file.

Adding to the problems are programs that represent many games including the Steam client, Desurium, and gtkboard (which includes Tetris and Mastermind).  I normally just leave these in Games by not specifying an Additional category.

Some games in the repos have the old Debian menu files but not the newer fdo/XDG files.  There is an optional "Debian menu" that can be merged with the newer menu files to create a "Debian" sub-menu with all the old menu entries.  However it has a different structure and is mostly redundant with the main menus - overkill for just getting some games to show up.  I encountered the missing *.desktop file problem often enough that I wrote the "deb-menu-conv" script to export old Debian menu files as something that resembles the new fdo/XDG files.  While deb-menu-conv handles the basics, some things like categories (called "sections" in the Debian menus) do not translate directly.  It has some hard-coded conversions for games but most anything else will need manual editing.  Debian menu files also allowed multiple menu entries per file while the fdo/XDG standard does not.  All entries found in a menu file are exported by deb-menu-conv and have to be split manually into separate *.desktop files.  This script saved me a whole bunch of time but some packages were still a pain (xmpuzzles, xteddy, x11-apps) due to the number of entries their old menu files contained.  I excluded most terminal and text-mode entries.

Some of the programs in the Debian menus are obsolete but I included them as a test of deb-menu-conv.  Many of the amusement-type programs (xfireworks, xsnow, xpenguins) require direct access to the primary X window (root window) that is commonly controlled by a desktop manager in modern desktop environments (xfdesktop in Xfce).  They can be used on Xfce if xfdesktop is terminated first (Settings Manager > Session and Startup > Session > xfdesktop > Quit Program).  Log out and back in again or create a launcher in a desktop panel to restart xfdesktop afterwards.

I also fixed some other bugs via the desktop entry files.  Doom3 doesn't get along with PulseAudio so I added pasuspender to the Exec line in its file.  Some games have both 32-bit and 64-bit versions but their installer only adds one or the other and they have different names.  This meant that a single menu entry couldn't support both so I had to replace it with two different files.  For several of the games with undocumented keys and lack of built-in help I added additional menu entries that open man pages, Readme documents, and web sites with relevant information. 

One parameter that is very useful is "TryExec".  This is a test where the menu system looks for the target file and only shows the menu entry if it is found.  Since it is targeting executables it can be used to check for both file and directory targets.  All of my desktop entry files have this set so that if they are all installed they won't crowd the menus with entries for unavailable programs.  This is also handy for menu entries that target the 32-bit and 64-bit versions of a binary.  However, if a game normally installs both then only one entry is used and it points to a script that dynamically loads the correct binary based on the system architecture.  I wrote or modified several game loading scripts for this reason (and to fix other bugs).

Installing a modified menu or desktop entry file is easy if it is unmanaged - just overwrite the original.  However, managed files are more complicated.  If an update is installed by the package manager then the modified file may be overwritten.  The solution is to divert the original file to a different location with dpkg-divert.  When apt applies the update it will then redirect the diverted file to the new location, leaving the modified one intact.

The installation of the modified files is easy when only a few files are involved.  But I ended up with dozens which I realized would be error-prone and time-consuming to install.  So I wrote the "file-overrides" script to handle installation automatically, make backups of replaced files, and be able to revert changes.  It has built-in help but I will summarize the basics here.

Overrides are located in a subdirectory structure that mimics the actual targets.  The default directory is /usr/local/share/file-overrides.d which contains two subdirectories, managed and unmanaged.  The difference is that managed targets are diverted (and moved) by dpkg-divert while unmanged targets are moved directly.  These subdirectories must exist else the script will abort.

The diversions and backups are located in /usr/share/Diversions/file-overrides by default and this directory must also exist.  The script will create "managed" and "unmanaged" directories as needed and will remove their contents automatically during reversions.

The structures are relative so that a replacement for managed file:


is located at:


During an override operation the original will be moved/diverted to:


prior to being replaced.  During a reversion the sequence is reversed, more or less.  While intended for fixing menu entries the script can be used to replace any file on the system.  It does have some special support for the *.desktop files.  Normally the script will only install an override file if the target already exists to keep the directories clean.  However, if the file is a desktop entry file with TryExec set, and the target executable exists, then the override will be applied even if the target *.desktop file does not exist.  The --force parameter applies all overrides regardless.

Another special case is with override files that are zero bytes in size.  This causes the target to be moved/diverted but not replaced, essentially causing a deletion.

See the readme.txt file for more info on included icons and such.

The archive contains everything.  Note that it has been developed and tested on Ubuntu 12.04 (Precise Pangolin)/Linus Mint 13 (maya) with Xfce 4.10.  It will probably work with newer releases (and Debian) and other desktop environments with the exception of the menu files which may need adjusting.  Compare them to the originals in /etc/xdg/menus.


md5sum: b15e343fe9c901f4556bed27fbce8cf8


Latest batch of UFW application profiles

I'm working on an updated kickstart script for Ubuntu 12.04 and it's late as usual.  Part of the reason was the Unity/Gnome3/Linux Mint/Xfce desktop environment choice problem.  Trying to select one that my customers would like, would work with full-screen 3D games, and my personal non-Xinerama multi-head (aka "Zaphod mode") desktop configuration has been difficult.  I also was dragged into one of the largest CAD projects of my career which took about 5x longer to finish than estimated (mostly not my fault). It paid well but at 70+ hours per week it really messed up the schedules of my other projects.

One of the major developments of the 10.04 installation was the creation of a bunch of application profiles for use with UFW, the "Uncomplicated FireWall".  I submitted these to Ubuntu for inclusion in future releases.  After some initial enthusiasm on the part of their developers for the contribution, the profiles have been more or less forgotten about.

There are multiple GUI interfaces for UFW but at the time only ufw-frontends supported profiles well.  About a week ago a member of the Gufw team contacted me about including the profiles as part of their package.  They also found some errors.  I'm not sure what their current version looks like but hopefully it supports the reference and category extensions to the file format I added.

One thing the profiles didn't have was a license.  After some consideration I settled on GNU GPL v3 or later.  Even though the profiles are not source code or configuration files related to compiling, the GPL can be used.

Because of all this I've updated the profiles to v1.4 and now include a license file and MD5 checksums for identifying which files are covered by the GPL.


Full-featured Ubuntu online installation using kickstart

This is an elaborate fault-tolerant Kickstart script for an on-line Ubuntu installation, optimized for home users, with extensive remote administration support and documentation. Not recommended for beginners.

This isn't just another trivial automated installation script although it started out that way. Basic installation presets led to integrated bug workarounds, setting defaults for many applications and servers, more features, etc. While you may disagree with some of my package choices, they were selected for my clients - not you. Change it if you have different needs. First, a little background on my deployments.

All of my clients have cheap desktop systems or laptops, usually outdated. Almost any CPU, chipset, GPU, and drive configuration. They're either stand-alone or connected together on small Ethernet networks. Some have broadband, some only dial-up (POTS). Ages vary from toddlers to senior citizens. A few are Windows gamers. This mix results in a wide variety of system hardware, peripherals, application requirements, and configurations. I've had to deal with most every type of kernel, application, and hardware bug. Every deployment unearths a new bug to fight. Some of these are Ubuntu's fault but many are upstream.

Inevitably I spend many hours doing full OS conversions to Ubuntu or dual-boot configurations. I've found that using a Live CD to install Ubuntu is about 4x faster than installing Windows when drivers, updates, and application installs are figured in. While I could set up slipstream builds of Windows I don't install it enough to bother with and the variety of versions (Home, Pro, upgrade, OEM,...) and licenses makes it impractical. Relatively speaking, I spend about 3x as long transferring documents, settings, and game/application files (scattered all over C:) to Ubuntu than I do installing either it or Windows. But I'll take any time savings I can get.

A while back, when Ubuntu 10.04 (Lucid Lynx) was released, I decided to streamline my installations. This wasn't just to save time. I also needed to make my installations more uniform as I couldn't remember all the various tweaks and bug fixes that I performed from installation to the next.
I had several goals for this project, not necessarily all at the beginning as some were the result of test installs, client feedback, and feature creep.
  1. Fix all the bugs that my clients encountered on their existing installs plus all the other Ubuntu annoyances I've been manually correcting.
  2. Do everything the "correct way" instead of blindly following HOW-TOs from amateurs that involved script and text file hacking that would be lost on the next update. I had to learn proper use of Gconf, PolicyKit, Upstart, init scripts, and dpkg.
  3. Configure all of the network features that my clients had asked for, usually file or peripheral sharing. Internet content filtering for kids was a requirement.
  4. Secure remote access and administration. It's bad enough when a client has a software problem. Having to waste time with an on-site visit is idiotic when it's not an Internet access problem and a broadband connection is available. The same kickstart configuration can be used for both an "administration" system as well as clients. Having them nearly identical makes both remote and verbal support easier.
  5. Make it easier to obtain diagnostic and status information, for me and the client.
  6. Research applications that meet customer needs and are stable. Configure them so the customer doesn't need to.
  7. Document everything, especially anything I spent significant time researching.

On all of these I mostly succeeded. There are still a few gaps but they're minor (for my deployments at least) but after working on this for 18 months I needed to get on with my life. I figure that after a few million deployments I should break even. I'm now busy updating the dozen or so I currently have.

So what's in it? The base is just a plain 10.04 (i386 or amd64) installation. Two reasons for that - it's the LTS release and I didn't have time to upgrade to newer releases or workaround their new bugs. It's supported for another year or so. I probably update it for 12.04 after it is released (and clean up my code). Highlights: 

Apache. Used for sharing the public directory (see below) and accessing the various web-based tools. The home page is generated from PHP and attempts to correct for port-forwarding (SSH tunnel) if it detects you are not using port 80.

Webmin. It's the standard for web-based administration. I added a module for ddclient (Dynamic DNS). The module is primitive but usable and I fixed the developer's Engrish.

DansGuardian. Probably three months work on just this. For content filtering there isn't really anything else. Unfortunately it has almost no support tools so I had to write them. Most of these have been announced in previous blog postings although they've been updated since then. The most complicated is "dg-squid-control" which enables/disables Squid, DansGuardian, and various iptables rules. Another loads Shalla's blacklist. It doesn't have system group integration so I wrote "dg-filter-group-updater" to semi-integrate it. There are four filter groups - no access, restricted (whitelist with an index page), filtered, and unrestricted. I added a Webmin module for it I found on Sourceforge. It's not great but makes it easier to modify the grey and exception lists. Included are lists I wrote that allow access to mostly kid sites (a couple of hundred entries). The entries have wiki-style links in comments that are extracted by "dg-filter-group-2-index-gen" to create the restricted index page. There's a How-To page for proxy configuration that users are directed to when they try to bypass it.

The only limitation is that browser configurations are set to use the proxy by default but dg-squid-control doesn't have the ability to reset them if the proxy is disabled. I spent two weeks working on INI file parsing functions (many applications still use this bad Windows standard for configuration files). While they seem to work I need to significantly restructure the tool to make use of them.

DansGuardian had no development for a few years but recently a new maintainer is in charge and patches are being accepted. Hopefully full system account integration will be added.

UFW. The Uncomplicated Firewall is a front-end to iptables and there is a GUI for it. One feature it has is application profiles, which make it easy to create read-to-use filter rules. I created about 300 of them for almost every Linux service, application, or game (and and most Windows games on Wine).

File sharing. The /home/local directory is for local (non-network) file sharing between users on the same system. There is also a /home/public directory that is shared over Samba, HTTP, FTP, and NFS. WebDAV didn't make the cut this time around.

Recovery Mode. I added many scripts to the menu for status information from just about everything. Several of my tools are accessible from it.

SSH server. You make a key with ssh-keygen, client_administrator_id_dsa (should be encrypted), and include the public (*.pub) part in the kickstart_files/authentication sub-directory. It is added to the ssh configuration directory on every system. Using another tool, "remote-admin-key-control", system owners (sysowner group) can enable or disable remote access. This is for several reasons including privacy, liability, and accounting (for corporate clients where the person requesting support may not have purchase authority).

When the remote-admin-key-control adds the key to the administrator account ~/.ssh/authorized_keys, you can connect to the system without a password using the private key (you still need to enter the key passphrase). The radmin-ssh tool takes this one step further and forwards the ports for every major network service that can function over ssh. It also shows example command lines (based on the current connection) for scp, sftp, sshfs, and NFS. You still need the administrator password to get root access.

X2Go. Remote desktop access that's faster than VNC. Uses SSH (and the same key).

OpenVPN. A partially configured Remote Technical Support VPN connection is installed and available through Network Manager. If the client system is behind a firewall that you can't SSH through, the client can activate this VPN to connect to your administration system so that you can SSH back through it. Rules for iptables can be enabled that prevent the client accessing anything on the administration system. It connects using 443/udp so should work through most firewalls.

Books and guides. Located in the desktop help menu (System > Help) is a menu entry that opens a directory for books. My deployments have subdirectories with Getting Started with Ubuntu 10.04 - Second Edition from the Ubuntu Manual Project and user guides. You can easily add more as the kickstart script grabs everything in its local-books subdirectory. For the end-user I wrote networks-and-file-sharing-help.html (in the same help menu).

For the installer the main source of documentation is the kickstart script itself. I got a little carried away with comments. The next major document is TODO.html which is added to the administrator's home directory. It was intended to list post-install tasks that needed to be completed since there are many things the installer can't do (like compile kernel modules). After adding background information on the various tasks, troubleshooting help, and example commands, it's basically another book. You should read it before using the kickstart script.

Scanner Server. Allows remote access to a scanner through a web interface. Simpler than using saned (but that is also available if you enable it). It had several bugs so I fixed it and added a few features (with help from a Ubuntu Forum member pqwoerituytrueiwoq). Eventually we hit the limit of what it could do so pqwoerituytrueiwoq started writing PHP Server Scanner as a replacement. For a 12.04 release I will probably use that instead. I wrote "scanner-access-enabler" to work around udev permission problems with some scanners (especially SCSI models).

Notifications. Pop-up notices will be shown from smartd, mdadm, sshd, and OpenVPN when something significant happens. Without the first two the user doesn't know about pending drive problems until the system fails to boot. I've also had them turn the system off when I was in the process of updating it and the SSH notification helps prevent that. The OpenVPN notification is mostly for the administration system and includes the tunnel IP address of the client. OpenSSH has horrible support for this kind of scripting. OpenVPN's scripting support is absolutely beautiful.

Webcam Server. A command-line utility that I wrote a GUI for. It has a Java applet that can only be accessed locally but a static image is available from the internal web server to anywhere.

BackuPC. It uses its default directory for backups so don't enable it unless you mount something else there. A cron job will shut the system down after a backup if there are no users logged in. It has been somewhat hardened against abuse with wrapper scripts for tar and rsync.

There are many bugs, both big and small, that are either fixed or worked around. The script lists the numbers where applicable. The TODO documents lists a bunch also.  Some packages were added but later removed (Oracle/Sun Java due to a licensing problem, Moonlight since it didn't work with any Silverlight site I tested).

There are some limitations to Ubuntu's kickstart support. I'm not sure why I used kickstart in the first place. Perhaps the name reminded me of KiXtart, a tool I used when I was a Windows sysadmin. Kickstart scripts are the standard for automating Red Hat installations (preseeding is the Debian standard) but Ubuntu's version is a crippled clone of it. In part it acts like a preseed file (even has a "preseed" command) but also has sections for scripts that are exported and executed at different points during the installation. About 90% of the installation occurs during the "post-install" script. The worst problem with Ubuntu's kickstart support is that the scripts are exported twice and backslashes are expanded both times. This means that every backslash has to be quadrupled. This gets real ugly with sed and regular expressions. Because of this you'll see "original" and "extra slashy" versions of many command lines. I wrote quad-backslash-check to find mistakes.

The other problem is that the way the script is executed by the installer hides line numbers when syntax errors occur, making debugging difficult. I wrote quote-count and quote-count-query to find unmatched quotes (and trailing escaped whitespace that was supposed to be newlines) which were the most common cause of failure.

I've made an archive of my kickstart file, its support files, and configuration files for various services on my server for you to download (12.5MB, MD5: b5e79e6e287da38da75ea40d0d18f07f ). The script, error checking and ISO management tools, and server configuration files are in the "kickstart" sub-directory. A few packages are included because they are hard to find but others are excluded because of size. Where a package is missing there is a "file_listing.txt" file showing the name of the package I'm using. My installation includes the following which you should download and add back in:

Amazon MP3 Downloader (./Amazon/amazonmp3.deb)
DansGuardian Webmin Module (./DansGuardian Webmin Module/dgwebmin-0.7.1.wbm)
Desura client (./Desura/desura-i686.tar.gz)
G'MIC (./GMIC/gmic_1.5.0.7_*.deb)
Gourmet (./Gourmet/gourmet_0.15.7-1_all.deb)
VMware Player (./VMware/VMware-Player-*.bundle)

VMware Player is optional.  It has kernel modules so the kickstart script only retrieves the first install file returned from the web server whose name matches the architecture.  It puts it in /root for later installation.

The target systems need network-bootable Ethernet devices, either with integrated PXE clients or a bootable CD from ROM-o-matic.

You need a DHCP sever that can send out:

filename "pxelinux.0"

The tftp server needs to serve the pxelinux.0 bootstrap, vesamenu.c32, and the menu files. These are available from the Ubuntu netboot images. The bootstrap and vesamenu.c32 are identical between the i386 and amd64 versions, only the kernel, initrd, and menus are different. You can use my menu files instead of the standard set in the netboot archive. The most important is the "ubuntu.cfg" file. You'll notice that my menu files list many distros and versions. Only the utility, Knoppix, and Ubuntu menus function fully. The rest are unfinished (and probably obsolete) experiments. FreeDOS is for BIOS updates.

My tftp server is atftpd which works well except it has a 30MB or so limit on tftp transfers. This only affects the tftp version of Parted Magic (they have a script to split it up into 30MB parts). It is started by inetd on demand.

I use loopback-mounted ISOs for the kickstart installs and all LiveCDs netboots. Because I have so many, I exceeded the default maximum number of loopback nodes available. I set max_loop=128 in my server's kernel command line to allow for many more.

The Ubuntu Minimal CD ISOs are the source for the kernel and initrd for the kickstart install. The architecture (and release) of the kernel on these ISOs must match the architecture of Ubuntu you want to install on the target system. You'll probably want both the i386 and amd64 versions.

PXE Linux doesn't support symlinks so my ISOs are mounted in the tftp directory under ./isomnt. Symlinks to the ISOs are in ./isolnk and are the source of the mounts. I set it up this way originally because the ISOs were in /srv/linux in various subdirectories so having the links in one place made it easier to manage. But my ISO collection grew too big to manage manually so I wrote "tftp-iso-mount" that creates the mountings for me. It searches through my /srv/linux directory for ISO files and creates isomnt_fstab.txt that can be appended to fstab. It also deletes and recreates the isomnt and isolnk directories and creates the "isomnt-all" script to mount them.

The ISOs are accessed through both NFS and Apache. I originally intended to use NFS for everything but I found that debian-installer, which performs the installation and executes the kickstart script (also on the "alternate" ISOs), doesn't support NFS. So I had to set up Apace to serve them. The Apache configuration is rather simple. There are a few symlinks in /var/www that link to various directories elsewhere. One named "ubuntu" links to /srv/linux/Ubuntu. The kickstart support files are placed in /srv/linux/Ubuntu/kickstart_files and are accessed via the link. NFS is still used for booting LiveCDs (for bug testing and demos). There is also a "tftp" symlink to /srv/tftp used for local deb loading (see below).

The kickstart script itself, Ubuntu-10.04-alternate-desktop.cfg, is saved to /srv/tftp/kickstart/ubuntu/10.04/alternate-desktop.cfg after being backslash and quote checked.

Several preseed values are set with the "preseed" command at the beginning of the script. You'll probably want to change the time zone there. License agreements are pre-agreed to as they will halt the installation if they prompt for input.

Like I mentioned earlier, the vast majority of work happens in the post-install script. The executes after the base Ubuntu packages are installed. The most important variable to set is $add_files_root which must point to the URL and directory of your web server where the rest of the kickstart support files are located (no trailing backslash). The script adapts for 32-bit and 64-bit packages as needed based on the architecture of the netboot installer. There is also a "late_command" script that executes near the end of the installation, after debian-installer creates the administrator account (which happens after the post-install script finishes).

The debug variables are important for the initial tests. The $package_debug variable has the most impact as it will change package installations from large blocks installed in one pass (not "true") to each package individually ("true"). When true, it slows down installation significantly but you can find individual package failures in the kickseed-post-script.log and installer syslog (located in /var/log/installer after installation). Setting $wget_quiet to null will guarantee a huge log file. The $script_debug variable controls status messages from the package install and mirror selection functions.

The $mirror_list variable contains a list of Ubuntu mirrors (not Medibuntu or PPAs) that should have relatively similar update intervals. This is used by the fault-tolerant mirror selection function, f_mirror_chk, that will cycle through these and check for availability and stability (i.e., not in the middle of sync). The mirrors included in the list are good for the USA. These are exported to the apt directory so that the apt-mirror-rotate command can use them to change mirrors quickly from the command line or through the recovery mode menu. When a package fails to be installed via the f_ftdpkg and f_ftapt functions, another mirror will be tried to attempt to work around damaged packages or missing dependencies.

To save bandwidth the post-install script looks for loopback mounted ISOs of the Ubuntu 10.04 live CD and Ubuntu Studio (both i386 and amd64 versions) in the isomnt sub-directory via the tftp link in the Apache default site. It copies all debs it finds directly into the apt cache. It also copies the contents of several kickstart support sub-directories (game-debs* and local-debs*). This is a primitive way to serve the bulk of the packages locally while retrieving everything else from the mirrors. You need to change the URLs in the pre-load debs section to the "pool" sub-directories of the mounted ISOs in "./tftp/isomnt/".

Because loading this many debs can run a root volume out of space, the $game_debs variable can be used to prevent game packages from being retrieved. Normally you should have at least a 20GB root (/) volume although it could be made smaller with some experimentation. An alternative to this method would be a full deb-mirror or a large caching proxy.

Set the OpenVPN variables $openvpnurl to the Internet URL of your administration system or the firewall it's behind. Set $openvpnserver to the hostname of your administration system (which can have the same values as it won't be connecting to itself).

Basic usage starts with netbooting the client system. Some have to be set to netboot in the BIOS and some have a hotkey you can press at POST to access a boot selection menu. The system then obtains an address and BOOTP information from the DHCP server. It then loads pxelinux.0 from the TFTP server which will in turn load vesamenu.c32 which displays the "Netboot main menu". Select Ubuntu from the list and look for the Ubuntu 10.04 Minimal CD KS entries. Select the one for your architecture and press the Tab key to edit the kernel boot line. Set any kernel parameters you want to be added to the default Grub2 configuration after the double dash (--), like "nomodeset". Set the hostname and domain values for the target as these are used in several places for bug workarounds and configurations. Then press Enter. The installer should boot. If nothing happens when you press Enter and you are returned to the Ubuntu boot listing menu, verify the ISOs are mounted on the server then try again (you will need to edit the entry again).

If there are no problems then you will be asked only two questions. The first is drive partitioning. This can be automated but my client systems are too different to do so. Then next question will be the administrator password. After that it will execute the post-install script and late-command scripts then prompt you to reboot. Just hit the enter key when it does as Ctrl-Alt-Delete will prevent the installer from properly finishing the installation (it's not quite done when it says it is). Full installation will take 2-3 hours depending on debug settings, availability of local debs, and Internet speeds.

In case of problems see the TODO document which has a troubleshooting section. The only problems I've had installing was missing drivers or bugs in the installer (especially with encrypted drives - see the TODO). My Dell Inspiron 11z, which has an Atheros AR8132/L1c Ethernet device, wasn't supported by the kernel the minimal CD was using. To work around it I made a second network connection with an old Linksys USB100TX. The Atheros did the netboot (the Linksys does not have the capability) but the installer only saw the Linksys afterwards and had no problems using it (other than it being slow).

I welcome comments and suggestions (other than my package choices and blog color scheme :D).

About Me

Omnifarious Implementer = I do just about everything. With my usual occupations this means anything an electrical engineer does not feel like doing including PCB design, electronic troubleshooting and repair, part sourcing, inventory control, enclosure machining, label design, PC support, network administration, plant maintenance, janitorial, etc. Non-occupational includes residential plumbing, heating, electrical, farming, automotive and small engine repair. There is plenty more but you get the idea.