Stubborn Tech Problem Solving

Full-featured Ubuntu online installation using kickstart

2011-12-15T23:58:00.000-05:00

This is an elaborate fault-tolerant Kickstart script for an on-line Ubuntu installation, optimized for home users, with extensive remote administration support and documentation. Not recommended for beginners.

This isn't just another trivial automated installation script although it started out that way. Basic installation presets led to integrated bug workarounds, setting defaults for many applications and servers, more features, etc. While you may disagree with some of my package choices, they were selected for my clients - not you. Change it if you have different needs. First, a little background on my deployments.

All of my clients have cheap desktop systems or laptops, usually outdated. Almost any CPU, chipset, GPU, and drive configuration. They're either stand-alone or connected together on small Ethernet networks. Some have broadband, some only dial-up (POTS). Ages vary from toddlers to senior citizens. A few are Windows gamers. This mix results in a wide variety of system hardware, peripherals, application requirements, and configurations. I've had to deal with most every type of kernel, application, and hardware bug. Every deployment unearths a new bug to fight. Some of these are Ubuntu's fault but many are upstream.

Inevitably I spend many hours doing full OS conversions to Ubuntu or dual-boot configurations. I've found that using a Live CD to install Ubuntu is about 4x faster than installing Windows when drivers, updates, and application installs are figured in. While I could set up slipstream builds of Windows I don't install it enough to bother with and the variety of versions (Home, Pro, upgrade, OEM,...) and licenses makes it impractical. Relatively speaking, I spend about 3x as long transferring documents, settings, and game/application files (scattered all over C:) to Ubuntu than I do installing either it or Windows. But I'll take any time savings I can get.

A while back, when Ubuntu 10.04 (Lucid Lynx) was released, I decided to streamline my installations. This wasn't just to save time. I also needed to make my installations more uniform as I couldn't remember all the various tweaks and bug fixes that I performed from installation to the next.
I had several goals for this project, not necessarily all at the beginning as some were the result of test installs, client feedback, and feature creep.

Fix all the bugs that my clients encountered on their existing installs plus all the other Ubuntu annoyances I've been manually correcting.
Do everything the "correct way" instead of blindly following HOW-TOs from amateurs that involved script and text file hacking that would be lost on the next update. I had to learn proper use of Gconf, PolicyKit, Upstart, init scripts, and dpkg.
Configure all of the network features that my clients had asked for, usually file or peripheral sharing. Internet content filtering for kids was a requirement.
Secure remote access and administration. It's bad enough when a client has a software problem. Having to waste time with an on-site visit is idiotic when it's not an Internet access problem and a broadband connection is available. The same kickstart configuration can be used for both an "administration" system as well as clients. Having them nearly identical makes both remote and verbal support easier.
Make it easier to obtain diagnostic and status information, for me and the client.
Research applications that meet customer needs and are stable. Configure them so the customer doesn't need to.
Document everything, especially anything I spent significant time researching.

On all of these I mostly succeeded. There are still a few gaps but they're minor (for my deployments at least) but after working on this for 18 months I needed to get on with my life. I figure that after a few million deployments I should break even. I'm now busy updating the dozen or so I currently have.

So what's in it? The base is just a plain 10.04 (i386 or amd64) installation. Two reasons for that - it's the LTS release and I didn't have time to upgrade to newer releases or workaround their new bugs. It's supported for another year or so. I probably update it for 12.04 after it is released (and clean up my code). Highlights:

Apache. Used for sharing the public directory (see below) and accessing the various web-based tools. The home page is generated from PHP and attempts to correct for port-forwarding (SSH tunnel) if it detects you are not using port 80.

Webmin. It's the standard for web-based administration. I added a module for ddclient (Dynamic DNS). The module is primitive but usable and I fixed the developer's Engrish.

DansGuardian. Probably three months work on just this. For content filtering there isn't really anything else. Unfortunately it has almost no support tools so I had to write them. Most of these have been announced in previous blog postings although they've been updated since then. The most complicated is "dg-squid-control" which enables/disables Squid, DansGuardian, and various iptables rules. Another loads Shalla's blacklist. It doesn't have system group integration so I wrote "dg-filter-group-updater" to semi-integrate it. There are four filter groups - no access, restricted (whitelist with an index page), filtered, and unrestricted. I added a Webmin module for it I found on Sourceforge. It's not great but makes it easier to modify the grey and exception lists. Included are lists I wrote that allow access to mostly kid sites (a couple of hundred entries). The entries have wiki-style links in comments that are extracted by "dg-filter-group-2-index-gen" to create the restricted index page. There's a How-To page for proxy configuration that users are directed to when they try to bypass it.

The only limitation is that browser configurations are set to use the proxy by default but dg-squid-control doesn't have the ability to reset them if the proxy is disabled. I spent two weeks working on INI file parsing functions (many applications still use this bad Windows standard for configuration files). While they seem to work I need to significantly restructure the tool to make use of them.

DansGuardian had no development for a few years but recently a new maintainer is in charge and patches are being accepted. Hopefully full system account integration will be added.

UFW. The Uncomplicated Firewall is a front-end to iptables and there is a GUI for it. One feature it has is application profiles, which make it easy to create read-to-use filter rules. I created about 300 of them for almost every Linux service, application, or game (and and most Windows games on Wine).

File sharing. The /home/local directory is for local (non-network) file sharing between users on the same system. There is also a /home/public directory that is shared over Samba, HTTP, FTP, and NFS. WebDAV didn't make the cut this time around.

Recovery Mode. I added many scripts to the menu for status information from just about everything. Several of my tools are accessible from it.

SSH server. You make a key with ssh-keygen, client_administrator_id_dsa (should be encrypted), and include the public (*.pub) part in the kickstart_files/authentication sub-directory. It is added to the ssh configuration directory on every system. Using another tool, "remote-admin-key-control", system owners (sysowner group) can enable or disable remote access. This is for several reasons including privacy, liability, and accounting (for corporate clients where the person requesting support may not have purchase authority).

When the remote-admin-key-control adds the key to the administrator account ~/.ssh/authorized_keys, you can connect to the system without a password using the private key (you still need to enter the key passphrase). The radmin-ssh tool takes this one step further and forwards the ports for every major network service that can function over ssh. It also shows example command lines (based on the current connection) for scp, sftp, sshfs, and NFS. You still need the administrator password to get root access.

X2Go. Remote desktop access that's faster than VNC. Uses SSH (and the same key).

OpenVPN. A partially configured Remote Technical Support VPN connection is installed and available through Network Manager. If the client system is behind a firewall that you can't SSH through, the client can activate this VPN to connect to your administration system so that you can SSH back through it. Rules for iptables can be enabled that prevent the client accessing anything on the administration system. It connects using 443/udp so should work through most firewalls.

Books and guides. Located in the desktop help menu (System > Help) is a menu entry that opens a directory for books. My deployments have subdirectories with Getting Started with Ubuntu 10.04 - Second Edition from the Ubuntu Manual Project and OpenOffice.org user guides. You can easily add more as the kickstart script grabs everything in its local-books subdirectory. For the end-user I wrote networks-and-file-sharing-help.html (in the same help menu).

For the installer the main source of documentation is the kickstart script itself. I got a little carried away with comments. The next major document is TODO.html which is added to the administrator's home directory. It was intended to list post-install tasks that needed to be completed since there are many things the installer can't do (like compile kernel modules). After adding background information on the various tasks, troubleshooting help, and example commands, it's basically another book. You should read it before using the kickstart script.

Scanner Server. Allows remote access to a scanner through a web interface. Simpler than using saned (but that is also available if you enable it). It had several bugs so I fixed it and added a few features (with help from a Ubuntu Forum member pqwoerituytrueiwoq). Eventually we hit the limit of what it could do so pqwoerituytrueiwoq started writing PHP Server Scanner as a replacement. For a 12.04 release I will probably use that instead. I wrote "scanner-access-enabler" to work around udev permission problems with some scanners (especially SCSI models).

Notifications. Pop-up notices will be shown from smartd, mdadm, sshd, and OpenVPN when something significant happens. Without the first two the user doesn't know about pending drive problems until the system fails to boot. I've also had them turn the system off when I was in the process of updating it and the SSH notification helps prevent that. The OpenVPN notification is mostly for the administration system and includes the tunnel IP address of the client. OpenSSH has horrible support for this kind of scripting. OpenVPN's scripting support is absolutely beautiful.

Webcam Server. A command-line utility that I wrote a GUI for. It has a Java applet that can only be accessed locally but a static image is available from the internal web server to anywhere.

BackuPC. It uses its default directory for backups so don't enable it unless you mount something else there. A cron job will shut the system down after a backup if there are no users logged in. It has been somewhat hardened against abuse with wrapper scripts for tar and rsync.

There are many bugs, both big and small, that are either fixed or worked around. The script lists the numbers where applicable. The TODO documents lists a bunch also. Some packages were added but later removed (Oracle/Sun Java due to a licensing problem, Moonlight since it didn't work with any Silverlight site I tested).

There are some limitations to Ubuntu's kickstart support. I'm not sure why I used kickstart in the first place. Perhaps the name reminded me of KiXtart, a tool I used when I was a Windows sysadmin. Kickstart scripts are the standard for automating Red Hat installations (preseeding is the Debian standard) but Ubuntu's version is a crippled clone of it. In part it acts like a preseed file (even has a "preseed" command) but also has sections for scripts that are exported and executed at different points during the installation. About 90% of the installation occurs during the "post-install" script. The worst problem with Ubuntu's kickstart support is that the scripts are exported twice and backslashes are expanded both times. This means that every backslash has to be quadrupled. This gets real ugly with sed and regular expressions. Because of this you'll see "original" and "extra slashy" versions of many command lines. I wrote quad-backslash-check to find mistakes.

The other problem is that the way the script is executed by the installer hides line numbers when syntax errors occur, making debugging difficult. I wrote quote-count and quote-count-query to find unmatched quotes (and trailing escaped whitespace that was supposed to be newlines) which were the most common cause of failure.

I've made an archive of my kickstart file, its support files, and configuration files for various services on my server for you to download (12.5MB, MD5: b5e79e6e287da38da75ea40d0d18f07f ). The script, error checking and ISO management tools, and server configuration files are in the "kickstart" sub-directory. A few packages are included because they are hard to find but others are excluded because of size. Where a package is missing there is a "file_listing.txt" file showing the name of the package I'm using. My installation includes the following which you should download and add back in:

Amazon MP3 Downloader (./Amazon/amazonmp3.deb)
DansGuardian Webmin Module (./DansGuardian Webmin Module/dgwebmin-0.7.1.wbm)
Desura client (./Desura/desura-i686.tar.gz)
G'MIC (./GMIC/gmic_1.5.0.7_*.deb)
Gourmet (./Gourmet/gourmet_0.15.7-1_all.deb)
VMware Player (./VMware/VMware-Player-*.bundle)

VMware Player is optional. It has kernel modules so the kickstart script only retrieves the first install file returned from the web server whose name matches the architecture. It puts it in /root for later installation.

The target systems need network-bootable Ethernet devices, either with integrated PXE clients or a bootable CD from ROM-o-matic.

You need a DHCP sever that can send out:

filename "pxelinux.0"
next-server

The tftp server needs to serve the pxelinux.0 bootstrap, vesamenu.c32, and the menu files. These are available from the Ubuntu netboot images. The bootstrap and vesamenu.c32 are identical between the i386 and amd64 versions, only the kernel, initrd, and menus are different. You can use my menu files instead of the standard set in the netboot archive. The most important is the "ubuntu.cfg" file. You'll notice that my menu files list many distros and versions. Only the utility, Knoppix, and Ubuntu menus function fully. The rest are unfinished (and probably obsolete) experiments. FreeDOS is for BIOS updates.

My tftp server is atftpd which works well except it has a 30MB or so limit on tftp transfers. This only affects the tftp version of Parted Magic (they have a script to split it up into 30MB parts). It is started by inetd on demand.

I use loopback-mounted ISOs for the kickstart installs and all LiveCDs netboots. Because I have so many, I exceeded the default maximum number of loopback nodes available. I set max_loop=128 in my server's kernel command line to allow for many more.

The Ubuntu Minimal CD ISOs are the source for the kernel and initrd for the kickstart install. The architecture (and release) of the kernel on these ISOs must match the architecture of Ubuntu you want to install on the target system. You'll probably want both the i386 and amd64 versions.

PXE Linux doesn't support symlinks so my ISOs are mounted in the tftp directory under ./isomnt. Symlinks to the ISOs are in ./isolnk and are the source of the mounts. I set it up this way originally because the ISOs were in /srv/linux in various subdirectories so having the links in one place made it easier to manage. But my ISO collection grew too big to manage manually so I wrote "tftp-iso-mount" that creates the mountings for me. It searches through my /srv/linux directory for ISO files and creates isomnt_fstab.txt that can be appended to fstab. It also deletes and recreates the isomnt and isolnk directories and creates the "isomnt-all" script to mount them.

The ISOs are accessed through both NFS and Apache. I originally intended to use NFS for everything but I found that debian-installer, which performs the installation and executes the kickstart script (also on the "alternate" ISOs), doesn't support NFS. So I had to set up Apace to serve them. The Apache configuration is rather simple. There are a few symlinks in /var/www that link to various directories elsewhere. One named "ubuntu" links to /srv/linux/Ubuntu. The kickstart support files are placed in /srv/linux/Ubuntu/kickstart_files and are accessed via the link. NFS is still used for booting LiveCDs (for bug testing and demos). There is also a "tftp" symlink to /srv/tftp used for local deb loading (see below).

The kickstart script itself, Ubuntu-10.04-alternate-desktop.cfg, is saved to /srv/tftp/kickstart/ubuntu/10.04/alternate-desktop.cfg after being backslash and quote checked.

Several preseed values are set with the "preseed" command at the beginning of the script. You'll probably want to change the time zone there. License agreements are pre-agreed to as they will halt the installation if they prompt for input.

Like I mentioned earlier, the vast majority of work happens in the post-install script. The executes after the base Ubuntu packages are installed. The most important variable to set is $add_files_root which must point to the URL and directory of your web server where the rest of the kickstart support files are located (no trailing backslash). The script adapts for 32-bit and 64-bit packages as needed based on the architecture of the netboot installer. There is also a "late_command" script that executes near the end of the installation, after debian-installer creates the administrator account (which happens after the post-install script finishes).

The debug variables are important for the initial tests. The $package_debug variable has the most impact as it will change package installations from large blocks installed in one pass (not "true") to each package individually ("true"). When true, it slows down installation significantly but you can find individual package failures in the kickseed-post-script.log and installer syslog (located in /var/log/installer after installation). Setting $wget_quiet to null will guarantee a huge log file. The $script_debug variable controls status messages from the package install and mirror selection functions.

The $mirror_list variable contains a list of Ubuntu mirrors (not Medibuntu or PPAs) that should have relatively similar update intervals. This is used by the fault-tolerant mirror selection function, f_mirror_chk, that will cycle through these and check for availability and stability (i.e., not in the middle of sync). The mirrors included in the list are good for the USA. These are exported to the apt directory so that the apt-mirror-rotate command can use them to change mirrors quickly from the command line or through the recovery mode menu. When a package fails to be installed via the f_ftdpkg and f_ftapt functions, another mirror will be tried to attempt to work around damaged packages or missing dependencies.

To save bandwidth the post-install script looks for loopback mounted ISOs of the Ubuntu 10.04 live CD and Ubuntu Studio (both i386 and amd64 versions) in the isomnt sub-directory via the tftp link in the Apache default site. It copies all debs it finds directly into the apt cache. It also copies the contents of several kickstart support sub-directories (game-debs* and local-debs*). This is a primitive way to serve the bulk of the packages locally while retrieving everything else from the mirrors. You need to change the URLs in the pre-load debs section to the "pool" sub-directories of the mounted ISOs in "./tftp/isomnt/".

Because loading this many debs can run a root volume out of space, the $game_debs variable can be used to prevent game packages from being retrieved. Normally you should have at least a 20GB root (/) volume although it could be made smaller with some experimentation. An alternative to this method would be a full deb-mirror or a large caching proxy.

Set the OpenVPN variables $openvpnurl to the Internet URL of your administration system or the firewall it's behind. Set $openvpnserver to the hostname of your administration system (which can have the same values as it won't be connecting to itself).

Basic usage starts with netbooting the client system. Some have to be set to netboot in the BIOS and some have a hotkey you can press at POST to access a boot selection menu. The system then obtains an address and BOOTP information from the DHCP server. It then loads pxelinux.0 from the TFTP server which will in turn load vesamenu.c32 which displays the "Netboot main menu". Select Ubuntu from the list and look for the Ubuntu 10.04 Minimal CD KS entries. Select the one for your architecture and press the Tab key to edit the kernel boot line. Set any kernel parameters you want to be added to the default Grub2 configuration after the double dash (--), like "nomodeset". Set the hostname and domain values for the target as these are used in several places for bug workarounds and configurations. Then press Enter. The installer should boot. If nothing happens when you press Enter and you are returned to the Ubuntu boot listing menu, verify the ISOs are mounted on the server then try again (you will need to edit the entry again).

If there are no problems then you will be asked only two questions. The first is drive partitioning. This can be automated but my client systems are too different to do so. Then next question will be the administrator password. After that it will execute the post-install script and late-command scripts then prompt you to reboot. Just hit the enter key when it does as Ctrl-Alt-Delete will prevent the installer from properly finishing the installation (it's not quite done when it says it is). Full installation will take 2-3 hours depending on debug settings, availability of local debs, and Internet speeds.

In case of problems see the TODO document which has a troubleshooting section. The only problems I've had installing was missing drivers or bugs in the installer (especially with encrypted drives - see the TODO). My Dell Inspiron 11z, which has an Atheros AR8132/L1c Ethernet device, wasn't supported by the kernel the minimal CD was using. To work around it I made a second network connection with an old Linksys USB100TX. The Atheros did the netboot (the Linksys does not have the capability) but the installer only saw the Linksys afterwards and had no problems using it (other than it being slow).

I welcome comments and suggestions (other than my package choices and blog color scheme :D).

Haphazard proxy support in Linux programs

2011-11-25T17:50:00.025-05:00

Some of my clients require Internet content filtering on computers their kids are using. The solution to that is DansGuardian. While it has many problems there really isn't a better F/OSS alternative. Its development has been stagnant for years but recently a new maintainer joined the project so submitted patches are being applied to fix bugs and add features (like system group integration).

DansGuardian requires a proxy. The common options are TinyProxy and Squid. TinyProxy has a few annoying bugs so I use Squid with my clients. One challenge with content filtering is preventing the proxy from being bypassed. The two solutions are transparent interception or an explicit-proxy with dropping of connections that aren't destined for the proxy ports.

With a transparent proxy all outgoing connections are routed via iptables rules to DansGuardian regardless of the client settings. While this simplifies deployment by eliminating client configuration it also prevents using different content filtering levels on a per-user basis as it masks the source port of the connection. Without the source port the associated user can't be identified. Since the systems I maintain have a variety of users within the same household and thus different filtering requirements, this doesn't meet their needs.

The alternative method is to use iptables rules that drop connections that aren't destined for the DansGuardian. Here are the nat rules that I use:

*nat :PREROUTING ACCEPT :POSTROUTING ACCEPT :OUTPUT ACCEPT -A OUTPUT ! -o lo -p tcp -m owner ! --uid-owner proxy -m owner ! --uid-owner root -m owner ! --uid-owner clamav -m owner ! --uid-owner administrator -m tcp --dport 80 -j REDIRECT --to-ports 8090 -A OUTPUT ! -o lo -p tcp -m owner ! --uid-owner proxy -m owner ! --uid-owner root -m owner ! --uid-owner clamav -m owner ! --uid-owner administrator -m tcp --dport 443 -j REDIRECT --to-ports 8090 -A OUTPUT ! -o lo -p tcp -m owner ! --uid-owner proxy -m owner ! --uid-owner root -m owner ! --uid-owner clamav -m owner ! --uid-owner administrator -m tcp --dport 21 -j REDIRECT --to-ports 8090 -A OUTPUT -p tcp -m tcp --dport 3128 -m owner ! --uid-owner dansguardian -m owner ! --uid-owner root -m owner ! --uid-owner clamav -m owner ! --uid-owner administrator -j REDIRECT --to-ports 8080 COMMIT

Fairly simple but note that I'm not dropping the packets. Any TCP connection that is destined for ports 80 (HTTP), 443 (HTTPS), and FTP (21) are rerouted to port 8090. Some accounts are excluded to prevent false-positive blocking by DansGuardian.

DansGuardian is using port 8080 (and connects to Squid on 3128). So what is 8090? Its an Apache server. One of the problems with programs that aren't configured to use the proxy is that the users won't know why their connections are failing. The web site, known as a network billboard, displays a page that informs them that their programs need to be configured to use the proxy and how to do it. This is much friendlier than just dropping the packets. DansGuardian uses ident2 to identify the user that is the source of the connection and applies the filtering rules specific to the filter group they are assigned to.

This configuration works very well with web browsers. Most use the system proxy settings through gconf on Gnome. Some need manual configuration so I created default configuration files and put them in /etc/skel so that new user accounts have them at creation. Unfortunately, many other programs rely on environment variables to determine the proxy address and Ubuntu's proxy configuration tool (gnome-network-properties) has a really stupid bug and they aren't set correctly. Some are set in bash in terminal windows but not in the session so any graphical program that doesn't use gconf fails to access the proxy correctly. It's easy to demonstrate. Open a terminal window and enter:

tail -f ~/.xsession-errors

Then create a custom application launcher in the panel and enter "printenv" for the command. Then just click it and check the output from tail. On my system, variables for "HTTP_PROXY" and the like aren't present. I created a fix for this. Just extract the file and add it to the end of ~/.profile and relogin. Run the tail/printenv commands again with a proxy set in System>Preferences>Network Proxy. Add this fix to /etc/skel/.profile to use it as the default for new user accounts.

Even with this fix it is surprising is how many Internet-using programs don't support proxies correctly. I tested every streaming media player I could find and a few other programs and here are the results with my systems (Ubuntu 10.04 Lucid Lynx i386 and amd64):

Clementine (0.7.1): Neither Last.fm and SomaFM work. Jamendo lists songs but doesn't play them but this is due to Ogg problems at Jamendo. Unlike other players Clementine's plug-in for Jamendo is not configurable for MP3 so I couldn't work around it. Mangatune and Icecast work.

Rhythmbox (0.13.1): Jamendo failed to work. Magnatune was really slow to load.

Miro (4.0.3-82931155): Could find video podcasts but not download them (except VODO which uses BitTorrent). Its integrated web browser would always show the network bulletin for any other link in the side panel.

Banshee (2.0.1): Internet Archive links work. Live365.com and xiph.org show results but nothing plays (I can copy the xiph links to VLC and they play). Miro Guide works (unlike Miro) but likes to freeze. Amazon MP3 Store, Jamendo, Magnatune (both extensions), RealRadios.com, and SHOUTcast.com extensions fail to load. Last.fm would log in but not much else. I noticed that according to ~/.xsession-errors Banshee is an exceptional media player.

Gnome MPlayer (0.9.9.2): Nothing fancy but it functioned with the streams I tried.

VLC (1.0.6): About the same as Gnome MPlayer. A lot of complaints about some playlists like radio.wazee when it encounters unavailable entries. Needs a less ugly way to handle error messages with playlists of Internet streams since they are usually just alternate servers.

Google Earth (6.1): It would connect to the DB and you could navigate the worlds but none of the Panoramio pictures would show. Wikipedia entries wouldn't show after being enabled until the app was restarted. Even then, clicking on "Full Article" resulted in the network bulletin page being shown (webkit?). Changing the preferences to use an external browser is an adequate workaround.

Totem (2.30.2): Functioned but was picky about some streams (radio.wazee).

gPodder (2.2): Useless.

Hulu beta functions but is mostly relying on Flash.

Skype beta (2.2.0.35): Connected to their network without problems and I successfully called their sound testing service.

Sun Java Plug-in (1.6.0_26 in Firefox 3.6.24): Useless with a proxy. Even without a proxy you have to work around IPv6 bugs (Debian bug #618725). With that working the online test usually fails and I've found that Pogo.com Boggle Bash is a better test. Manually setting the proxy with jcontrol doesn't have any effect. Debian is dropping the plug-in so it may not matter.

FrostWire (5.1.5): Useless with a proxy. It uses Java so not surprising. It has its own proxy settings but it couldn't connect to anything even with manual settings.

Update - Added a few more tests:

Desura (110.22): Could login and see items I had ordered (free demos) but could not download them for installation or show any web pages. Some of the links on the menu bar opened in Firefox but showed the network bulletin. Apparently it was resolving the links (maybe querying their servers) to localhost:8090 and then sending that to the default browser even though Firefox could access the Internet through the proxy without problems.

Konqueror (4.4.5): No problems (KHTML).

Epiphany (2.30.2): No problems (webkit).

X-Moto (0.5.9): No problems. Can use environment variables, manually-specified proxy, or SOCKS proxy.

DraftSight (Beta V1R1.3): Couldn't connect to the registration server initially. The browser in the Home panel showed the network bulletin. Setting the proxy manually in "Tools>Options>System Options>General>Proxy server settings" and restarting allowed the registration to function but not the Home panel browser. I found that reapplying the proxy settings (without changing anything) then right-clicking the Home panel and reloading it fixed the problem for that session but it would reoccur if DraftSight was restarted.

Clarification: My proxy configuration doesn't use authentication or SOCKS. My bug work-around script supports the environment variables for authentication but I didn't test it.

Update 20111202: I removed Sun Java because of the security problems and switched to OpenJDK/IcedTea6 (1.9.10) but it didn't do any better. I did try FrostWire again with a manually specified proxy but it had no effect. I did come across an interesting Java library for proxy detection named proxy-vole but it won't solve my immediate problem.

Update 20111204: Corrected the DansGuardian/Squid port usages mentioned in the article and added a forgotten DansGuardian anti-bypass iptables rule. They now match my test environment.

I think part of the problem is that the developers test against a proxy and if the program works then its assumed to be proxy-compatible. That can be misleading, especially when multiple components are involved, as some may use the proxy while others access the network directly (Miro being a prime example). Adding some iptables rules to drop anything bypassing the proxy would close that testing hole.

Documentation standards for commands

2011-11-23T16:14:00.005-05:00

Here are some references for shell script developers, man page creators, README writers, etc. While documentation styles are a bit haphazard and vary with OS and programming language, there are some standards.

For man pages see man-pages(7). What does that mean? You open a terminal window then type:

man 7 man-pages

The GNU project has some guidelines on writing software manuals. They recommend using Texinfo to create them.

The Debian Policy Manual says where the different documentation files should be located but not what they should look like.

The most detailed standard I've found is the Open Group Base Specifications utility conventions and typographical conventions.

I'm not going to admit to following these but please post any other IT technical writing style guides you know of. :D

Extracting EML files

2011-09-21T01:17:00.003-04:00

EML files are a problem for some of my users on Ubuntu. They receive these as Email attachments but can only view them as text (usually in gedit) even if they contain pictures. The senders are probably using Outlook Express or a related mail application to attach them. While some non-Microsoft mail clients can open them properly this is a hassle for my users as they all use web mail. There is a command-line tool, munpack, that will extract non-text objects automatically (part of the mpack package in Ubuntu/Debian). To make it easier for them I wrote a little script that integrates munpack with their file manager via a mime type association. To use it, download munpack_eml and extract the files. Put munpack_eml in /usr/local/bin with root ownership and u=rwx,go=rx (0755) permissions. Put munpack_eml.desktop in /usr/local/share/applicatons with root ownership and u=rw,go=r (0644) permissions. Then right-click on any *.eml file from your file manager and you should see and option to extract the contents with munpack.

Simple off-site backup of a MD RAID 1 system

2011-08-29T18:11:00.006-04:00

Standard backup tools like BackupPC are great for backing-up moderate amounts of user data but they can be impractical with huge data stores such as multi-terabyte RAID arrays as they need a backup store that is larger than the source data. My simple solution is to clone the array with another drive and store it off-site.

For this to work I had to categorize the data between smaller dynamic files (like documents) and larger static files (videos). The smaller files are backed up daily with BackupPC. The larger files are not backed up. Both are stored on a RAID 1 (mirror) array for redundancy in case of drive failure. On my server BackupPC uses a different, smaller RAID 1 array for a backup store. Since it is only backing up part of the data it doesn't have to be the same size as the main array. For backing up the larger/static files (and everything else) I simply add another drive to the main array, let it sync, then remove it and store off-site.

Ideally this system would use hot-swap but I don't have removable bays so I have to power-off the server each time. The rest of the procedure is relatively easy. With a RAID 1 array I have two drives (sda, sdb) and the added drive may show up as sdc. I say "may" because Ubuntu uses UUIDs for drive mappings and the actual device assignments may change. I always check with:

cat /proc/mdstat

to verify what devices are being used. I also check the partition sizes of all drives using "fdisk -l" and make sure the new drive has the same size partitions as the original RAID members. The partitions need to be of type fd "Linux raid autodetect" but no formatting with mkfs is necessary. Next I grow each RAID 1 MD device from 2 to 3 devices. For example:

mdadm -G -n 3 /dev/md0

This just tells the kernel that the array will now have three devices but does not assign another device to it. To allocate the device:

mdadm -a /dev/md0 /dev/sdc1

Resync should begin immediately. To monitor, I just use "cat /proc/mdstat" but the kernel will also send status messages to the console. After resynching, I disable the backup device by failing it:

mdadm -f /dev/md0 /dev/sdc1

This results in the RAID degradation warnings to be emailed to root. Next I remove it:

mdadm -r /dev/md0 /dev/sdc1

Finally, I shrink the array back to two devices:

mdadm -G -n 2 /dev/md0

This works well for my simple server setup. Obviously some scripting could be used to automate it. While this works well for a 2-drive RAID 1 array, it doesn't scale well with a larger number of drives or other RAID types.

Expanding Ubuntu Recovery Mode

2011-01-17T17:36:00.017-05:00

Recovery Mode is a text-based interface to a few quick repair tools that is installed by default with most Ubuntu releases and derivatives. I wrote a few add-ons for it that increase its usefulness in remote repair and diagnostics situations. These were developed and tested on Ubuntu 10.04 (Lucid Lynx).

Starting Ubuntu in Recovery Mode (aka. Friendly Recovery) is relatively easy. Just hold down the shift key after the BIOS POST to get Grub2 to show its menu, then just select the kernel with the "recovery" option. Also note the memtest86+ option which is useful for identifying bad RAM.

Adding on to Recovery Mode is relatively simple. At its heart is a shell script, "/usr/share/recovery-mode/recovery-menu", that is started at the end of the single mode (runlevel S) boot. It looks through the options subdirectory and starts every script it finds, passing it a parameter of "test". It looks for a return status of 0 and the description of the script on stdout. Scripts with valid responses are added together and shown in a menu listing using the whiptail dialogger. The user selects one from the menu to execute it.

My additions are more informative than corrective. The intention is to help with diagnostics when dealing with a remote non-technical client. They are also useful for beginners who lack command-line experience and simply don't know where to look for system status information.

Many of my scripts check their respective system configuration and return a non-zero status if required executables are not installed or configured. This keeps the menu from getting cluttered. For example, the sensors script checks for output from the sensors command. Lack of such indicates that the hardware sensors haven't been configured with sensors-detect or the required modules haven't been added to /etc/modules. When this happens it does an exit 1 when started with the test parameter. The ddclient script looks for run_daemon="true" in /etc/default/ddclient and the presence of the ddclient executable. The ssh script looks for the sshd process and its description changes if it is found or not. If you write your own, the only limitation to keep in mind is that the description returned should be 45 characters or less as longer ones will corrupt the whiptail display.

Some of the scripts deserve special attention:

shallablud: works with my shall-bl-update v1.3 or later. It forces an update to the Shalla blacklists for DansGuardian.

lynx: requires the Lynx text browser. It does a su to the default admin member (the first one listed in the admin group) before starting. It defaults to the DynDNS.com check IP page. I used Lynx because it has options for lockdown (prevent shell escapes, etc.) that the others don't offer.

wicd: requires wicd-curses. While the netroot script already provides network activation before switching to a shell, it just starts dhclient to get an IP address and nothing else. This was something I requested back in Hardy. It's better than nothing but is rather useless if you only have a wireless connection. Wicd solves the problem but creates another - it conflicts with Network Manager. Luckily the packages themselves don't conflict on Lucid but the daemons do. The script will stop Network Manager before starting wicd-curses (which starts the wicd daemon). To keep this from happening when starting wicd from a root shell you need to either stop Network Manager first or modify the Upstart job configuration to keep it from starting in recovery mode (runlevel S). The conf file also needs to be diverted by dpkg to keep it from being overwritten on updates (and reverting the changes). The commands to do this are:

dpkg-divert --rename --divert /etc/init/network-manager.conf.original /etc/init/network-manager.conf cp /etc/init/network-manager.conf.original /etc/init/network-manager.conf sed -i 's/$.*and started dbus$$).*$/\1\n\t and runlevel [!S]\2/' /etc/init/network-manager.conf

You need to either add a sudo in front of these or open a root terminal with "sudo su". The divert tells dpkg to rename the file and always redirect new installations to "network-manager.conf.original". The file is then copied back to use as a template. The sed expression then adds a condition to not start in runlevel S.

This only solves half of the problem. The Wicd daemon still needs to be prevented from starting during regular operation (runlevel 2) unless you plan to use it instead of Network Manager. Wicd's configuration hasn't been changed to Upstart yet so it's still using init scripts. To disable it do:

mv /etc/rc2.d/S20wicd /etc/rc2.d/K80wicd

This by itself is not enough. If wicd-gtk is installed, it will start when the desktop loads and start the daemon if it is not active. You need to purge it with aptitude or apt-get. In addition, another function somewhere will also start the wicd daemon. The only option I've found is to change the wicd executable, which is just a script that starts the daemon with Python, to not function unless the runlevel is single mode. These commands will make the change:

dpkg-divert --rename --divert /usr/sbin/wicd.original /usr/sbin/wicd cp /usr/sbin/wicd.original /usr/sbin/wicd sed -i 's/$[[:space:]]*exec[[:space:]]\+.*$/[ \"$RUNLEVEL\" = \"S\" ] \&\& \1/' /usr/sbin/wicd

If you make this change you won't have to disable the init script. You will also have to fix the AppArmor profile for dhclient so that wicd can use it (bug #588635). Just add the text in the report before the entry for Network Manager.

One option that isn't listed in the menu is "fsck". This is easy to fix as the script just needs execute permission (bug #566200).

Currently the "resume" option doesn't function (bug #651782).

If you want to prevent the "root" and "netroot" options from providing an uncontested root prompt try my rootlock.

Consider a theoretical example of how this all works with a remote user. They have a problem with X not starting and contact you. They are a considerable distance away and don't have time to ship their PC to you for repair. The system is bootable and they have high-speed Internet so remote access is possible. You tell them how to enter Recovery Mode and how to start wicd. It automatically gets an IP from a wired connection but if they are using wireless they have to select an AP from whatever wicd finds. If they are using Network Manager and their normal wireless connection is encrypted, you will have to set it up beforehand with wicd as SSIDs and keys aren't shared with Network Manager (or the root account which is the one being used here). If they have a dynamic WAN IP address then you have them start ddclient (which also needs to have been configured) or start Lynx and read to you the WAN IP from DynDNS.com. Then they can start sshd. At this point you should be able to access it remotely over SSH assuming that any intervening firewall/NAT routers are forwarding the correct ports. Obviously you should be using key-based authentication with SSH, not passwords. If you can't access it remotely you can still have them perform updates with the dpkg option (also an upgrade), fix the X configuration with failsafeX, or read you the root mail, SMART drive status, and sensor readings (if configured).

Obviously many problems can't be fixed this way but if it saves you a road trip or two it's worth it.

Update: I filed bug #706145 to get these into Ubuntu. Following the normal submit/reject/resubmit/ignore cycle it should be in the repositories within a few years.

A slightly less open Ubuntu recovery mode

2010-12-24T13:34:00.014-05:00

Ubuntu recovery mode is a basic boot configuration for repairing a broken system. In this mode it skips most configuration files and daemons in order to achieve a functioning root prompt. For the security-conscious administrator this itself is a problem.

There have been complaints about unchallenged root access in recovery mode. Ubuntu uses sudo for root access and the root account is disabled via a "*" password. If you forget the passwords of the admins (any user account in the admin group) then this makes it possible to easily reset it.

Originally, recovery mode went straight to a root prompt which wasn't useful to non-technical types. With the addition of Friendly Recovery, a menu is displayed with a list of repair options. The menu is just a Whiptail selection dialog driven by the "/usr/share/recovery-mode/recovery-menu" script which queries other scripts in the "./options" subdirectory. The sub-scripts provide simple repair options like failsafeX, apt-get clean, and update-grub. These are useful to non-technical types for attempting simple repairs to problems. They won't fix complicated problems like gdm crash loops but may save the administrator an on-site visit or two. The root and netroot scripts provide root shell access which is where security becomes a concern, not just because of black hats, but also fools blindly using repair commands like ":(){ :|:& };:" and "rm -rf /".

There are several options for limiting root access.

1. Set a grub password that prevents running recovery mode or editing menu entries. This means the administrator has to make any repairs. If the network is failing then that means on-site.

2. Set a root password with "sudo passwd". The password will then be required to access the shell from the Friendly Recovery screen but this also allows direct root logins during normal operation (although you might not care about that).

3. Disable the shell options in Friendly Recovery. These commands remove the options from the menu and prevents them from reappearing if the friendly-recovery package is updated. This allows users to run the automated commands but makes it more difficult for the administrator to get root access in recovery mode. You'll need to use sudo before these or start a root shell with "sudo su" first.

mkdir /usr/share/recovery-mode/disabled dpkg-divert --divert /usr/share/recovery-mode/disabled/root \ --rename /usr/share/recovery-mode/options/root dpkg-divert --divert /usr/share/recovery-mode/disabled/netroot \ --rename /usr/share/recovery-mode/options/netroot

4. Set a root password only in recovery mode. To do this I wrote rootlock.conf. This is a job configuration for Upstart that is added to the "/etc/init" directory (with root:root ownership and -rw-r--r-- permissions). It is triggered by runlevel changes. Within is a script that when the runlevel is "S" (single) mode, which indicates recovery mode, it copies the password from the first admin group member to the root account in /etc/shadow. In runlevels 2-5, it changes the root password back to "*". This allows root logins from the Friendly Recovery menu if the password of the first admin is entered. In normal operations direct root login is disabled. This makes a lost admin password more difficult to fix but for a capable administrator that is only a minor annoyance.

Don't use it if you have set a root password previously because you want a normal root login available. It will be disabled by this job.

I've tested this on Ubuntu 10.04 (Lucid Lynx) extensively and it seems robust but I'm awaiting feedback on the ubuntu-devel mailing list. Check back for updates.

Disabling unchallenged root logins in recovery mode will not keep a knowledgeable hacker out. This is only possible if you use full-disk encryption like LUKS/dm-crypt for which only the administrator has the key. This will prevent the user from booting with a LiveCD and editing shadow directly but will require the administrator to start the system every time it is powered on or rebooted.

quote-count: A debugging tool for shell scripts

2010-12-16T18:03:00.004-05:00

I've been doing a lot of shell scripting lately with Dash and Bash. Complicated scripts with lots of text handling make debugging difficult, especially when they are being used in sub-shells which obfuscate line numbers in error messages. One of my more common mistakes is an unmatched quote. These can be rather difficult to find so I wrote quote-count, a simple analysis tool that counts quotes in lines.

It just accepts a single filename as a parameter and counts single, double, and back quotes on each line and prints their totals. It prints out a warning if the any of the counts is odd-numbered which may indicate a mismatched quote. It also warns if the line is a comment so you easily ignore those. It isn't brilliant as it doesn't handle escaped newlines, in-line comments, escaped quotes or quotes encapsulated within other quotes. It could be enhanced to handle these cases but it's already saved me a lot of debugging time as is. The output from running it on itself looks like this:

1 Q:0 DQ:0 BQ:0  # COMMENT 
2 Q:0 DQ:0 BQ:0  # COMMENT 
3 Q:0 DQ:0 BQ:0  # COMMENT 
4 Q:0 DQ:0 BQ:0  # COMMENT 
5 Q:0 DQ:0 BQ:0  # COMMENT 
6 Q:0 DQ:0 BQ:0  # COMMENT 
7 Q:0 DQ:0 BQ:0  # COMMENT 
8 Q:0 DQ:0 BQ:0  
9 Q:0 DQ:8 BQ:0  
10 Q:0 DQ:2 BQ:0  
11 Q:0 DQ:2 BQ:0  
12 Q:0 DQ:2 BQ:0  
13 Q:0 DQ:2 BQ:0  
14 Q:0 DQ:2 BQ:0  
15 Q:0 DQ:0 BQ:0  
16 Q:0 DQ:0 BQ:0  
17 Q:0 DQ:0 BQ:0  
18 Q:0 DQ:0 BQ:0  
19 Q:0 DQ:0 BQ:0  
20 Q:0 DQ:0 BQ:0  
21 Q:0 DQ:2 BQ:0  
22 Q:4 DQ:5 BQ:1  # ODD
23 Q:0 DQ:2 BQ:0  
24 Q:5 DQ:4 BQ:1  # ODD
25 Q:0 DQ:2 BQ:0  
26 Q:5 DQ:3 BQ:2  # ODD
27 Q:0 DQ:2 BQ:0  
28 Q:0 DQ:4 BQ:0  
29 Q:0 DQ:2 BQ:0  
30 Q:0 DQ:2 BQ:0  
31 Q:0 DQ:0 BQ:0  
32 Q:0 DQ:0 BQ:0  
33 Q:0 DQ:0 BQ:0

I've tested it with both Dash and Bash on Ubuntu 9.10 and Mandriva 2010.1 so it should work with most systems.

Another typo I occasionally encounter is escaped whitespace at the end of a line. The intent always is to escape a newline but sometimes in my editing I end up with a space or tab after the backslash. These can easily be found with grep:

grep -E -r -n '\\[[:space:]]+$'<filename>

I wanted to add this check to quote-count v1.0 but found that the "while read" loop removes everything after the trailing backslash. Richard Bos sent me a modified version that included the check as a pre-processor utilizing a simple grep trick. I added it in although it used an array which Dash doesn't support.

UPDATE: v1.2 released and link updated. I found some bugs in v1.1 with the TEW check. I also cleaned up the report output a bit.

Reading through the quote-count report for my larger scripts was tedious so I wrote quote-count-query which compares the original source file with the quote-count report and shows the affected lines with two preceding and following lines for context.

Two more utilites for DansGuardian Users

2010-11-24T00:13:00.009-05:00

I reduced the DansGuardian user account and blacklist maintenance hassles with my previous two utilities but while working on whitelisting I found the need for a few more.

In DansGuardian (DG) terms a blacklist bans something, a greylist allows something (overrides blacklisting) but still filters it, and an exceptionlist allows something without filtering (overriding greylists and blacklists). The "something" can be URLs, IP addresses, server names, etc., depending upon the specific list type. Blacklisting a site is easy but blacklisting a specific type of content is very difficult and error-prone. It works the same way as anti-malware utility definitions - if the undesirable items are on the list, and they match a particular requested target, then it's blocked. If not, then it gets through. It's a big Internet out there and trying to block all the bad is rather difficult. "Bad" is also relative and what is bad for one person/religious group/company/government may not be bad for another. Whitelisting has the opposite problems in that you gain strict control over what is available but trying to predict where the user wants to go, determining if that is a safe destination, and maintaining the lists is also difficult.

I found that I needed to use both blacklisting and whitelisting. I use whitelisting with younger children and blacklisting for older. Older children won't put up with strict constraints and will either figure out how to bypass them or simply go somewhere else to browse the Internet. Younger children are easier to keep happy but you still have to spend time figuring out all the web sites they will want access to, preferably with the initial configuration so they're not whining every five minutes about another toy/game/whatever site they can't access.

With DG a "whitelist" configuration is basically a blacklisting of all sites with a "**" in the bannedsitelist file with entries in greylist and exceptionlist files to bypass it. The exceptionlist file entries will enable site access but this is not what you want for allowing a user to browse a particular site because it disables all filtering. Use greylist files instead. This way if there is an offensive part of a site that you didn't know about (or it gets defaced by black hats) then you still have the filters to rely on. The exception lists are useful for sites that are not normally browsed but may trigger the filters inadvertently such as Linux distro repositories using http.

One of the problems with whitelisting is that the user won't necessarily know where they can go on the Internet. To solve this problem you need an index page of some sort. This is the problem I encountered when creating my greylists and I came up with a solution.

I didn't want to maintain an index separately from the greylists so I figured out a way to embed the data in the lists. DG recognizes a # in the list as a comment. I added a comment at the end of each list entry with a Wiki-style link after it. This isn't all that unusual as Debian/Ubuntu did something similar with the menu.lst file in Grub. The comment hides data that isn't relevant to DG but the defined format allows extraction of the data to create an index. Soon after I started adding the links to the list entries I figured out two things - it was a lot of typing and was going to be a very big index. To organize the index better I added a category tag on the end which could be used in the index. The final format is:

<exception link> #[<URL><space><label>][Category:<category text>]

The brackets are required characters. The parsing is somewhat whitespace tolerant but in the Category tag don't leave any spaces between the colon and the category text (sed and regex expressions can be tedious). Example:

gutenberg.org # [http://www.gutenberg.org Project Gutenberg][Category:Books]

To save some typing I wrote add-exceptionlist-url-comments which creates a default URL comment. First it pads the end with tabs (up to 5) to keep it pretty. The default link is made by slapping an http protocol prefix on the exception entry. It then uses wget to try to fetch the default web page and scrape the page title to use as a default link label. This works for most pages and redirects but not those that are using a meta refresh. It finally adds an undefined category tag at the end. Anything in the list that starts with a # is ignored. Note that not every entry will need a link. Some sites you don't want may serve data to a site you want. A lot of USA government sites that are kid-specific will link to media on the main government sites which aren't of interest to kids and just clutter the index. Some web stores also use third-party search services which will need exceptions but not links. In many cases you'll want a link that points to a specific part of the site, not just the server root, so you'll have to edit the defaults.

To create the index page I wrote exceptions-index-page-generator. It looks for the bracket-formatted URLs in the input files. It also builds a list of category tags, assigning a default tag (defined in the script) to any that are missing. It then creates a basic html file with entries separated by category. If a category has more than a certain number of entries (default 5 as defined in the script) it makes two columns to reduce the page length. It doesn't try to normalize category names so they must match in the entries exactly in order to be combined. It ain't pretty but it works. These are both command-line utilities but are rather easy to use.

UPDATE: I updated exceptions-index-page-generator. Version 1.1 adds a category table of contents to the top of the page. It will also make two columns of these if the number of categories exceeds the column threshold.

You can use my greylists to test with and as a base for your own lists for younger children. I haven't performed in-depth checking of these but they look relatively safe. Some of the entries may seem odd but they're intended to aid holiday gift buying. You will also notice that I used html entity codes in the labels for some punctuation as they didn't display correctly in Firefox.

A pair of utilities for DansGuardian users

2010-11-15T21:21:00.028-05:00

Content filtering is a requirement of the home desktop system configuration build I'm working on. Young children are part of the client base so it's mandatory. DansGuardian is basically the only free option available. It's a server daemon so it has command-line configuration only. Once it's running parents don't need to mess with the basic settings but they need to be able to set filtering controls for children without a lot of hassle. On Ubuntu it doesn't come with any blacklists but third-party lists are available. Shalla Secure Services has one of the most comprehensive list that's free for home use but installing and updating it is also a hassle. I wrote a pair of scripts to solve both of these problems.

There are a few options for DansGuardian GUI. Some firewalls like SmoothWall have plug-ins for it. Two popular stand-alone ones are DansGuardian-GUI from Ubuntu CE and WebStrict from Saliby. Unfortunately they both rely on Tinyproxy which has a bug with DansGuardian that prevents many pages from loading. They also drag in FireHOL which I don't need.

Since remote administration is a requirement for my desktop configuration I installed Webmin. A plug-in is available, DansGuardian Webmin Module, which allows easier control than straight command-line methods including a semi-automatic configuration for multiple filter groups. There's one bug with the latter that I had to fix first and the default DansGuardian binary location in the module's configuration was incorrect for Ubuntu (it's at /usr/sbin/dansguardian) but that's all.

When working with multiple filter groups the goal is to have DansGuardian automatically apply the correct filter based on the user account. Correlating user port activity to filter groups is tricky. Since my targeted desktop systems are stand-alone and won't have multiple simultaneous users I chose the Ident method using Ident2. I tried Bisqwit's identd (bidentd) but the version on Ubuntu 10.04 (Lucid Lynx) has a nasty looping bug that is triggered by local queries. Getting this to work only requires activating the ident authplugin and creating the filter groups.

While the module makes configuration easier for the admin, it's still not that friendly for a parent. The filter groups make it easy to set user restrictions based on group membership but DansGuardian filter groups are completely separate from system groups. They can only be changed from the command line or with the Webmin module. I wanted parents to be able to use the standard desktop user administration tool, users-admin (System > Administration > Users and Groups) to assign users to special DansGuardian groups that could then be converted to filter group memberships. There once was a patch for DansGuardian that integrated the two but it's not included upstream. So I came up with a system group naming scheme and wrote dg-filter-group-updater, a GUI tool that automatically creates the filter group list (/etc/dansguardian/lists/filtergroupslist by default) from the system group membership. Installing it is easy. Just copy the script to "/usr/local/sbin" with root ownership and 755 (rwxr-xr-x) permissions. Download this desktop file and put it in "/usr/local/share/applications" with root ownership and 644 (rw-r--r--) permissions which will cause a menu entry to appear in the System > Administration menu. This is for Gnome as it uses gksudo to get root access by you can convert it for KDE by changing the "gksudo" to "kdesudo" or "kdesu" then changing the "Categories" entry for KDE (look at other KDE desktop menu files in /usr/share/applications). For this script to be useful you have to set up the required system groups first and assign users.

DansGuardian references group filters by an index number. The first group is "filter1" which corresponds to the configuration file "dansguardianf1.conf" and is the default. Typically in a multi-group configuration this filter is set to disable Internet access with a "groupmode = 0" setting. By "Internet" I mean "http" as DansGuardian can't really help with "https" (TLS/SSL) or much else. The rest you have to block with firewall rules or a filtered DNS like OpenDNS. The module's multiple group tool is the one named "Set Up Lists&Configs For Multiple Filter Groups" on its main page. Before using it, backup the "/etc/dansguardian" directory as this option only works once and then locks itself out. Restoring the directory is the only way to revert. When you use this tool you will have a few options to chose from. The scheme is up to you (I used separate). I recommend selecting "Use of Default Group" and "To Set Aside Unrestricted Group". I used four groups:

#1 "No_Web_Access" default (filter1, groupmode = 0)
#2 "restricted" (filter2, whitelisted with groupmode = 1 in its conf file and ** in its bannedsitelist file)
#3 "filtered" (filter3, filtered with groupmode = 1 and nautynesslimit = 100)
#4 "unlimited" (filter4, groupmode = 2)

The idea here is that unassigned accounts are automatically blocked by filter1, young children are sandboxed with filter2, older children are filtered with filter3, and adults unrestricted through filter4. Since the restrictions are more about maturity than age the groups don't have names that refer to the latter.

The dg-filter-group-updater script requires system group names to have a specific format of "dansguardian-f#..." where # is the corresponding filter number. Anything after the digits are ignored so you can create more descriptive group names that a non-technical user can recognize in the users-admin tool when assigning members. These groups should be created as system groups (GID < 1000). I created my groups with addgroup:

addgroup --system dansguardian-f2-restricted
addgroup --system dansguardian-f3-filtered
addgroup --system dansguardian-f4-unlimited

Obviously you need to have a "sudo" before these or get a root terminal with "sudo su" first. Since filter1 is the default you won't be assigning users to it and don't need a matching system group. Next you just need to assign users to each group. If you assign the same user to more than one, DansGuardian will use the lower numbered filter in the resulting filter group list. Afterwards just launch the script via the menu item "DansGuardian filter group updater" and enter your admin password. First it will read through the dansguardian.conf file. The file location is set by the "dg_conf" variable in the script and is the only hard-coded value you need to worry about. From the conf file it locates the filter group list file and the number of filter groups. It then starts a new filter list group file (overwriting any existing one). Next it reads through /etc/groups and looks for the "dansguardian-f#..." groups, extracts the users for each, and adds them to the filter group list file in "username = filter#" format. It then restarts DansGuardian. So all a parent needs to do is assign users to groups with users-admin and then launch the script from the menu item to apply the changes.

The script is based on the same code I used for webcam-server-dialog so it will work with any dialogging program installed. Other than that it only uses basic text manipulation tools including grep, sed, and cut. If it doesn't start then launch it from a terminal window or do a "tail ~/.xsession-errors" to see any messages it put out (including those from DansGuardian when it restarts). Most error messages are displayed in a dialog box.

While dg-filter-group-updater solves the basic user administration problem, the lists for filtering (filter3 in my example) still need to be configured. The Ubuntu package only includes basic advertisement-blocking blacklists. Adding third-party blacklists is complicated as you have to merge them in with "Include" statements in the main lists. The lists are organized by categories so you can pick and choose what to filter. Annoying but you only have to do it once if you're using simple filter groups like mine. The problem with blacklists is that they have to be updated often. Shalla Secure Services has some update scripts but they didn't impress me much or did what I wanted. My policy with third-party anything (clipart, CAD libraries, templates) is to keep them separate as references and use other files for customization. To that end I wrote shalla-bl-update. It downloads the list and creates a MD5 file to track the installed version. When it is executed again it checks the MD5 published on the web site against the installed version and downloads the list again if it differs. It has some fault tolerance included as it will retry if the file fails to download or the downloaded file fails a MD5 check. The lists are located in "/etc/dansguardian/lists/shalla" by default. Just download the script from the link and put it in "/usr/local/sbin" with root ownership and 755 (rwxr-xr-x) permissions. It's designed to be started by cron. To have it run daily do "ln -s /usr/local/sbin/shalla-bl-update /etc/cron.daily/shalla-bl-update". It produces no output as cron will Email root whenever anything it runs does. It has a debug mode you can enable by editing the script if you want it to fill your mailbox. It will restart DansGuardian after a successful list update.

Update: I updated shalla-bl-update to v1.2 which adds an optional check for empty system groups. The idea here is that if there are specified system groups used by dg-filter-group-updater, and these groups use the Shalla lists, then these groups should have members. If they don't then there is no point trying to update the Shalla lists. You need to edit the script and set the system_groups variable to the names of the system groups used by dg-filter-group-updater. The grep expression it is using will find partial matches. You can specify "--force-update" to override the check with empty groups.

Update: I've released v1.3 of shalla-bl-update and the link has been updated. Changes: --force-udpate now sets debug=true and clears existing md5. Retries can now be aborted interactively in debug mode. Because of this, the script now uses bash because of the reqirement of the timeout capability of the "read" command. Added "test" parameter for use with Ubuntu Recovery Mode. DansGuardian is not restarted if RUNLEVEL=S (single mode, essentially Recovery Mode). Added --help parameter.

Note: There is a patch by Philip Allison that integrates DG with system groups but the lead developer, Andreas Büsching, has been too busy to integrate it or keep up with maintenance.

UFW application profiles

2010-10-19T21:34:00.011-04:00

Uncomplicated Firewall (ufw) is a front-end to iptables. One of its features are "application profiles" which are INI-style files that contain profile names and ufw settings. This allows packages to include their own firewall settings and make them available to ufw when installed.

Using profiles is relatively easy. To see what profiles are on your system, go to a terminal and enter "ufw app list" to see the names. The profiles are located in the directory "/etc/ufw/applications.d" and the names referenced are the "[section names]" in the files. Note that ufw also references the services list in "/etc/services" for rules. If a section name conflicts with an entry in the services file then the latter takes priority (and ufw warns you every time you use it).

There doesn't seem to be any documentation on the file format and the example files mentioned in the docs don't exist on my Karmic or Lucid systems but the existing files for OpenSSH server and Apache are good examples to determine it from:

[section name] (The identifier that ufw references)
title= (shown in "ufw status")
description= (doesn't seem to be used anywhere)
ports= (the port list)

This is the profile for OpenSSH server:

[OpenSSH]
title=Secure shell server, an rshd replacement
description=OpenSSH is a free implementation of the Secure Shell protocol.
ports=22/tcp

Multiple protocols are specified as "80/udp|80/tcp" with a vertical bar separating them. If just "80" was specified then both udp and tcp are assumed. The port can be a comma delimited list (80,443) or a range with a colon (81:82) or combined (80,443,81:82/udp|8080/tpc). If a range is specified then a separate entry for each protocol is required (81:82/udp|81:82/tcp).

I've been working on a Ubuntu 10.04 deployment configuration for my clients and one of my requirements is a user-friendly firewall for mobile users. While ufw is a command-line application GUIs do exist. Gufw is rather basic and doesn't support application profiles. My clients don't know much about network protocols but they can pick an application by name out from a list. It does list some applications but they seem to be hard-coded. Another GUI is ufw-frontends (ufw-gtk) which does support them. My only complaint with it is that when a profile is used there isn't any way to see what ports it affects - all you see is the profile name. In many cases the title and description are more informative than the profile/section name so I hope the tool shows them in future revisions.

With my deployment configuration selecting the firewall GUI was the easy problem. The hard one was the profiles themselves. Application profiles are easy to make but few packages include them. Many of my clients are gamers and most of the best games have online multi-player capability. This isn't just a Linux problem as all of them want to play games on Wine also. Most of these games are client/server and they need ports unblocked when hosting a private server. The profiles are easy to write but finding out which ports need to be forwarded can be very frustrating. Many gamer-oriented web sites provide aggregated ports lists but most of these are unverified and usually specify way more ports and protocols than are necessary. Developer sites either don't list them or list them without specifying the protocols (TCP/UDP or both) or traffic direction. With home users generally you only care about incoming connections to the server - not outgoing. Since most of the ports used are unofficial and not controlled by IANA many games have port collisions with other games, often because they are based on the same engine (like those from id Software and Epic Games) or the use the same API (DirectX/DirectPlay and GameSpy Arcade). It's very rare that you find an list as accurate or concise as that of Novalogic. Several open-source applications only document their ports the old-fashioned way - in the source code. With some I had to install them and use "netstat -nap" to figure out what was used (which sometimes conflicted with the documentation). Another complication is that several games, like Quake 3, require a different port to be opened for every simultaneous client.

I couldn't really avoid the task so I spent several days writing profiles. You can download them all from here. These are intended to be used as incoming exceptions to a "deny all" rule. Just extract and copy them to "/etc/ufw/applications.d" with root ownership and rw-r-r (644) permissions. Start ufw-frontends and click the "Add Rule" button. In the Destination/Port section select the Application radio button and choose the profile from the list. For applications like Quake 3 that have many possible port configurations I created a few different ones which should cover most situations. Unfortunately the ambiguous profile names in some files are going to be confusing. On a few I tried to make them more readable but fixing ufw-frontends so that it shows the title would be a better solution. Unavoidably there are several duplicates and overlaps with other applications which shouldn't harm anything unless the conflicting servers are both operating at the same time. Many servers can be configured with alternate ports but my profiles only specify the common defaults.

Both ufw and ufw-frontends have limitations that I hope will be addressed in the future. Support for port triggers, dynamic configuration based on the network connection, or just warning when profile port ranges overlap would be helpful. If you add all my profiles to ufw the first thing you will notice with ufw-frontends is that it doesn't handle large numbers of profiles well. To help address the problem I've added a new parameter to the profile file format that hopefully ufw and ufw-frontends can utilize in the future. This is easy to do because INI files don't have much of a standard and ufw ignores everything other than the original ones. The parameter I added is "categories" for classifying profiles. This will allow users of ufw and related GUIs to quickly filter large profile lists. I put in a wishlist bug report about it for ufw. I didn't want to bother creating my own standard from scratch so I used the freedesktop.org menu spec categories since they're already used for organizing desktop menus. I had to break the standard a bit by mixing main categories, usually "Network" with "Game", but this shouldn't be a problem.

The second parameter I added was "reference". This was due to the ridiculous amount of research I had to go through in finding port numbers for each application. Multiple "reference" parameters can exist for each profile, each listing a one-line item. The references indicate the basis for the profile configuration, like "netstat -nap|grep python", to indicate how the port was determined. Mostly these are web site references with a link specified in [URL label] wiki format.

Obviously there are many more servers and daemons to add but generic ones like DirectX, GameSpy, and GGZ Gaming Zone cover many. This brings up a possible optimization - a "meta" or "prerequisite" parameter. Because a lot of games share the same ports due to underlying common code, it would be simpler to define a profile that simply links to other profiles to specify ports. This way a profile could be specified for every individual program but not add a lot of duplicate rules to keep track of.

I only encountered one problem with ufw's profile implementation. It happened when I created a profile for 0verkill. Apparently ufw doesn't allow section names to begin with a digit but I can't imagine why this limitation would exist. Besides that and the way ufw-frontends handles huge profile lists this firewall configuration works well. I don't know if all of the profiles are correct as I didn't have time to test everything. Some may open more ports than a game or application requires and some may not open enough. Feedback is welcome.

Note: The NFS profile (nfs-kernel-server) requires static port mapping. The references in the file will lead you to articles on how to configure NFS this way but I changed the common 4000:4003 ports to 4194:4197 as these aren't in Wikipedia's list or used by anything I could find with Google. There may be a Netfilter module that handles the NFS random port usage better as one exists for saned (nf_conntrack_sane) but I'm unaware of one.

Update: I updated the profiles package to v1.1 which includes a bunch more Linux games and some corrections. NFS profiles have been split into three different files representing the common address ranges. Using static ports for NFS is kind of a hack and I reported bug 688446 about possible solutions.

Update: I released v1.2 of the profiles which made some changes to http due to bug #694894 (which is mostly the fault of Debian and IANA). I also added another parameter, "modules", which specifies the connection tracking module (nf_conntrack) for some protocols. Currently in ufw-gtk some of these can be enabled under "Edit > Preferences > IPT Modules". Having a separate dialog for them doesn't make a lot of sense as they are specific to a particular protocol and you usually need them enabled. The modules act similarly to "port triggering" but are more intelligent as they understand the handshakes of their respective protocols and can identify which additional ports have been negotiated between server and client. I also found out that with NFS v4.1 that the dynamic port problems are being eliminated.

Update: I released v1.3 of the profiles. This is a minor release that only adds Skype, toribash, and webcam-server. The download link has been updated.

A script for auto-configuring saned network connections

2010-10-07T21:17:00.007-04:00

Host-connected image scanners can be shared through saned (part of sane-utils in Ubuntu). It can be run continuously as a daemon or on-demand through Inetd. Basic configuration for either mode is simple and generic but adding the network address to the saned.conf file in CIDR notation is not. When you are setting up systems for multiple clients on different networks and IP ranges, this is a bit of a nuisance. To automate this I wrote saned-subnet-conf which will automatically add an entry for whatever network the host connects to through Network Manager or the ifupdown utilities directly.

Whenever a network connection is made or broken, scripts can be triggered. These scripts need to be located (or linked to) in "/etc/network" in specific subdirectories, the choice of which determines when they execute. Variables are passed to them that can be used for changing behavior based on the network interface, address assignment mode used (DHCP, static, ppp, etc.), and other values. See the interfaces man page for some hints. Network Manager executes these scripts with "/etc/NetworkManager/dispatcher.d/01ifupdown" which uses the run-parts utility. Network Manager does not trigger the "pre" directories due to a design decision.

To install the script, first download and extract the script, then put it in "/etc/network/if-up.d". You'll need to use sudo or have a root terminal for the copying (and most of the rest of the commands). Make it owned by root:root with rwxr-xr-x (0755) permissions. Whenever a network interface is brought up by ifup or Network Manager the script will execute. It uses scanimage to look for scanners and if any are found it will then use the ip command to get a CIDR version of the network address and produce an entry for saned.conf if one doesn't already exist. The last part is important as the script will add an entry for every network the host connects to. If you want to block a particular network, let the script add it to saned.conf and then comment the entry out with a # as the script won't add it again if it finds it anywhere in the file. Make sure you restart saned anytime you edit saned.conf (see below). If you want to keep the script from adding entries in relation to a particular network interface you'll have to edit the script and have it exit based on the IFACE variable. Look at the "$METHOD = loopback" entry for a rough idea. If you enable the VERBOSITY=1 entry the script will generate a log file in /tmp that includes all the variables. Currently the script only supports IPv4 addresses as my network doesn't use IPv6 so I can't test it.

Setting up saned is rather easy. During installation you have the option of running it as a daemon. To enable this later use "dpkg-reconfigure sane-utils" and indicate "Yes" to the standalone server option, or just edit the "/etc/default/saned" file and set "RUN=yes". The server daemon will start automatically at boot but you can start (or stop, restart) it manually with "invoke-rc.d saned start" or "/etc/init.d/saned start". To see any messages from saned use "tail /var/log/daemon.log".

To have saned start automatically when a client connects, indicate "No" to the standalone server option or set "RUN=no" in the default config file. Then add (as per the man page) the required entry to "/etc/inetd.conf" if it doesn't already exist. You can use a text editor but a safer way is with the update-inetd utility with "update-inetd --add "sane-port stream tcp nowait saned.saned /usr/sbin/saned saned". If you watch the log (tail -f -n 20 /var/log/daemon.log) you will see saned start and stop automatically whenever a client connects. Don't run a daemon and have an Inetd configuration active at the same time as they will conflict over network port access (6566 by default). To disable the Inetd entry use the command "update-inetd --disable sane-port".

To configure clients to use the server just add the server IP address or host/domain name to "/etc/sane.d/net.conf" and start whatever scanning program you want to use. You can get a list of available scanners with scanimage -L but note that neither saned or scanimage supports scanners connected via a parallel port.

On Ubuntu 10.04 (Lucid Lynx) and some earlier versions access to scanner devices isn't handled correctly for anyone other than standard users (UID=1000+) on the host. As a workaround you can use my Scanner Access Enabler to correct the permissions until reboot. In the future, scanner network access may be handled by Avahi but it doesn't work with Karmic or Lucid due to another bug.

Update: Forgot to mention that scanimage is used to look for scanners first before adding a saned.conf entry.

Scanner Access Enabler

2010-10-03T18:17:00.008-04:00

There is a problem with scanner device permissions on Ubuntu. Regular users (UID>999) can access libsane applications like Xsane and Simple Scan without problems. Linux Scanner Server, which is running in Apache as www-data, can't access them without a chmod o+rw on each scanner device. Nobody seems to know how the permissions work so this has to be fixed manually in a terminal. This is not n00b friendly so I created a GUI application that automatically changes the permissions of every scanner device.

The application relies on scanimage and sane-find-scanner utilities to identify scanner device ports then simply does a chmod against all of them. It supports USB, SCSI, and optionally parallel port (-p parameter) scanners and has been tested against the same ones I used for my LSS patch. It uses the same universal dialog code as webcam-server-dialog so it should work with almost any desktop environment.

To install first download the archive and extract the contents. Move the script to "/usr/local/bin/scanner-access-enabler" and set it for root:root ownership with rwxr-xr-x (0755) permissions. Copy the destop menu entry to the /usr/local/share/applications directory with root:root ownership and rw-r--r-- (0644) permissions. You may have to edit the desktop file as it uses gksudo by default. On KDE you may want to change the Exec entry to use kdesudo instead. If you specify the -p option on the Exec line you may have to quote everything after gk/kdesudo. If you don't have one of the GUI dialoger utilities installed and plan on using dialog or whiptail then you need to set "Terminal=true" else you won't see anything.

On Ubuntu the menu item will be found under System > Administration. If you want users to be able to activate scanners without a password and admin group membership, you can add an exception to the end of "/etc/sudoers" file. Simply run "sudo visudo" and enter the following:

# Allow any user to fix SCSI scanner port device permissions
ALL ALL=NOPASSWD: /usr/local/bin/scanner-access-enabler *

While you can use any editor as root to change the file, visudo checks for syntax errors before saving as a mistake can disable sudo and prevent you from fixing it easily. If you mess it up, you can reboot and use Ubuntu recovery mode or a LiveCD to fix it.

Update: I released v1.1 which adds filtering for "net:" devices from saned connections. This didn't affect the permission changes but made for a crowded dialog with both the raw and net devices shown.

Update: v1.2 adds a non-interactive/silent mode activated through a "-s" parameter.

A patch for Linux Scanner Server v1.2 Beta1

2010-10-02T21:22:00.022-04:00

I just spent several days testing, fixing bugs, and adding features to Linux Scanner Server v1.2 Beta1. LSS is an easy way to share a non-networkable scanner through a web server. While the interface doesn't allow cropping like Xsane or Simple Scan it does support multiple file outputs, printing, and OCR through Tesseract. Development has stalled with the beta and I encountered some bugs when testing it on Ubuntu 10.04 (Lucid Lynx). Instead of complaining about it, I fixed them.

Bugs fixed/features added:
Noise in Apache logs caused by unquoted variables and non-critical stderr outputs from ls and rm.
Adding scanners would fail if the scanner name included a forward slash.
Multiple scanner support broken due to a lack of newlines between entries in the scanner.conf file.
No support for scanners connected via parallel-ports.

I wanted to try to break the beta before deploying it to my clients and I did - as soon as I connected a second scanner. I decided to fix the bug even though none of my clients have more than one attached to any given system. I just happen to have a bunch of them on hand and, thanks to a local tech recycling center, I added a few more. Sane supports scanners connected to parallel ports but LSS doesn't so I decided to fix that, well, just because. Yes - I went out and paid money for more scanners including an obsolete parallel port Mustek model just to fix LSS.

The deciding factor in doing this was that LSS is based on a shell script and a lot of sed scripts. Shells scripts are about the only programming language I know to any depth (and Applesoft BASIC). Some of the regex/sed stuff still throws me but I had help from some of my LUG mates. These are the scanners I tested with (and tested simultaneously):

AGFA SnapScan 1212U (snapscan:libusb:002:003)
Brother Industries MFC-440CN (brother2:bus4;dev1)
Hewlett-Packard ScanJet ADF (C7190A, identified as 5200C) (hp:libusb:004:002)
Hewlett-Packard ScanJet 4470c (rts8891:libusb:004:003)
Hewlett-Packard ScanJet 6100c (C6260A but identified as C2520A) (hp:/dev/sg5)
Microtek ScanMaker E3 (microtek:/dev/sg3)
Mustek 600 III EP Plus (/dev/parport0)
UMAX Vista-S8 (umax:/dev/sg4)

Fixing the multiple scanner support was a pain. LSS relies on scanimage for all scanner functions. Getting scanimage to provide a newline at the end of the device list was trivial but the message printing function for the web page doesn't tolerate them and they all have to be converted to HTML breaks. Forward slashes in the model names from scanimage also required escaping but not anywhere else (like in the device paths). This got into sed loops which are really hard to do.

Adding support for scanners on parallel ports was also difficult. They have to be defined manually in the sane config files (/etc/sane.d/*.conf) but scanimage doesn't report them regardless. The sane-find-scanner utility does find them and will indicate what brand is on which port but no additional details like the model name. Since sane can use auto-probing to find which parallel port the scanner is on there is no deterministic way to use the information from sane-find-scanner and the sane conf files to indicate a specific model. The only solution I could come up with is to manually specify parallel port scanners in a separate "scan/config/manual_scanners.conf" file and then merge it after the rest are detected. The format is the same as for scanners.conf but the value for ID needs to be specified as %i (same as the device entry in the format line for scanimage). The modified LSS index.cgi script will replace it with an auto-incremented value when merging. The NAME= value doesn't matter but forward slashes have to be escaped with backslashes and anything longer than 30 characters will be truncated in the pull-down list on the Scan Image page.

Setting up a parallel port scanner is a bit confusing. The Mustek model I used was configured in /etc/sane.d/mustek_pp.conf simply by uncommenting the line "scanner Mustek-600-IIIEP * ccd300". The second parameter is the name. The third is the port with an * indicating autoprobing which in my case became /dev/parport0. The last is the actual driver. With scanimage the device is not specified by the port but rather the backend driver and then the name. With the settings I used it became "mustek_pp:Mustek-600-IIIEP" (also specified for the "DEVICE=" value in manual_scanners.conf). If only the backend is specified scanimage will default to whichever is enabled in the conf file. I only have the one parallel port scanner (the ScanJet 4470c has USB and parallel but there's no driver for the latter) so I don't know how it handles multiple ones configured in the same file/backend.

There are still bugs in LSS. The most obvious one is a fault with the "Print_Message" function. There are several page updates that don't occur, mostly the "Please wait" ones that are supposed to display during scanner detection and image scanning. I don't know enough about the interaction between javascript and the browser to identify if it is a bug in the code or an architectural problem with the page design.

Another bug is with the scanner names. As you can see from my list above, some of the scanners are not named correctly. It may be that the model reported is the base one that the actual model is compatible with and sane just isn't more specific than that. LSS just uses whatever scanimage reports. This isn't a major problem as most systems will only have one scanner.

A third problem is with the scanner driver options that LSS specifies - basically none. Some scanner/driver combinations require specific options to be specified else the scanner has problems. The only one I encountered with the models tested was that the default resolution of 200 was unacceptable to one of them so it was downgraded to 150. These errors show up in the Apache logs (/var/log/apache2/error.log) but only refer to index.cgi and not a specific point within the file. I'm not sure how this bug could be fixed. Parsing the options out of the sane conf files may work but different versions of the same base model may require different settings.

Future enhancements that would be nice are cropping and internationalization support but that's more than I'm going to take on. My LUG mates also suggested using anything other than shell scripts.

To use the patch, first download and extract LSS into your web server data directory (/var/www/scan). Then download the patch archive, extract it into the "/var/www" directory and apply with

patch -p1 --directory /var/www/scan --input=/var/www/scan_1.2_Beta4.patch

Just reload any browser window that has the old version loaded to make the new one active. Restarting the server is not necessary.

LSS is GPL 2 but it's not clear in the package as the author didn't follow the recommended method for applying the terms.

Note - there's a nagging problem with Ubuntu in that LSS can't access any scanners due to device permissions. It runs as user www-data but the old scanner group no longer exists so devices need chmod o+rw applied manually. For regular users (UID >1000) it seems to happen automatically but nobody seems to know how that works. I wrote Scanner Access Enabler to solve the problem.

Update: If you also have saned configured for scanner sharing then duplicates may be detected from both the raw devices and saned shared versions. If you don't want the ones from saned then comment out "localhost" in the "/etc/sane.d/net.conf" file and restart saned.

Update: I and pqwoerituytrueiwoq have been making more improvements to the beta. You can follow along and download updated files from this thread at the Ubuntu forums.

20110207 Update: I and pqwoerituytrueiwoq made a bunch of fixes and I've released 1.2 Beta 4 of the patch. The links and instructions above have been updated. It is a recursive patch so it will affect several files. You still need to add the favicon for it to "/var/www/scan/inc/images". I performed a feasibility study of adding proper preview, settings selection, and cropping. I found that the difficulty of adding them to the existing code base is extreme, even though some are needed to get LSS functioning correctly. For example, the Brother MFC-44CN doesn't scan because the modes that LSS uses (like "Color") are hard-coded in the html and don't match up with what the Brother driver offers. Because of these problems (and my lack of time) I've ended my involvement with the project. For my needs Beta 4 functions adequately. I also found another scanner project, phpSANE, that seems to have a better code base on php although it has many limitations otherwise.



webcam-server-dialog: A basic front-end to webcam-server
2010-09-23T20:56:00.013-04:00
I'm working on a new Ubuntu configuration to deploy for friends and family.  One of the capabilities I wanted to add was simple remote webcam viewing.  I saw an article about webcam_server.  While it's old and only does images it met my requirements.  The primary limitation is that it's a command-line application.  It can be launched from a XDG desktop file and will default to /dev/video0 but on systems with more than one video device a terminal is needed.  I did spend some time testing streaming with VLC but it doesn't show a device list for selection either and it's streaming configuration dialogs are confusing at best.  To deploy webcam-server I had to make it more friendly which meant making a video device selection dialog for it.  My current programming hammer is Bash shell scripting (Applesoft BASIC was the other option) but it's not enough for GUI design.  To add that capability I turned to what I call "dialoger" utilities that can produce GUI dialogs and provide feedback to command-line applications.
When I started this project the only dialoger I knew about was dialog which is text-based and I needed something that would run in X.  From researching alternatives to dialog I found xmessage which led to gxmessage and eventually Zenity.  Plenty to choose from and all different.  Now I had another problem - which one to use with each desktop environment?  I normally use Gnome but some of my clients use XFCE or LXDE.  There is also the possibility of a KDE user in the future.  While Ubuntu includes Zenity by default, which would work with XFCE also, a GTK application isn't the best choice on KDE.  I could use kdialog but either I had to make custom versions of the script for each environment, select the dialoger with a script parameter, or try to select it dynamically.  In the great tradition of overdesign I chose the latter.
After spending several days solving a 5-line problem with 300+ I ended up with webcam-server-dialog.  It supports dialog, whiptail, Xdialog, xmessage, gxmessage, kdialog, and Zentiy.  Because it's rather generic it can be expanded to support more without a lot of effort.  It looks for processes that indicate a particular desktop environment or window manager, uses a built-in priority list of dialogers for that environment, checks for availability, then uses the best available to provide the GUI.  The core of this script is nothing more than "ls /dev/video*" dumped into an array but the focus here was eye-candy.  Getting this to work with all of them was difficult as they all have different command-line parameters, behaviors, and bugs, even between those that are supposed to be clones of each other.  Some use exit return status for indicating button presses, some stdout, some both depending on the mode.  The fun ones were dialog and whiptail which use stdout for drawing the screen and stderr for indicating list choices.  There are a lot of comments and commented-out debug lines if you want to use this with your own projects (GPL v3).  It could be useful as a front-end to other programs that lack selection lists like xawtv.
To use it, install webcam-server then put the script in /usr/local/bin with root ownership and rwxr-xr-x permissions.  I also made an XDG destop menu item for it.  Put webcam-server-dialog.desktop in /usr/local/share/applications.  For an icon I used the one from camorama and just copied /usr/share/pixmaps/camorama.png to /usr/local/share/pixmaps/webcam-server-dialog.png (make the directory if it doesn't exist).  You can add parameters to the Exec line in the desktop file but the run dialogs always reference port 8888 (I'm too tired of working on this to make the text dynamic) and quotes don't pass through well so the caption format had to be hard-coded in the script.
In addition to direct web-page access a Java applet is included.  It works well since it can automatically reload the image at a selectable interval.  It has one really bizarre limitation - it will only connect from localhost.  Apparently the developers decided to limit it that way instead of implementing remote authentication but you probably could use it from within an ssh connection.  This security feature has a bug in it's implementation when resolving the hostname so it requires the IP (127.0.0.1).  I created a custom PHP web page that works around this.  Just rename it to index.php and put it in the client directory on your web server (/var/www/client by default) along with applet.jar from the /usr/share/doc/webcam-server directory and link to it from your default home page.  You'll need PHP support installed (php5 metapackage).
I also tested it in Mandriva 2010.0 which worked but with one problem - the webcam-server executable is named webcam_server so just replace all instances of one with the other in the script.  You'll also need v4l-info which is in the xawtv-common package.  If you are using the firewall you have to add "8888/tcp" to the allowed ports for remote access.



Your Linux system keeps falling and it can't get up
2010-09-03T15:44:00.026-04:00
Once in a while a Linux PC technician will encounter a system that has problems with lockups (a.k.a. hanging or freezing).  Sometimes it is failing hardware but other times it's a software problem.  Here are the common causes for this and how to identify which is the source of your problems.  While I predominantly use Ubuntu (and some Mandriva) these tests are valid for most any distribution.
1.  A kernel crash or panic is rare but generally fatal.  As any PC tech knows, the first test is seeing if the Caps Lock, Num Lock, or Scroll Lock LEDs change state when the corresponding keys are pressed as this is performed by the PC, not the keyboard controller.  If they don't change then you know you've got a freeze problem.  If the keyboard lights are flashing repeatedly during the freeze then it's a panic.  More severe problems can prevent the kernel from failing gracefully enough to even do that.  These failures can be caused by fundamental compatibility problems with a kernel module and a critical piece of hardware (like a BIOS with a broken ACPI implementation), a device that was given commands it didn't like and has stopped responding (like a video GPU) or just failing hardware (bad RAM, overheating CPU, and loose PCI or AGP cards).
With hardware problems it is best to start by opening the case and blowing the dust out since that is the source of most overheating problems.  After cleaning a few systems you'll understand why you should charge extra for smokers, pet owners, and homes with shag carpet.  I find it easiest to use an air compressor with a tank and a blow-off nozzle to do the job as canned air is too weak.  Leave the system plugged in (but powered off) to keep it grounded as air streams can produce static electricity which can damage electronics.  With modern systems the only hazardous voltages are in the power supply so if you don't insert any metal objects into that you won't get shocked.  Try spinning all fans with your finger or a plastic probe to see if they turn freely without resistance.  Replace any that drag with anything other than a sleeve-bearing fan (like ball or fluid bearings).  Power supply fans can be replaced but it requires soldering or swapping connectors as there isn't a standard connection for their internal fans.  While it's fun to use the air nozzle to spin up the fans to 100K+ RPM it's bad for the bearings.  They also get cleaner when you hold the blades stationary (I use the end of a big nylon tie).  Start with the power supply and then the CPU and work your way around to the front case vents.  Keep wires tied up and away from the fans so they don't jam the blades or block airflow.  Check for cards and memory modules that are not fully seated in their slots and partially-connected drive cables.  If possible, remove heat sinks from CPUs and other chips and check for adequate heat sink grease (should be an even but thin layer across the entire mating surface).  Check that the chips are properly seated in the sockets and that the heatsink is pressing down evenly on them else they may tilt and lose contact.  Check for failing capacitors.  If you find any bad caps it's probably easier to replace the motherboard unless you have good soldering skills.  Use your eyes and nose - if something looks or smells burnt then it probably is.  Keep in mind that power supplies often have stronger "electrical" smell to them due to hand-soldering during manufacturing.
Laptops are harder to clean.  Most can be opened by prying off the top bezel around the keyboard, usually starting with the section enclosing the display hinges.  Some have screws and some just latch.  Then the keyboard can be removed and the top mounting plate.  There are how-to disassembly videos on the Internet for popular models that are often modded by hackers.  Some laptops have externally removable heat sinks for easy cleaning.  Just because it's easy doesn't mean that users clean them.  I once recycled a high-end Sager laptop (about $5K USD) that had overheated and failed.  The heatsink had plugged with lint and it kept shutting down so the parents gave it to their kids to play with.  The kids laid it on their bed and had it running (bottom fans so no airflow whatsoever) and it overheated enough melt the case around the heatsink.  Made me sick to throw it out but the motherboard wasn't practical to fix after that.  Modern CPUs will reduce their clock speed when overheating but can't reduce power dissipation entirely and can still overheat and fail when running at minimum levels.
Intermittent failures are harder to diagnose so continuous monitoring with hardware or software tools is needed.  Hardware temperature monitoring can be performed with a cooking probe, thermocouple meter, or a dedicated PC temperature monitor that mounts in a drive bay.  The CPU, GPU, power supply fan exhaust, and hard drives are the ones to focus on.  Temperature limits for devices vary.  CPUs and GPUs can often hit 60°C but 50°C is rather hot for a hard drive.
For fan monitoring you can leave the case cover off and keep an eye on them or install a PC fan monitor/controller with a display which mounts in a drive bay (and often includes temperature monitoring probes).  Thermostatically-controlled fans will vary a lot but if you are having an overheating problem due to inadequate speed from a non-faulty fan then it may be too far out of the primary air stream to have an adequate response time.  Better models have adjustable thresholds or remote sensors but the best solution is one controlled by the motherboard via a 4-pin PWM fan connector.  Be careful here - I burned out a CPU fan controller when I used a CPU fan that consumed more current (amperes) than the motherboard's controller could handle so check the specifications before plugging it in.  I had to convert mine to a drive connector which meant it ran at maximum speed and sounded like a vacuum cleaner.
A voltmeter is useful for monitoring power supply voltages on the connectors under various loads.  Generally supply voltages should be within 10% of the stated voltages on the power supply label.  CPUs and GPUs usually have a local regulator on their boards as they need voltages that differ greatly from the normal 12/5/3.3 volts that most power supplies provide.  The BIOS often has control over the CPU voltages and configuration so a bug in the BIOS (or incorrect manual settings) can cause erratic lock-ups by making the CPU unstable.  Usually a CMOS reset or BIOS update can fix this.  One way to test for an unstable or faulty CPU is to underclock it via manual settings in the BIOS (or jumper or switch settings on really old motherboards) and see if stability improves.
To verify what the CPU needs for power and clock rates you first need to identify exactly what one you have as manufacturers have many versions and steppings and their requirements may differ.  To see what you have use the command "less /proc/cpuinfo".  Use that information to search for exact specifications and compare it to your system.  Pay close attention to power requirements as some motherboards, even with the same socket, can't handle some CPUs.  This results in unstable CPU voltages and intermittent failures, especially under heavy loads.  This problem tends to occur with long-lived socket designs where the CPU family is expanded to include models with higher power requirements (essentially changing the motherboard requirements) that earlier motherboard designs can't meet even though the CPU fits in their sockets.  I've damaged a few boards that way.  Heatsinks and fans also need to meet the requirements of the CPU.  Mass-market PC systems usually have very little power margin between the shipped CPU requirements and the system cooling capabilities so failing to upgrade them both can result in instability.  Many of these cheap systems use a ducted case fan for cooling and just replacing a failed one requires tracking down the specifications for the fan and finding a replacement that matches in airflow (CFM) and features (connector type and thermostatic control).  Standard CPU fan/heatsink combos usually can't be used as they don't fit in the case or the motherboard lacks mounting holes for them.
Most modern motherboards have built-in sensors as do CPUs, GPUs, and storage devices.  These can be queried by software for status information, monitoring, and logging.  Some BIOSes report the sensor values and error conditions and advanced servers often have separate hardware modules for remote monitoring of them.  The standards that the sensor systems conform to are imprecise so custom drivers and algorithms are needed by external software for each implementation.  Software tools include lm-sensors and smartmontools.
The lm-sensors utilities report what thermal/fan/voltage sensors you have on your motherboard (if available and supported) and their current status.  You first run "sensors-detect" to identify what kernel modules are needed and have it add them to /etc/modules and reboot (or just load them with modprobe).  Then just run "sensors" to get the current status or use a graphical application like the Gnome Sensors Applet, KSensors or the XFCE4 Sensors panel plug-in.  Note that wildly extreme readings may not indicate a fault but rather an unused sensor input or an unsupported implementation.
Most modern hard drives and SSDs have a monitoring and diagnostic system called SMART which can be access with smartmontools.  While SMART can tell you about problems, it is not good at predicting failures.  You use the smartctl program and specify the storage device to query.  For most systems the primary storage device is named "sda" by the kernel so the command would be "smartctl -a /dev/sda | less".  Most modern drives report temperature, log errors, and have built-in self tests that smartctl can activate.  While the underlying registers on the drives are well-defined, what they represent is not so conversion data is needed by smartctl to interpret the values.  It will tell you if it recognizes the model or not.  The obvious status to check is the "overall-health self-assessment test" which tells you if any of the register values exceed an alarm threshold.  More specifically the parameters of type "Pre-fail" are important.  Also note the "worst" temperature value as it could indicate a prior significant overheating incident which is most likely to occur under heavy load (like during a backup or a RAID rebuild).  Graphical tools include GSmartControl and Palimpsest disk utility in DeviceKit (a.k.a. gnome-disk-utility) but root access may be needed by them.  Another is hddtemp which only reads the temperature but has a daemon that can be monitored through the sensor monitoring tools mentioned above.
RAM can be tested with Memtest86+ which is installed in Ubuntu by default.  Reboot and hold the left Shift key down before Grub loads and starts booting.  You'll get the Grub menu with Memtest86+ listed.  You can also download a bootable ISO or USB image from the Memtest86+ site to test with.  In the early days of PCs the memory had parity checking but modern RAM doesn't so the only way to identify a failure is by using a RAM test.  If you are worried about memory problems then get ECC memory.  This costs only a little more than standard RAM but the motherboard has to support it and it can reduce performance and limit overclocking.  With ECC memory the BIOS can provide much more memory diagnostic information and testing.  For example, wiping unused memory locations is a standard process that is performed at a user-definable interval to see if any bits changed state by themselves.  Servers often use ECC memory but usually these are registered ECC memory modules which are sometimes called "server memory".  The "registered" aspect isn't a certification - it's a signal amplifier built into the module for use in systems that have more modules than the motherboard's northbridge can communicate with directly.  These are not compatible with standard memory or motherboards that use it.  RAM memory modules have an SPD device that indicates it's specifications.  To read it (and other BIOS information) use the command "dmidecode | less".  Another source of intermittent memory problems is faulty configuration by the BIOS, either manually by the user or a faulty automatic configuration.  A CMOS reset or BIOS update can often fix this.
Diagnosing a freezing system is difficult since you can't check log messages easily with a frozen system and the logs are often truncated as a result of it.  The kernel (and Grub) have built-in remote communication options which can help with this.  These out-of-band remote connections can be made through a serial console or with Netconsole and another system.  A serial console can be used like an SSH connection but requires a hardware RS-232 serial port which is rare on modern systems.  On Ubuntu 10.04 (Lucid Lynx) there is a Memtest86+ serial console configuration already in the menu that can be used to test memory remotely but it's probably more useful for headless (i.e. no display) servers.  Netconsole requires a network connection (it uses UDP) and another system running a syslog server.  For kernel crashes the Linux Kernel Crash Dump tools can be used to obtain crash data that is useful for diagnostics or reporting kernel bugs but I haven't used it yet.
Check the kernel messages and logs with "dmesg | less", "less /var/log/kern.log", "less /var/log/syslog".  There are many different log files  including compressed backups of previous logs.  Some require you to be root to access them.  With Ubuntu you just add "sudo" before the commands or just get a root login with "sudo su".  Midnight Commander's internal editor is helpful for reading logs including the compressed ones.  The built-in editor is not the default in Ubuntu - you have to enable it within MC with F9 > Options > Alt-I > Alt-S (use Esc 9 instead of F9 when connecting through a serial console).
Most distros have boot options that can be issued through the boot loader to the kernel to change its behavior or deactivate specific functions.  Ubuntu and Debian have many but every distro has it's own.  These can help to isolate problems or provide long-term stability when added permanently to the boot loader options.
2.  An input error resulting in the loss of keyboard/mouse control acts like a freeze but isn't.  The first clue is to see if there is any screen activity at all (most desktops at least have a clock applet running).  Hardware causes include a faulty peripheral, USB hub, PS/2 port, or KVM switch.  With PS/2 ports a failure with one device usually prevents the other from working.  A simple test is to plug in a USB mouse or keyboard and see if they work during the freeze.  When a kernel bug is responsible the keyboard works in the BIOS setup and Grub menu but fails during boot (I've had problems with a bug related to an Intel i8042 PS/2 controller).  These can be intermittent between boots but once it's working during a session it usually stays working.  It can also be a bug in X.org if they work in a tty terminal but not in X (as when booting into Ubuntu's recovery mode).  I've encountered a freezing problem that affects only the mouse.  It often occurs when an OpenGL game crashes.  Besides restarting X with the keyboard (knowing the menu hotkeys helps here), I've found that launching the game again and then exiting usually fixes the problem.
Check your X session logs during the freeze by switching to a tty or connecting remotely through a SSH or serial console connection.  Login, then do "less /home/<username>/.xsession-errors" and see if there are any crash messages from running applications.  Most desktop applications will log messages there if they don't have their own log.  If you have no control at all, reboot but don't log in to a graphical session (at the display manager login screen) as the session log will be overwritten as soon as you do.  Don't just hit the reset or power button when rebooting - try the Magic Sysrq keys first or connect remotely and issue a "reboot" or "init 6" command.
An example of another traumatic but non-system freeze is when Nautilus hangs as this makes it difficult to do anything with the Gnome desktop until killed (it usually restarts automatically just like Windows Explorer).  A Nautilus error would show up in .xsession-errors while a crash would also show up in the kernel logs.  If the session log is rather big, making it hard to isolate messages related to a particular application, you can open a terminal window and try running the suspect application from there as any error messages would show up in that window instead.  You can also capture the messages to a file by copying the screen or using shell I/O redirection to a file which is helpful when submitting bug reports.
X.org input driver errors will show up in it's log at /var/log/X.#.log where the # represents the instance that was running.  Normally it's X.0.log unless you have multiple sessions running like multiple X logins or a non-Xinerama dual-head configuration.  A different session ID could also be used if X crashes back to the display manger login screen and it thinks another session is still running (due to a leftover lock file) when you login again.
3.  Outright X.org crash.  When it involves screen corruption it's obvious but that symptom isn't always present.  Sometimes this happens when switching to or from a tty terminal or when an OpenGL application is running full-screen.  Sometimes it happens with the display manger at the login screen.  If the keyboard lights don't toggle then try switching to a tty.  If that doesn't work then try killing X with left-Alt+SysReq+K (or Ctrl+Alt+Backspace if enabled).  If that doesn't do anything either then try a remote connection (or just pinging it).  If that also fails then you are facing a kernel crash (which can be caused by a misbehaving video device due to integration between X, the drivers, and the kernel).  If you do get remote access then save and review the logs including that of the display manger (/var/log/gdm/:0-greeter.log).  These crashes are usually caused by video driver problems.  In Ubuntu 8.10 through 10.04 (and several other distros) almost any Intel 8xx series graphics device will cause problems.  There is a lot of architectural changes occurring with video which involves the kernel, X.org, and DRI and there has been a lot of breakage.  Some drivers are not keeping up with the changes and some latent driver bugs are being discovered.  The older Intel devices are currently the worst (of the "supported" devices) even though Intel is the one of the companies that is pushing these changes and has engineers working on it.  But not all video crashes are the fault of the driver.  Some may be kernel bugs with the motherboard chipset that the video GPU is triggering.  This was a common problem with AGP ports and video device manufacturers like Nvidia wrote their own AGP modules for specific chipsets.
To save time, instead of analyzing the logs for something that indicates a driver problem, search the Internet for distro bugs relating to the one you have.  Identify your graphics device with "lspci | less" or "lshw | less" and then search with Google for the device part number and the distro like "Ubuntu 10.04" or "Lucid".  Check the release notes for known problems and possible workarounds.  With Ubuntu there are usually workarounds in the Ubuntu help wiki.
4.  CPU overload.  Some process is hogging the CPU and slowing everything down to a crawl.  More common with single cores but can still happen with multicores due to memory bottlenecks.  If you can't get a graphical process management tool like Gnome System Monitor to load then switch to a terminal and use the "top" command to see who the culprit is (probably Flash but X.org driver faults can cause overloads without a outright freeze or crash).  Identify it by process ID and kill it using top's built-in kill option (press k).  You can also list processes with "ps -A" then use "kill -s <signal> <process number>".  If there are multiple instances of the same process then use "killall -s <signal> <process name>".  The signal is 15 (terminate) by default which means "ask nicely".  If that doesn't work then use 9 (kill) which isn't as friendly.
5.  I/O overload.  Something is hogging the hard drive/SSD which can slow everything down to the point of being non-responsive.  You can usually identify this by the hard drive activity LED being lit continuously.  You can narrow down the list of processes responsible with the lsof command with "lsof | less" but you'll find the output can be overwhelming.  If you know which file it is then you can identify the process responsible with fuser.  Interactions between Firefox's database and the EXT filesystem can cause I/O overload intermittently but it's not as often with newer versions.  A lack of storage space can cause it if applications that are trying to write to the disk don't handle failed writes well, especially logs and temporary files.  They may hang and start hogging the CPU also.  To check available storage space use "df -Th".
6.  Memory hogging.  Some process is eating memory and increasing in size.  If the RAM is used up then the swap partition is used which can manifest itself as #5 also.  Eventually the system runs out of memory and the kernel starts killing processes to fix it.  Identifying the culprit is essentially the same as for CPU hogs.  Use the command "free" to check memory and swap usage.

This is only the start of the diagnosis.  Once you identify the source of the problem then you can try to find a workaround, file bug reports, and test patches.  This all seems rather complicated but after you've fixed a few dozen systems you eventually recognize specific symptoms and behavior patterns right away and can quickly narrow down the problem.  What differentiates real technicians and hackers from the amateurs is the stubborn resolve to find the problem.



Duplicating subsets of package selections between systems
2010-05-15T16:23:00.027-04:00
I'm building a pair of Ubuntu systems for kids.  For a variety of reasons, including lack of time and hardware problems, this has taken far longer than expected and I ended up with one running 9.10 (Karmic Koala) and the other 10.04 (Lucid Lynx).  Since the Karmic system has the most testing effort into it (reviewing games for stability and kid appropriateness) I needed a way to duplicate the selection of games on the Lucid system while filtering out everything else since some packages are release-specific.
Using Synaptic it is possible to export (File > Save Markings) either a list of packages that have changed selection state but the changes haven't been applied yet or optionally a list of all packages and their status.  While Synaptic can filter displayed packages by repository or type (the "section" parameter in the deb INFO file), these have no effect on the markings export.
The apt-get command (or dpkg directly) can be used to create a full list of packages but to produce a filtered list you need to use aptitude.
Aptitude has a search function that can be used to filter based on almost anything in a deb package INFO file.  For example, to get a list of games available and save it to a game_selections.txt file:
aptitude search '?section(games)' >game_selections.txtThe question marks in the search pattern indicate that a search term follows.  Because the parenthesis are special characters to the shell they need to have quotes around them.  In this example "section" indicates the section parameter and "games" is term that is being searched for.  Any package with a section parameter set to "games" will be listed:
...
i   chromium-bsu       - fast paced, arcade-style, scrolling space
i A chromium-bsu-data  - data pack for the Chromium B.S.U. game
p   chromium-data      - transitional dummy package for chromium-bs
p   circuslinux        - The clowns are trying to pop balloons to score points!
p   circuslinux-data   - data files for circuslinux
...
The leading character indicates it's state with "i" for installed, "i A" for automatically installed (either recommended or a dependency), and "p" for purged (i.e. no trace of existence on system which is the default state of all packages).  To filter for installed packages only you add the "?install" parameter:
aptitude search '?section(games) ?installed' >game_selections.txt

...
i   chromium-bsu       - fast paced, arcade-style, scrolling space shooter
i A chromium-bsu-data  - data pack for the Chromium B.S.U. game
i   glchess            - Chess strategy game
i   glines             - Five or More puzzle game
i   gnect              - Four in a Row strategy game
i   gnibbles           - Worm arcade game
i   gnobots2           - Avoid robots game
i   gnome-blackjack    - Blackjack casino card game
...
The status and descriptions will cause syntax errors when using the file as input to aptitude so a format needs to be specified to filter them out.  The "-F" parameter is used to indicate a format change and "%p" equates to the package name:
aptitude search -F '%p' '?section(games) ?installed' >game_selections.txt

...
chromium-bsu
chromium-bsu-data
glchess
glines
gnect
gnibbles
gnobots2
gnome-blackjack
...
To use game_selections.txt as input to aptitude on another system just use command substitution (see the sh or bash man page) to redirect it from the shell:
aptitude install $(< game_selections.txt)On Ubuntu you need to use sudo in front of this command or use "sudo su" to create a root shell and issue it from there.



Restricting SSH logins to specific groups on Ubuntu
2009-12-25T02:00:00.003-05:00
On Ubuntu I have a user account "administrator" which is in the admin group.  It has a complicated password for security.  OpenSSH by default allows all users to attempt to login remotely.  Since user accounts often have weak passwords it's unsafe to allow this.  I could use ssh-keygen to create keys instead but the systems I support are not in the same physical locations so an ad-hoc arrangement is easier as I can't predict what I'll be connecting with.  To set up this restriction all I needed to do was edit /etc/ssh/sshd_config (see the man page for the file) and add "AllowGroups admin".  Then I had sshd reload the config with "/etc/init.d/ssh reload".  After that only members of the admin group could log in and all others receive generic "Permission denied, please try again." messages.  It supports blocking or allowing by user and hosts also.


Basic apt key management
2009-09-29T13:16:00.007-04:00
Ubuntu's keyserver, keyserver.ubuntu.com, has a lot of problems lately.  I was trying to add the Pidgin repository to work around bug #389322 but kept getting timeout errors from the server.  One of my systems did successfully get the key so all I had to do was transfer it to the others.
These are the commands you can use to do the same.  I used a terminal because I couldn't find a way to export the keys graphically with either the Synaptic package manager or Seahorse.
First list the keys:
gpg --list-keys --no-default-keyring --keyring /etc/apt/trusted.gpg
/etc/apt/trusted.gpg
--------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key 
sub 2048g/79164387 2004-09-12
...
pub 1024R/A1F196A8 2009-01-20
uid Launchpad PPA for Pidgin Developers
The "--no-default-keyring" and "--keyring" tells gpg to use only the specified apt trusted keyring.  Next, find the key you want to export from the list and specify either the key ID or user ID with the following command:
gpg --no-default-keyring --keyring /etc/apt/trusted.gpg --armor --export A1F196A8>pidgin.gpg
The "--armor" tells gpg to output an encoded text key instead of binary one.  The Pidgin package signing key ID is A1F196A8 and it is captured to a "pidgin.gpg" file in the current directory.  Then you copy the key file to your other systems and add it to their apt keyrings using either Synaptic (Settings > Repositories > Authentication > Import Key File) or the apt-key command in a terminal:
sudo apt-key add pidgin.gpg
Then you add the repository as shown in the Pidgin download page.  To upgrade Pidgin you can use Synaptic (Reload, Mark All Upgrades, Apply), the text-based package manager aptitude (u,U,g), or apt-get on the command line:
sudo apt-get update
sudo apt-get upgrade
Then you just need to restart Pidgin and try connecting to your Yahoo! Messenger account.



Writing UDEV rules to get a SCSI scanner working on Ubuntu
2009-08-15T00:13:00.026-04:00
I'm building some Ubuntu 9.04 (Jaunty Jackalope) systems for relatives and using them as a way to get rid of a lot of old hardware that has been taking up space in my office.  This includes several old USB, parallel port, and SCSI scanners.  SCSI scanners pretty much ruled in the days before USB as they were much faster than parallel ports.  However, they were a pain to configure and required heavy (and usually short) cables which made them difficult to fit into your work area.  I tested a Microtek ScanMaker E3 (MRS-600E3) and UMAX Vista S8 scanner first.  They worked without problems although the former was picky about termination.  Unfortunately a Hewlett-Packard ScanJet 6100C (Q2950A) didn't work at all.  Checking the kernel messages indicated that it was represented by /dev/sg7 but the permissions were 0660 root:root so sane couldn't access it.  Changing the permissions solved the problem but the /dev directory is a virtual filesystem controlled by udev and the changes are lost after reboot.  I could just put a chmod comand in /etc/rc.local but that is the wrong way to fix it.  A search on launchpad found bug #378989 which describes the problem with this model.  I'm not sure if the fault lies with udev or HAL but creating a udev rule is a simple enough way to fix it for now.  I'll describe how to create such a rule using this as an example but udev rules can do much more than just change device permissions.
First you need to be root.  Either add "sudo" to the beginning of the following commands or switch to a root shell with "sudo su".  Next install lsscsi which makes it easy to identify device node assignments:
apt-get install lsscsi
Then run it to get a list of SCSI devices:
lsscsi -g
[0:0:0:0]    disk    ATA      Maxtor 33073U4   BAC5  /dev/sda  /dev/sg0
[0:0:1:0]    cd/dvd  LITE-ON  COMBO SOHC-4836V SG$4  /dev/sr0  /dev/sg1
[4:0:5:0]    process HP       C2520A           3644  -         /dev/sg7
[5:0:0:0]    disk    USB 2.0  Flash Disk       0.00  /dev/sdb  /dev/sg2
[6:0:0:0]    disk    Generic  USB SD Reader    1.00  /dev/sdc  /dev/sg3
[6:0:0:1]    disk    Generic  USB CF Reader    1.01  /dev/sdd  /dev/sg4
[6:0:0:2]    disk    Generic  USB SM Reader    1.02  /dev/sde  /dev/sg5
[6:0:0:3]    disk    Generic  USB MS Reader    1.03  /dev/sdf  /dev/sg6
Note that the scanner is at /dev/sg7.  With this information you can then use udevadm to find out what is known about the device in the udev database and where in hierarchy of systems it lies:
udevadm info -a -p /sys/class/scsi_generic/sg7

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:1e.0/0000:01:01.0/host4/target4:0:5/4:0:5:0/scsi_generic/sg7':
    KERNEL=="sg7"
    SUBSYSTEM=="scsi_generic"
    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:1e.0/0000:01:01.0/host4/target4:0:5/4:0:5:0':
    KERNELS=="4:0:5:0"
    SUBSYSTEMS=="scsi"
    DRIVERS==""
    ATTRS{device_blocked}=="0"
    ATTRS{type}=="3"
    ATTRS{scsi_level}=="3"
    ATTRS{vendor}=="HP      "
    ATTRS{model}=="C2520A          "
    ATTRS{rev}=="3644"

    ATTRS{state}=="running"
    ATTRS{timeout}=="0"
    ATTRS{iocounterbits}=="32"
    ATTRS{iorequest_cnt}=="0x8"
    ATTRS{iodone_cnt}=="0x8"
    ATTRS{ioerr_cnt}=="0x1"
    ATTRS{modalias}=="scsi:t-0x03"
    ATTRS{evt_media_change}=="0"
    ATTRS{queue_depth}=="2"
    ATTRS{queue_type}=="none"

  looking at parent device '/devices/pci0000:00/0000:00:1e.0/0000:01:01.0/host4/target4:0:5':
    KERNELS=="target4:0:5"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:1e.0/0000:01:01.0/host4':
    KERNELS=="host4"
    SUBSYSTEMS=="scsi"
    DRIVERS==""

  looking at parent device '/devices/pci0000:00/0000:00:1e.0/0000:01:01.0':
    KERNELS=="0000:01:01.0"
    SUBSYSTEMS=="pci"
    DRIVERS=="aic7xxx"
    ATTRS{vendor}=="0x9004"
    ATTRS{device}=="0x7178"
    ATTRS{subsystem_vendor}=="0x0000"
    ATTRS{subsystem_device}=="0x0000"
    ATTRS{class}=="0x010000"
    ATTRS{irq}=="22"
    ATTRS{local_cpus}=="ffffffff,ffffffff"
    ATTRS{local_cpulist}=="0-63"
    ATTRS{modalias}=="pci:v00009004d00007178sv00000000sd00000000bc01sc00i00"
    ATTRS{enable}=="1"
    ATTRS{broken_parity_status}=="0"
    ATTRS{msi_bus}==""

  looking at parent device '/devices/pci0000:00/0000:00:1e.0':
    KERNELS=="0000:00:1e.0"
    SUBSYSTEMS=="pci"
    DRIVERS==""
    ATTRS{vendor}=="0x8086"
    ATTRS{device}=="0x244e"
    ATTRS{subsystem_vendor}=="0x0000"
    ATTRS{subsystem_device}=="0x0000"
    ATTRS{class}=="0x060400"
    ATTRS{irq}=="0"
    ATTRS{local_cpus}=="ffffffff,ffffffff"
    ATTRS{local_cpulist}=="0-63"
    ATTRS{modalias}=="pci:v00008086d0000244Esv00000000sd00000000bc06sc04i00"
    ATTRS{enable}=="1"
    ATTRS{broken_parity_status}=="0"
    ATTRS{msi_bus}=="1"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""
Note that the DRIVERS=="aic7xxx" indentifies the Adaptec AHA-2940 SCSI card.  All of this data can be referenced by a udev rule to identify when and how to manipulate the device.  That is what udev does - run everything through a list of rules, matching or excluding attributes as specified by a rule, then performing an operation when the conditions of a rule is met.  The manual for udev is at /usr/share/doc/udev/writing_udev_rules/index.html and it gives many good examples of what you can do.  In this case the scanner device needs different permissions and group ownership so that users can access it with Xsane.  Most of the rules included with packages are in /lib/udev but local rules can be added to /etc/udev/rules.d and they can override existing rules.  There is a file name standard for the rule files (see the README in the directory) - they always start with a number (which indicates priority) and end with ".rules".  My rule file is "/etc/udev/rules.d/45-scsi-scanner.rules", owned by root and in group root with 0644 (rw-r--r--) permissions.  You have to reboot to make it active.  This is what it contains:
# permissions for HP ScanJet 6100C SCSI scanner
SUBSYSTEM=="scsi_generic",ATTRS{vendor}=="HP",ATTRS{model}=="C2520A", NAME="%k", SYMLINK="scanner%n", MODE="0660", GROUP="scanner"
So what does this all mean?  First the SUBSYSTEM keyword says it only applies to devices in the "scsi_generic" subsystem (as per the first few lines that udevadm reported).  The "==" is a comparison operator.  Next the ATTRS{vendor} keyword specifies that an attribute named "vendor" in the subsystem (or any parent subsystem) has to have a value of "HP" (which the SCSI module reports via the SCSI card).  Then the ATTRS{model} keyword tells udev to look in the same subsystem that matched the vendor for a model attribute that matches "C2520A".  If it finds one, and since there are no other comparisons specified, then the rule matches and the rest is processed.  NAME is the keyword for setting the device node name (sg7 in this case) and the %k is a string substitution operator that udev will expand to the original name assigned by the kernel (again sg7). The "=" is the assignment operator.  So this part of the rule sets the NAME assignment key to the original "sg7" effectively keeping the default device node "/dev/sg7" as is.  The SYMLINK keyword creates symlinks to the default device node.  The %n operator is expanded by udev to the kernel number of the device (the 7 in sg7).  The resulting symlink will be scanner7 in this case and if the default node changes due to a SCSI device being added or removed the symlink will change to match (scanner5 for sg5, etc.)  For the scanner rule it is for convenience only as a device named "scanner" is easier to figure out than "sg", especially when trying to do user support over the phone.  The MODE just sets the permissions in octal and GROUP assigns a specific group membership of "scanner".
When this rule is activated, /dev/sg7 will be root:scanner with rw-rw---- permissions and a /dev/scanner7 symlink will also be created that points to it.  For the user to access the scanner they need to be in the scanner group.  If the scanner group doesn't exist (not in /etc/group) then you can add it with:
addgroup --system scanner
This will dynamically create a system group somewhere in the range of 100-999.  Any users added to the group need to relogin for it to take effect.



Introducing Winesharer - so pre-alpha it doesn't even work
2009-08-02T03:12:00.006-04:00
About a year ago I was setting up a Ubuntu system for a family of non-technical users.  They like to play games and one of their favorites is Diablo II.  It works well on Wine if you set it to use Glide and install a Glide wrapper.  The game's copy protection is properly supported by Wine but it's not necessary as Blizzard removed the CD check in the recent patches.
There are two problems with installing Windows applications like Diablo II for multiple users.  First, because of the isolation of user accounts on *nix systems, you have to login and repeat the install process for each account.  Second, each installation after the first wastes disk space and with Diablo II it's several gigabytes, especially if you have a lot of mods.
It's possible to manually copy the first installation and edit the Wine registry, menu entries, and fix symlinks, but it's tedious.  So I began messing around with some shell scripts to automate the process.  I'm not an expert with shell scripts but I improved with time and some help from my LUG-mates.  After several false starts I got a basic script functioning.  That solved the first problem but not the space issue.
After messing around with some LiveCDs, I got the idea to try to share the wine directory with a union mount.  First I tried FunionFS but it had several bugs that prevented it from working (like not being able to change an existing file).  So I switched to Aufs.  It worked but it can't be run by a user as it requires root permission to mount.  To keep it easy for users I had to use pam_mount and mount it at login.  I added the ability for the script to export a sample mount entry for pam_mount.conf.xml to save time.
I then got the idea to add handling for separate Wine directories for each application (like CrossOver Bottles).  That brought up another issue, the desktop menu entries for Wine's utilities like winefile and winecfg.  I needed to add duplicate entries with different WINEPREFIX settings and associate them with each application.  I came up with a primitive solution for locating the entries and duplicating them in an alternate location as a submenu below the primary application's menu (Wine > Programs > (application) > Wine Utils).
I then got another idea - application merging. One problem with games is that there are a lot of third-party mods and other customizations for them.  There are also a lot of updates.  This requires editing of configuration files, extracting files from archives, and file management.  These can't always be done easily with Linux tools.  One problem is patches for older games are often in zip files.  The contents are intended to overwrite existing files but sometimes the filenames have different case.  If you extract them with a native Linux application you end up with duplicates instead of overwrites.  The other problem with Linux tools is that they always give a "/" or "/home/user" oriented view when the user expecting C:, especially when following an online instructions for installing a patch or mod.  The concept of application merging is simple - select some utilities that have minimal dependences, install to a separate Wine directory, then add it as an Aufs branch to the mount for each main application.  You install each one once and share it but then it can be used within every other Wine application directory (and behaves as if it was installed in each) without wasting much storage space.  Wine's registry files are text so entries for a merging application can be duplicated with just diff and patch.
Then I realized this had another major benefit for game mods - it would be possible to install multiple, even conflicting mods, and mount them as separate Aufs mount points at the same time.  This is especially useful for games that don't have integrated mod management.  Using Diablo II as an example, you would install and update it, share it with wineappshare.sh, then mount it via Aufs with a new read/write directory branch.  Then install a mod like MedianXL (which ends up in the read/write branch).  Then unmount the directory, move and share the read/write directory as a new read-only branch for MedianXL and mount it on top of the Diablo II directory with a new read/write branch.  If you mount the Diablo II directory again using a different read/write branch, you can run regular Diablo II and the MedianXL version at the same time.  Effectively they are in separate "bottles" but share the bulk of the install in read-only branches so there is little additional overhead.
Of course the tricks don't stop there.  You can imagine putting the shared parent branches on a compressed volume to save space, mount it on a server via NFS, and use pam_mount to mount the read/write branches on a USB drive on the client.  Imagine the possibilities for a gaming cafe system.
At this point I knew Winesharer would revolutionize Windows gaming on Linux.  Just as soon as I finish writing it.  Then perfecting it.  Then documenting some really complicated examples.  And make some awesome video demonstrations.  And write the book.  Then push it out of the code cave and bask in the glory.  At least that was the plan about a year ago before life and reality got in the way.  So now I'm down to the old "release early and often" process which completely eliminates the "shock and awe" value.  At least this way it may have an impact before Wine is forgotten due to lack of interest in running legacy Windows applications and everyone switching to GNU/Hurd.  
Winesharer consists of three scripts:

wineappshare.sh - strips out user-specific directory links, tracks down related icons and XDG menu entires, and copies the Wine directory (bottle) to a shared location - /srv/wine by default.  The hardest part was finding the menu entries (*.desktop) as Wine doesn't keep track of them.  I had to do a linear search by grepping for matching WINEPREFIX values then calculating what the matching menu directory (*.menu) path would be.  It also doesn't help that the "Icon=" references can specify icon file extensions with ".png", ".xpm", or not at all.

winemergeprep.sh - makes registry diffs and a list of modified files for "mergeable" utilities.  The file listing excludes any unchanged/unused files like the fake DLLs that Wine adds in the System directory.  It is run twice - after initial creation and configuration of the Wine directory and before the target application is installed, and again after installation and configuration of the application.  The applications I was using are 7-Zip, xplorer², FontPage, IPaddress, SciTE (or EditPad Lite), @icon sushi, and Dependency Walker.  Some of these are only for debugging.  After prep the mergeable application directory is shared with wineappshare.sh like other applications and wineappinstall.sh performs the special handling of their branches and patches when other applications are installed for users.

wineappinstall.sh - where things got ugly.  Setting up the read/write directory and creating the template pam_mount entry for Aufs mounting was easy.  So was copying over the icons.  Recreating symlinks between the user's profile directories (My Doucments > ~/Documents, etc.) was a lot more complicated.  I wanted to do it correctly by following the XDG Base Directory Specification, first by checking for a local (user) configuration, then the system-wide defaults, then look for commonly-used defaults, and finally just defaulting to ~.  I got that part sort of working.  The final problem was trying to merge the shared application menu entries, the "mergeable" applications entires, and any existing entries without damage.  Doing this in an orderly (and deterministic) fashion is difficult and shell scripts aren't great for text processing.  That's where I left off.
The scripts all require the Wine directory name and it's assumed to be in ~.  For example, specifying Diablo II's directory would just be ".wine-Diablo_II".  Note that spaces should not exist in the directory name.  The Winesharer scripts handle them but others, like Dan Kegel's winetricks, had trouble with them.  Second, the scripts search through the registry for the username of the installer in order to change it to the target user's name later.  Because of this it needs to be globally unique (in ALL Windows applications) so the scripts don't change something that is not related to the user.  I had a Wine administration account named "wineadmin" which should be safe as long as there isn't any client/server wine (the drinking type) management applications that use the same keyword or value in the registry.  The sharing directory is in /srv to comply with v2.3 of the Filesystem Hierarchy Standard.  I was using a "_rw" suffix for the read/write branch directories.
What's next?  Nothing.  This was intended to be a one week feasibility study but suffered from a ridiculous amount of feature creep.  Between the earlier draft scripts and command-line tests I know it's possible to do but text processing in shell scripts is tedious and I don't have the time to finish it.  The scripts are ugly, broken, and can't handle all possible problems.  I do like cats so I made it a point to eliminate cat abuse but I left a lot of dysfunctional grep|sed marriages since reducing them is time consuming and the extra processes give my idle Phenom cores something to do.  There's a lot of arrays and case statements.  I didn't follow any column limits either.  Because this was a work-in-progress I also discarded the Unixy notion of minimal feedback - my scripts write entire novels to the terminal.  There are a lot of comments, especially in the unfinished portions of wineappinstall.sh (which is guaranteed to not work).  I didn't even begin implementing the integrated multimedia help styled after 0verkill.  Instead, this pre-alpha work includes a bonus pack of bugs.  I don't think the scripts can fail in such a way as to wipe out your filesystems and install Vista but I'm not guaranteeing they won't either.  This mess was developed on Ubuntu 8.04 (Hardy Heron).
My goal with this project is to inspire others to implement a more robust (not to mention functional) solution incorporating these ideas.  A user space union filesystem like FunionFS would be more convenient than Aufs but I don't know of any alternatives.  I think that having submenus for mergeables in each application menu is ugly.  A front-end utility for dynamically setting WINEPREFIX and launching them would be better.  One problem I thought of but don't know how to handle are applications that require registration keys at installation instead of first-run.  For some, their registration keys can be purged from the registry and they will prompt for a new key when executed again.  Others won't and may lock-out and refuse to run even with a valid key, requiring a full reinstall.



The fun of legacy hardware
2009-07-08T19:19:00.026-04:00
I have an old embedded system that uses an SBC-MAX (pdf) board from Computer Dynamics.  The unit was part of a vehicle monitoring system that used Windows 98 (one of many fundamental flaws in the design).  It has a K6/2 333MHz CPU and 128MB of EDO DRAM.  It features a whole bunch of integrated devices and a fairly broken BIOS.  As an embedded system it has a PC/104 bus for which I have a few modules including a GPS.  I'm would like to get that working but getting Ubuntu to even install on it has been a pain.
The source of the problem is an ITE IT8330G PCI-ISA bridge with IDE controller that is only supported by the ide-generic driver.  This is rather obsolete and isn't loaded in most kernel images including bootable CDs.  The latest Ubuntu CD that would boot is the 7.10 (Gutsy Gibbon) alternate CD.
Gutsy's install worked up to where it loaded packages where it would hang after a while.  I suspected it was running out of memory so I tried again.  After formatting the partitions I switched to a different terminal and activated swap before continuing.  This solved the problem:

free
fdisk -l /dev/hda
swapon /dev/hda5
free
When the install completed, it hung at restarting so I power-cycled it.  It does this at power-off (halt) as well which may be a board limitation as the system originally used a serial port to shut off power via an "intelligent" power supply made by Dynamic Engineering.  The system booted and Grub loaded initrd (containing the kernel) and then the init scripts started but then it stalled for a while - ending up at an initramfs prompt.  Rebooting and editing the boot line in Grub to remove the "quiet" and "splash" entries resulted in more detailed messages which showed it couldn't find the drive.  Basically the only driver that supports the ID8330G is ide-generic and it's not in linux-generic which Gutsy and later releases use by default.
The solution to getting Linux to boot is to add the driver but it's in linux-386.  To install it, I rebooted with the CD and entered "rescue" mode.  After a series of prompts it gives you the option to open a root terminal on a chrooted partition.  I selected the root partition and got a bterm session.  On Gutsy, it's not a friendly environment as you don't get tab completion or history so you get a lot of finger exercise.  First thing was to activate swap then use apt-get to install linux-386.
This is were the next problem was encountered.  Gusty is obsolete so the repos have moved to the old releases server.  Trying to fix the sources.list file with vi was impossible due to refresh and scrolling bugs with either it or bterm.  I tried Midnight Commander (mc) and had to set the TERM environment variable (export TERM=linux or whatever) but it was also rather ugly.  I eventually figured out a sed script to fix them faster:

cp /etc/apt/sources.list /etc/apt/sources.list_orig
sed 's/us\.archive\|security/old-releases/' /etc/apt/sources.list_orig >/etc/apt/sources.list_oldr

This just looks for "us.archive" or "security" and replaces them with "old-releases".  The next problem was that Gutsy's installer had disabled the repos since it couldn't find them during installation.  Another sed script fixed this:

sed 's/^#[# ]*\(deb .*$\|deb-src .*$\)/\1/' /etc/apt/sources.list_oldr > /etc/apt/sources.list

This looks for lines starting with the comment character # followed by "deb" and attempts to skip other comment lines.  Then I ran "apt-get update" and "apt-get install linux-386" and was good to go - almost.
After rebooting it ended up at the initramfs prompt again.  I entered "modprobe ide-generic" and it found the drive.  I entered a Ctrl-D and the boot completed.  I filed bug 128833 about this a while back but I know now that the driver can have problems with more-specific IDE drivers so normally it isn't loaded.  To fix it I wrote a basic init script named "idegeneric" and put it in "/usr/share/initramfs-tools/scripts/local-top":

#!/bin/sh

PREREQ=""
prereqs()
{
	echo "$PREREQ"
}
case $1 in
# get pre-requisites
prereqs)
	prereqs
	exit 0

	;
esac

modprobe ide-generic

I just basically copied one of the other scripts and modified it.  I then created a new initrd image with the command "update-initramfs".  It loaded on the next reboot without problems.
Next I updated Gutsy with "apt-get -y upgrade" which updated a whole lot of stuff and installed a new kernel (which included the idegeneric script automatically).  On a system this old it's like watching grass grow so I worked on something else for a few hours but later found it had locked up (the Magic SysRq keys didn't work).  After a power cycle I ran dpkg --configure -a" and it completed without problems but it found some bad inodes on the next reboot and had to fix them (should have done this first before finishing).  Then it was time to upgrade to something a little more modern, Ubuntu 8.04 (Hardy Heron).  The command to do this is "do-release-upgrade".  The first thing it does after finding a new release is change the release targets in the sources.list file to "hardy".  Next it loads in the new package lists.  Of course this failed as it was looking for them on the old-releases server so I had to change the entries back to "us.archive" first.  Then it was happy and began the upgrade.  After a reboot Hardy loaded without problems.
An upgrade to Intrepid Ibex (8.10) was next but "do-release-upgrade" couldn't find a newer version.  This is because Hardy is a "Long Term Support" version and the next LTS release will be 10.04 which isn't out yet.  To fix it, you have to edit "/etc/update-manager/release-upgrades" and change "Prompt=lts" to "Prompt=normal".  This upgrade continued without problems.
Jaunty Jackalope (9.04) is the latest but after the upgrade it failed to load the driver.  A message was shown:

ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports

Looks like "probe_mask=0x3f" was needed at the end of the modprobe line in the script to make it happy.  Luckily the older Intrepid kernel was still configured in Grub so I was able to boot it instead.  I added the parameter to idegeneric, then ran "update-initramfs" with the "-k" parameter to specify the Jaunty kernel (referencing it in the form that "uname -r" returns).  After a reboot it loaded without problems but then started segfaulting all over the place.  Apparently ide-generic is broke in Jaunty or doesn't like the ID8330G so it's back to Gutsy and restart the process.  I stopped at Hardy as I have better things to do.  I did try Puppy Linux but both the regular and "retro" versions failed to find the drive even with the "all-generic-ide" boot option.



My Paperwork Reduction Act
2009-06-03T22:23:00.006-04:00
I have a habit of keeping receipts of every sales transaction I make.  This is good for taxes, returns, rebates, billing, resale, and just trivia.  After a decade or so it really piles up.
Several years ago I built my first PC, a Pentium 133MHz system with a Micronics M54Hi motherboard and a screaming Quantum Fireball 7200RPM 4GB SCSI drive (which later failed so I guess the screaming was a bad thing).  This was one awesome Doom playing system.  The advantage of SCSI over IDE was the number of devices a port could support (7 instead of 2), speed, and the ability to brag that your system cost 10x what everyone else bought.  Of course a 50 pound, 30in high tower case with 3x the space needed was essential.
I also bought a HP ScanJet 3C (C2520A) 600dpi 8.5x14in SCSI scanner.  This alone was about $900.  While the scanner did get some use it mostly just sat on the shelf collecting dust and depreciating.  It was rather large and I didn't have enough space on my desk to keep it handy.  Several PCs later I bought a Visioneer OneTouch 8100 USB scanner for $50 - which was a waste of money as Staples had it on sale a month later for $25.  This worked well and was much smaller but I still didn't get around to catching up with the paperwork.  A unique feature of the 8100 was that it's power jack matched the plug on my Toshiba laptop power supply.  However the power supply output was not compatible and after getting the plugs mixed up one day I had to buy another scanner.  This time I got an Agfa SnapScan 1212U which has worked rather well.
Last week, while tripping over another box of paperwork I decided to finally start scanning things in.  I'm using Xsane on Ubuntu 8.04 (Hardy Heron) and it's working rather well.  I fit as many receipts as I can on the glass then preview them, draw a selection box around each in the Preview window, then hit the Scan button.  Some of the receipts are too long for the Agfa so I installed an Adaptec SCSI card and hooked up the ScanJet.  Xsane (0.995) doesn't let you switch between two scanners except at startup but you can run two instances of it simultaneously.  It keeps track of preferences for each scanner separately too.  Some of the documents I'm scanning have several numbered pages and Xsane has a numeric filename auto-increment function.  It's not very flexible but it does the job.  My only complaint is that some hotkeys, like Ctrl-V, are used for changing settings instead of the default copy/paste functions that most apps use so I have to use alternates like Shift-Insert.  This makes copying and pasting filenames into the save dialog boxes annoying.
I find that scanning most things in at 300dpi grayscale and saving them as JPEGs works best.  For items where color is important I use 600dpi.  The ScanJet has one design flaw in that it's pad is white while the SnapScan's is black.  The white pad causes printing on the back side of receipts to show up when scanning the front so it takes some fiddling around with the contrast and brightness settings to suppress it.
I'm not just scanning in receipts either. I'm also scanning contracts, notepads, holiday greeting cards, photographs, and user manuals I can't get PDFs for.  It takes a long time but DVDs take up a lot less space than file cabinets.



Be wary of CPU upgrades on old motherboards
2009-03-24T21:05:00.011-04:00
Twice now I've encountered burned power connectors on older ATX motherboards.  The first was a pair of Tyan S2460 boards with dual Athlon MP CPUs.  The owner was having problems with them and thought one of the CPUs had failed.  He gave them to me and while testing I noticed the problem.  Both of these boards were designed to the original ATX specifications and only had a 20-pin power connector.  The CPU regulators used +5 volts as their input supply to generate the CPU voltages.  The increasing power requirements of CPUs in this era (especially dual CPUs) resulted in increased current (amperes) through the connectors.  Since increasing current results in increasing heat dissipation (due to wire and connector resistance) and the increased heat also increases the resistance the same, a thermal runaway condition can occur.  This greatly reduces the life of them - the plastic of the connectors bakes and melts and eventually the solder on the motherboard connector pins melt.  All of this is bad for stability.
I deemed the boards were useless and, not finding any economical Socket A replacements, decided to upgrade two other systems and give the rest of the parts to a PC recycling center.  One of the CPUs, an XP 1800, replaced a 1.3GHz Thunderbird in a Gigabyte GA-7ZXE.  This solved a problem with it's Nvidia 6600GT AGP video card and the newer drivers requiring SSE support.  Everything was good for a few months but eventually instability set in.  I found the +5V pins (red wires) on the ATX connectors had burned and fused together.  The 6600GT had it's own power connector and I didn't think one CPU would cause an overload on the ATX connector.  Burned twice you might say.
Newer versions of the ATX specification, ATX12V, added the second four-pin CPU power connector that uses +12V instead which greatly reduces the problem.  In electrical physics, for the same amount of power (P, in watts), doubling the voltage (V) halves the current (I).  The voltage (the amount of energy each electron is carrying) doesn't affect wire or connector heating, only the current (the quantity of electrons moving through the wire) and the conductor's resistance (R, in ohms) does.  This heating effect is described by the formula P=I²R and is called "I squared R" losses.  Something to keep in mind when upgrading or recycling old systems.
Note that this problem doesn't just affect the connectors.  I almost fried a 400W ATX power supply while testing the Tyans with two CPUs because of the current requirements on the +5V rail was more than it could handle.



A Linux user's review of Windows 7 Beta
2009-02-11T22:46:00.172-05:00
After years of being a Windows user (since 2.0) and an administrator I've learned to ignore the marketing hype surrounding new Windows versions.  But I tried out the Windows 7 beta just so I can settle arguments about what it can or can't do.  It's only a minor upgrade from Vista with some stuff added and removed but the fanatics have been evangelizing it like it's a start of a new era in computing technology.  Linux netbooks^® really get them hyper.
Table of Contents

1. Preparation

2. Installation

3. Storage

4. Encryption

5. Interface

6. Applications

7. Legacy

8. Control and Security

9. Shortcuts to Panic

10. Divide and Conquer

11. Sharing Almost Redefined

12. Epilogue



^ 1. Preparation


I can't give a fair comparison between Vista and Windows 7 so I'll be referencing XP a lot.  I haven't used Vista much as I have no need for it internally and only used it at a customer site a while ago.  The customer I did ECAD work for insisted on it which reduced the workstation performance by half compared to the previously installed XP, even with  Aero disabled.  Over RDC it was even worse with constant stalling especially compared to a Windows Server 2003 system on the same network (disabling auto-tuning didn't fix it either).  I charge by the hour so it really wasn't a problem.  I haven't worked for them since the Windows 7 beta was released so I can't install it on the same system.  I'm not going to install Vista in VMware to compare either.  I feel that a virtual machine is not a good way to benchmark a desktop OS because there are too many variables.

I downloaded the 64-bit DVD ISO (3.2GB) and installed it on VMware Player 2.5 on 64-bit Ubuntu 8.04.2 (Hardy Heron).  I'm using a Phenom 9550 with 8GB PC2-6400 ECC memory and a pair of Maxtor IDE drives using software RAID (md), LVM, and LUKS/dm-crypt.  Drive encryption doesn't affect performance much with modern processors as the drives are so much slower.  An XP VM runs just fine on the same system.  I gave the Windows 7 VM 2GB of memory and a pair of 16GB virtual volumes.  The beta is the Ultimate edition which will be the most expensive of the lot.



^ 2. Installation


Windows 7 Setup is graphical like those on most Linux distributions (which they've had for many years) but the functionality isn't much better than the XP installer.  The only really useful addition is the ability to use USB storage devices to load drivers instead of requiring a floppy drive.  Like in XP it is single-task based so that every partition edit is immediately applied while most Linux installers queue up a series of operations and then perform them in a batch.  The installation and updates took a long time with several reboots (and of course entering the 25 digit key).  I don't remember the details about the one Vista installation and update process was like but both are faster to install than XP with it's service pack and several updates that often have to be performed sequentially.  The progress messages from the setup screen did seem to indicate some updates were installed but after logging in Windows Update installed some more.  The earlier messages may have been referring to updates on the ISO that weren't slipstreamed.

Ubuntu's graphical installation process is a lot faster, especially considering the number of applications included.  Its updating can take longer but the package manager updates everything, not just the OS.  I use an internal mirror with netbooting via PXELinux that's partially automated using Kickstart so my installations are very fast and already updated.  You can achieve some of the same benefits on Windows with the AIK, WinPE, WDS, and slipstreaming but you still have to deal with licensing, product activation, and updating.  Windows Update only covers Microsoft products so other applications need their own update functions else you have to do them manually.  Like Ubuntu and Debian most Linux distributions come with text or graphical installers or both.  Text installers are not as friendly but work on systems with limited memory.  Graphical installers are easier but require more memory.  Many of them are integrated into Live CDs that can be used for web browsing or playing music while the OS is being installed in the background.  There are a few Windows Live CDs, mostly based on BartPE.  I haven't tried any of them but once I made a Windows 98 Live CD with a DriveSpace volume that had Quake installed as a feasibility study for RAM disk usage on an embedded system.  Using Windows 98 on an embedded system was STUPID but I wasn't the engineer in charge of the project.

The full install was about 9GB but I suspect there is some debug code and related utils taking up space.  There is a separate 200MB system partition with about 32MB for the boot loader and its language support files.  An XP install with IE 7 added is about 3.5GB at most.  A Ubuntu install is about 3.5GB (with a lot more applications) and comes on a CD.

After installation the first issue I encountered was that Windows 7 doesn't have a driver for the default AMD PCNet virtual network device.  I had to set ethernet0.virtualDev = "e1000" in the vmx file which it recognized as an Intel PRO/1000 MT device.  I started with a VMware configuration from an XP VM.  If you are starting from scratch you may find EasyVMX helpful.  I then extracted and installed the VMware Tools.



^ 3. Storage


Some of the features I wanted to try included Logical Volume Management (or "Dynamic Disks" as Microsoft calls it), software RAID, and drive encryption.  I didn't use these technologies until I started using Linux.  I knew that Windows had the capability but systems I built had hardware RAID and didn't need LVM or encryption.  With Windows 7 Setup there is no way to configure these during installation.  You have to create a basic disk and then convert it afterwards.  What's annoying is that if you later erase the partition contents, leaving only the Dynamic Disks and RAID layout, the installer will install to them and they will be fully operational when Windows first starts.  While the installer does indicate which partitions are dynamic or are using encryption it doesn't do so for RAID or anything on the system partition.  Linux installers usually show the individual partitions and the child volumes for everything so it's easy to follow.  I'm surprised by how primitive Windows Setup is considering how long it's been since they released XP and features like Dynamic Disks have been available since Windows 2000.  It may be the result of the decision to only offer these features in specific editions of the OS (which causes problems in Vista with mixed installations).  I haven't found anything definite about Dynamic Disk availability and the various editions of Windows 7 yet.  Linux distributions don't have these restrictions and most graphical installers can set up RAID and LVM.  For drive encryption the only installer that can handle it (that I'm aware of) is the alternate text-mode installer on Ubuntu (and Debian) but I suspect more distributions will add it.  Drive encryption is really important for laptops.

I then set up Dynamic Disks first as software RAID requires it.  Applying the change was instantaneous.  While setting up the RAID mirror with the Disk Management tool I encountered an error that stated the boot configuration of the system could not be updated and I should use bcdedit.exe to fix it manually.  I didn't bother and I found that the system wouldn't from the mirror if the primary was removed.  On Linux this is also a little tricky with the GRUB boot loader.  The problem is that in a failure scenario it's hard to deterministically identify at boot which is the good drive versus the bad if the latter is partially functioning but not syncing.  Obviously this requires a BIOS that can boot the next good drive if one has failed completely (or partially after a timeout).



^ 4. Encryption


BitLocker  is Microsoft's drive encryption system.  Another implementation is BitLocker to Go which targets USB storage devices.  I tried to "turn on" BitLocker for the C: drive but found it can't use a dynamic volume.  So I went back to Disk Management only to find that you can't revert a dynamic volume to basic if the OS is on it.  So I had to reinstall.  But then I found that the installer won't allow you to delete a a dynamic partition.  I had boot Knoppix and use cfdisk to delete the type 42 SFS partitions.  Then I was finally able to reinstall and activate BitLocker, then change the disks to dynamic, and then set up RAID mirroring.

On Linux these problems don't exist.  Any combination of software RAID, encryption, and LVM volumes can be created and stacked in any order on almost any storage device.  On my Ubuntu system I set up RAID first, then LVM, some logical volumes (usually one per user) within the LVM group, then LUKS/dm-crypt volumes on those, then format them to ext3.  I split the encrypted volumes up this way because even a single bit error can cause major damage on encrypted data.  With separate volumes I limit the damage if one develops an error.  This means the partition headers for RAID and LVM are unencrypted but I can't imagine they contain any significant data that could compromise the system.  Unintentionally I tested the reliability of this configuration over several months amid random system crashes.  I eventually narrowed down the problem to RAM - I had been burned by a bad batch of Crucial BallistiX PC2-8500 2.2V modules like a lot of people.  Only the final crash lost any data and I recovered most of it (the critical data was backed up elsewhere).

From what I've read, to set up BitLocker on Vista required creating a 1.5GB partition for the loader/authentication system which obviously can't be encrypted else it can't boot.  In Windows 7 the partitioning wasn't necessary and it looks like only the 200MB system partition is used (unless part of C: is not encrypted which I can't determine).  My system doesn't have a TPM so I had to use a USB key as it won't allow just a PIN.  At first it insisted that a TPM was required but I found that this was due to the default setting in the Group Policy.  After changing the setting it still wasn't working.  I noticed that my USB flash drive wasn't enumerating properly and in Device Manager the USB Mass Storage Device was reporting "This device cannot start. (Code 10)".  Turns out that I had over-optimized the vmx file and needed to add some settings back.  BitLocker was now satisfied and I selected the new option "Require a Startup key at every startup" and selected the USB drive for the key.  Next I had to choose what to do with the recovery key - save it to removable drive, a file to an unencrypted location, or print it.  Both key files are small and have long names, probably a serial number.  The recovery key is a text file that includes a description of what it is for.  The startup key is a binary with a BEK extension with hidden and system attributes set.  The final screen had a "Run BitLocker system check" option selected by default.  It restarts the system and attempts to read the startup key from the USB drive.  After I clicked the Continue button it just kind of did nothing until I found that a restart prompt dialog was hidden behind the Control Panel window (minor bug).  It restarted and booted back into Windows and reported that the test failed.  After several attempts I tried it without the test and it proceeded to encrypt drive.  It did take a while but the system was usable while this was occurring.  You could say its ability to encrypt the volume the OS was operating from is an advantage but I'm not sure it makes up for Windows Setup not being able to do it.  After it completed I rebooted with the USB drive but the key check failed.  Apparently it couldn't see the drive.  I searched with Google a bit and found many other reports of the same problem with Vista.  Many users thought the BitLocker utility wasn't saving the key to the USB drive but I think they were confused because it was hidden.  It may be an issue with the BIOS not enumerating the device or the boot application not communicating with my particular USB drive.  I ended up having to enter the recovery key and Windows booted.  The recovery key is 48 digits long and is entered as 8 groups of 6 digits which is easy to enter with a numeric keypad.  Each group apparently includes a checksum as it validates them as you type.  After the last group is entered correctly it begins loading Windows immediately instead of waiting for you to press Enter.

On Linux, LUKS/dm-crypt uses PINs exclusively which decrypt a second key stored in the partition header which is then used to decrypt the volume.  LUKS has eight key slots and any key can be used to add or delete the other keys.  This can be combined with other authentication mechanisms via boot scripts for TPM, USB keys (via pam_usb), and pretty much anything else.  The problem with it is that while scripts exist for some of the authentication options there isn't much of a standard implementation and many distros don't include all the functionality.  Both have biometric support - Windows has a control panel applet and fprint tools are in the Ubuntu repositories but I don't have a fingerprint reader and can't test either.  Like with Windows a part of the OS needs to be unencrypted so it can authenticate the PIN and unlock the rest of the volume.  On Linux this is the /boot subdirectory which contains the Linux kernel.  This means the entire kernel loads and can provide access to any hardware device for which it contains a built-in module (driver).  For both Windows and Linux the boot-time authentication system is the primary attack vector for an encrypted filesystem. If an attacker installs spyware it could copy the key elsewhere for later retrieval.  The Linux kernel and start-up scripts can can be configured boot off USB key instead and WinPE could probably act in the same capacity for Windows.  But this security hole also exists with the BIOS and even with TPM it's still possible to hack into or around (although not easily).

You can "suspend" BitLocker which simply means it stores the key plaintext on the drive someplace making it pretty insecure.  Obviously you could do the same with LUKS/dm-crypt but who would want to?  According to the help file it's for some tasks like BIOS updates which may be due to the way TPM operates.  You can also "turn off" BitLocker which decrypts the drive, a surprising option.  Like the original encryption pass the system is usable while it is decrypting.  Kind of neat but again I can't imagine where someone would need it (even for forensics).  I went ahead and tried it anyway.  The decryption pass is very slow but it worked and I could boot without entering the keys again.  Just for fun I turned it back on again to see how long it would take to re-encrypt.  That's when I encountered a fundamental limitation of the whole architecture.  Remember that I had to set up a Dynamic Disk and RAID after encrypting because BitLocker required a basic disk structure?  Well now that it was dynamic it couldn't encrypt it.  So I had to reinstall all over again using Knoppix to delete the partitions, etc.

An alternative encryption option is the file-level Encrypting File System.  It can unlock files automatically upon login.  Starting with the 8.10 (Intrepid Ibex) release, Ubuntu has added a similar feature, Encrypted Private Directory that also unlocks with logins.  In additon, LUKS/dm-crypt volumes can be controlled by login when combined with pam_mount.  With both Windows and Ubuntu the encryption keys are themselves encrypted when used by the login process and are associated with the user's password (a common weak point).  The pam_mount implementation in Ubuntu (and Debian) has a few problems and one is that the association between the user's password and the decryption key is not maintained if the user changes their password.  Of course with both Windows and Linux these encryption options are still vulnerable to spyware.

In summary, Windows and Linux have comparable features with LVM, software RAID, and encryption.  The Windows solution has a nice GUI and adds a couple of questionable features but it's implementation is very inflexible and limited by a crippled installer, architecture, and licensing.  It's not that these features are "enterprise only" as they can be very useful for home systems and laptops.  But they're unlikely to see usage outside of large corporate environments because of the limitations.  Ubuntu and other distributions are very flexible and without licensing problems but the encryption installation and authentication functionality needs to be streamlined better.  Both encryption solutions for Windows and Linux distros have multiple options for management with directory services but I don't have one set up and it's outside the scope of this review.



^ 5. Interface


The desktop looked plain enough.  It wasn't using Aero since VMware Player's experimental DirectX acceleration feature isn't enough to get it working even with registry hacks.  I don't find the desktop 3D effects trends useful and don't have Compiz enabled on my system since it currently doesn't work well with multi-head desktops and full-screen OpenGL applications.  Those problems are expected to be solved with future releases of the DRI.

The new "superbar" taskbar is interesting.  Basically you can pin a menu item to it which is similar in functionality to the Quick Launch toolbar you could enable in the taskbar on XP.  Task buttons for running applications also end up there which at first seems confusing as you can't just look at it and tell what's running or not.  The difference between a pinned menu item button and a running task button is that the latter will pop-up a list of active windows when it has focus, similar to the "Group similar taskbar buttons" function in XP when there are more active task buttons than will fit on the screen.  If a pinned menu item also has open windows then they are just listed above it so it makes sense it a way.  Right-clicking on an item allows you to unpin it and lists recent files used with the application.  The button grouping function on XP slowed me down when I had many CAD documents open so I disabled it.  I would have to use the superbar a lot to know if I like it better.



^ 6. Applications


There still isn't much in the way of applications included but some of the existing ones have improved.  Some that were in Vista have been removed and others added in.  This is not a criticism of the lack of bundled applications but when you compare it to Ubuntu and the relative installation sizes you wonder what's using up all the space.  Internet Explorer 8 continues to try and catch Firefox but is also available for XP.  Like web browsers in most Linux distributions it doesn't include the Java, Flash, or Silverlight/Moonlight plug-ins.  Notepad is still useless as it doesn't handle LF-only newlines and still has no syntax highlighting even for batch files.  I normally install EditPad Lite.  The calculator has improved and is equivalent to the default gcalctool on Ubuntu.  WordPad is substantially better and can open and save OOXML and ODF files but I didn't test for compatibility with Word 2007 and OpenOffice.org Writer 3.  It closest Linux equivalent is probably AbiWord but I normally use Writer.  Ubuntu includes OpenOffice.org but removing it and adding AbiWord saves about 300MB of space.  Even combined they are much smaller than an Office 2007 installation.  Paint now has multi-level undo.  It also has a lot more scalable shapes but having them in a raster graphics editor seems like a waste of code.  Paint is basically useless for photo editing and is only good for people who think vector editors are hard.  The closest F/OSS competitor is Tux Paint which doesn't have the editing tools but is more fun.  Both WordPad and Paint now use the Office 2007-style ribbon.  I'm indifferent about the ribbon.  It makes some functions easier to find but in applications with a lot of functions it makes the rest harder.  There's also the standard selection of basic games with a few more network-enabled versions.

Windows Media Player 12 currently doesn't support XP and is still bloated fatware compared to foobar2000, Media Player Classic, and VLC (on Ubuntu I use Rhythmbox).  Considering the number and types of plug-ins available it probably qualifies as its own OS.  Initially it can only rip to WMA, MP3 and WAV, but plug-ins for Ogg and other formats are available.  It's not very good at solving codec problems.  I tried a few videos including Elephant's Dream (DivX MPEG-4) and it reported it couldn't play the files and the problem may be the file type or codec.  A "Web Help" button took me to a web page about error #C00D1199 (not very helpful).  On Ubuntu when Totem doesn't have the correct codec it offers to install what it needs.  With Windows, advanced users often just squash the problem by installing every possible codec.  The other audio utility, Sound Recorder, can only save to WMA when in XP it could only save to WAV.  In Ubuntu, gnome-sound-recorder can save to FLAC, Speex, Ogg Vorbis, and WAV.

I don't watch much TV and I've never used a PVR although I've been intending to set one up.  Windows 7 Ultimate includes Windows Media Center.  There are several similar PVRs for Linux, most of which are free and support Windows also.

The data backup and restore functions are more integrated then the buggy backup utility from Veritas in XP.  In the properties panel of files and directories there is a "Previous Versions" panel that can be used to restore them from backups and restore points.  When you connect a removable drive one of the autoplay options is to "Use this drive for backup".



^ 7. Legacy


There's some old Windows NT holdovers in XP.  The Explorer Install New Font/Add Fonts dialog and ODBC Data Source Administrator are the two most obvious.  I wanted to see if they were still around in Windows 7.  The Install New Font option in the Fonts folder has disappeared as font installation is handled by a TTF file context menu.  That left the ODBC utility.  I went through the same routine as I would have on XP with an ODBC registration for an Access 2007 database.  I didn't want to bother installing an Office 2007 trial so I installed the Access Database Engine (a.k.a. ACE) and then went into the Administrative Tools and ran the ODBC Data Source Administrator.  I clicked the User DSN Add button but only an SQL Server driver was listed.  I double-checked System DSN and, not seeing it there either, went through the Program Files directory to verify it actually installed.  After some searching it turned out to be a 64-bit feature.  The link in the Administrative Tools was to the 32-bit version of the utility and I had to use the 64-bit version located at C:\Windows\SysWOW64\odbcad32.exe instead.  So I finally was able to select the Access mdb/accdb driver, enter a name, and then select a database.  Then I was greeted by an old familiar dialog.  It may seem petty but I get annoyed if I have to map a drive letter to select a database on a server share.

<rant> For the record I think the Jet/ACE database is unreliable garbage and would take SQL Server any day.  Unfortunately I have to defer to software engineers that insist it's easier with .NET to work with Jet/ACE than SQL Server because "SQL is hard".  The Jet engine is the abandoned offspring of the SQL Server team - deprecated in favor of SQL Server Express.  The Access team couldn't seem to live without a particular query function so they forked Jet and their version is known as ACE.  If you install Access 2007 on Windows 7 you end up with both Jet and ACE.  In spite of what Microsoft says about upsizing (data migration) from Jet/ACE to SQL Server they are not quite compatible due to different field types and date ranges.  I have tried using Access 2003 with a SQL Server back-end but it couldn't handle things like tables with auto-incrementing key fields.  I haven't tried the same with Access 2007 and don't plan to since I've come to the conclusion that using a fat client instead of a web interface for data entry and reports is a waste of drive space. </rant>

One annoyance was that any online help that used the HTML Help Control for chm files (like the ODBC Data Source Administrator) would open a window that was locked to always be on top.  This makes working with any application full-screen impossible with the help open.  I had to open the chm files manually to get around it.  Most of the other applications use the newer help system which has the opposite problem - when you want to lock it on top you can't.  Problems like this don't occur on Linux because the window controls are provided by the window manager while on Windows each application has to implement it's own controls.  If a Windows application developer doesn't feel like adding an "always on top" control then you go without or use a third-party utility that can override the window properties; or maybe not legally as the EULA states "You may not...work around any technical limitations in the software".  Another irritation is the number of fixed-size windows with list boxes.  I'm not overly fond of scrollbars especially when I have lots of desktop space.  They did add some new hotkeys for window management which is an improvement but some of them don't work without Aero.



^ 8. Control and Security


Like Ubuntu with it's sudo implementation, the first user account created in Windows 7 has superuser privileges.  This user can elevate their access on demand but doesn't log in directly with a superuser account (same with Ubuntu and the Linux root account).  Explorer has a context menu on executable files allows them to be run with administrative privileges.  The same can be done with most Linux file managers by configuring an "open with" sudo entry.  On both most processes run using various system accounts.

This brings up everyone's favorite Vista feature - UAC.  It has undergone some behavior modifications.  If you are logged in with an account that has superuser privileges it only pops up if you try to install something or copy anything to certain directories like C:\Program Files or C:\Windows.  Like in XP you get a security warning (not UAC) if you try to run an executable from any network location but not if you just copy it to the desktop or public folders (C:\Users\Public) and run it.  Non-admin users get more UAC prompts.  Considering typical user behavior is to use the first account with the least annoyances this looks like a set-up for a "blame the users" malware excuse that will keep Microsoft's lawyers happy.  Every time I see "UAC" I think of the fictional company Union Aerospace Corporation  that was responsible for the demonic invasion in Doom.

One odd feature is PC Safeguard which is apparently an updated version of an add-on named SteadyState.  SteadyState is available for XP but it's not something I encountered before.  It adds a data reversion function for a documents and settings in a shared user account.  The changed data is stored as a file on the drive that is cleared at logout.  On Linux you could achieve the same thing by mounting a temporary home directory on Aufs and deleting the rw branch at logout.  Some of the reviews I've read promoted it for safer web browsing (remember that integrating IE with the OS was a feature).  Compared to Live CDs this seems like a hack for administrators who don't know how to lock down a desktop or set up thin clients.  Promoting it as a recovery mechanism reminds me of a CAD system I used long ago that had wonderful crash recovery right up to the last command issued.  It was a feature that had evolved because all it did was crash.

One interesting feature is the integrated parental controls.  On Linux there are graphical solutions for web access control (DansGuardian with the Ubuntu Christian Edition GUI and Mandriva's drakguard) but Windows 7 includes controlling access to applications by user and optionally by the ambiguous ESRB ratings.  I've encountered the need for per-user application access control with a parent that wanted to play games that he felt were too violent for his kids.  I solved the problem by changing ownership and permissions on the XDG desktop entry but a GUI would make it easier for a parent to do it.  I like using parental controls, not because they work, but because it encourages kids learn more about computers while trying to get around them.



^ 9. Shortcuts to Panic


Lately there's been a lot of concern about Linux malware and XDG desktop entries.  On Windows, programs that are runnable (executable) are determined by their filename extensions (exe, com, bat, cmd) which are hidden by default in Explorer.  On Linux and other Unix-like operating systems the filename doesn't matter.  Their files have attributes (read, write, and execute) similar to Windows (read-only, hidden, and system).  In general a file can't be run as a program directly unless the execute attribute is set.  It is possible to get a script (similar to a Windows batch or command file) to run without being marked executable if you call the program that interprets it (sh, bash, etc.) and tell it to run the script.  When it comes to sending a program through e-mail the executability of a Windows program remains as the filename doesn't change (although some extensions are blocked by some e-mail servers).  File attributes are not included with the file so the e-mail client that receives them saves it according to the settings on that system.  On Unix-like systems that usually means without the executable attribute set.  An XDG desktop entry is a text file that is essentially the equivalent of a Windows shortcut (lnk) file.  Since the XDG file is not a program it usually doesn't have the execute attribute set which is technically correct.  But the problem with both Windows lnk files and XDG files is that they can be created to run any program and specify any parameters so effectively they act as programs by proxy.  They can be used as a basic malware carrier by being configured to create a simple program when launched.  They could also be used to exploit an existing security hole in an application by having it connect to an outside source that has been maliciously created to exploit that hole.  There has been some debate about requiring the file managers and desktop environments to not launch XDG desktop files without the execute attribute set.  Currently Gnome and KDE don't require the attribute but the file manager Thunar in Xfce does.  Unlike Windows, which always has Explorer, on Linux there are many desktop environments used so a security problem with one is not necessarily a problem for the others.  The problems of Windows lnk files has existed since the early days of Windows.  Here is an example shortcut Target command line that can create an entry in the current user's Startup folder which will show a directory listing at every login:


C:\Windows\System32\cmd.exe /C echo cmd /K dir "%HOMEPATH%\Documents">"%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup\dir.bat"


This will work with Windows 95 and newer.  Give it a nice helpful icon (C:\Windows\hh.exe) and hook it up to the ftp client (available since Windows 95) and you got yourself a script kiddie-quality spyware.  Safeguards like caution dialogs can be implemented on both Windows and Linux but don't underestimate the ignorance of users under the influence of social engineering, especially when they've grown used to being flooded with UAC prompts in Vista.



^ 10. Divide and Conquer


I wanted to try recreating the common Linux practice of separating user files from the rest of the OS with /home as a mount point for a separate partition.  Whenever I rebuild a standalone Windows system I usually spend half my time backing up and restoring user documents so they don't get overwritten when I reinstall or use a recovery disk.  I almost never use the Windows "repair install" option as usually the system has malware and replacing Windows components does nothing for any other application executables or their cached updates.  With Linux you don't have to worry about wiping out /home during a clean reinstall and at most it will require fixing ownership of the existing files.  In addition, the root account uses /root for it's home so it's easy to isolate any problem with other user home directories.  With Windows its a lot more complicated due to a history of unconstrained application file management.  While the directory structure suggested where files should go there wasn't anything preventing them from being written all over the place and overwriting system files.  This led to crazy solutions like Windows File Protection which tried to automatically fix the damage after it happened.  While system damage problems have been mostly solved with the introduction of Windows Resource Protection in Vista, many badly-written and legacy applications still store user files in the same directory as the application itself - resulting in permissions problems when multiple users access the same files.  Newer applications should use the appropriate special folder but some access them by an absolute path and don't follow if the target directory is relocated.  By "should" I mean that they can't save them elsewhere because the "Designed for Windows" logo requirements prevents them from doing so; in other words - security by marketing agreement (Bad Sony! You don't get a logo!)  On Linux, user applications running under a user's account can write to their home directory and not much else.  With the rest of the system being open source a distribution's staff can fix them if they don't follow the FHS rules.

With previous versions of Windows I've attempted to move user home directories and special folders to a different drive and it was a mess. While some special folders like My Documents had a Target property that could change the target directory and move the contents, others like Favorites didn't have that option and required editing the registry and manual copying.  I would try to move the files using XXCOPY but dealing with access control list (ACL) problems and applications that insist on bad file management made it more hassle than it was worth.  In Windows 7 all of the special folders are now relocatable but legacy applications are still a problem.  In XP each user had the "My Documents" special folder while other user file locations like "My Music", etc. were just normal directories contained within it.  In Windows 7 these are all special folders and are no longer nested in My Documents.  This is helpful because they can be redirected to different locations instead of being dragged along with My Documents.  Another annoyance in XP was that other user-related application data was stored in a different directory higher in the tree.  It is now hidden within the user's home directory (same as Linux).

On Windows, NTFS supports volume mount points or "mounted drives" which act similarly to directory mounting options in Linux.  You can target a mounted drive to any empty directory.  During installation I reserved a portion of the virtual drive for a 2GB partition.  Since C:\Users always has at least the initial user account in it I couldn't mount it there.  I created a directory for a new user and mounted the partition there using the Disk Management tool.  I then created the user account and changed the ownership and permissions on the directory.  ACLs make this complicated with inherited permissions and such.  Linux directory permissions are much simpler but and add-on for ACLs is available.  Normally the user home directory and special folders are only created when the user initially logs in so I logged out and back in as the new user.  I've noticed that the initial "Preparing your desktop" operation takes a very long time with nothing obvious to account for it.  The new home directory and special folders are created and probably the initial registry for the account but I couldn't tell what else it did.  It's at least twice as long as my XP VM and I did make another account without the mounted drive just to make sure it wasn't interfering.  On Ubuntu adding a user takes only a few seconds including storing the password, setting group memberships, and copying /etc/skel to the new user's home directory.  There is an additional few seconds delay with the initial log in with Gnome where it creates its configuration directories but that's it.  After the long pause I found the mounted drive directory under C:\Users wasn't used.  Instead it created a new directory as <username>.<domainname> as it would if the system had joined a domain.  To be able to use the mounted drive I would have to change all of the special folders to use the other user directory or give it a drive letter and set them to that.  This wouldn't work for the hidden application data directory either.

Special Folders can also be redirected to a share on a server.  Not useful with a stand-alone system but common in larger business network environments.  This doesn't help a lot of legacy applications for the reasons discussed above but it's convenient when users need to access the same data from multiple systems and allows for simpler backup.  I knew I could redirect the special folders to a Samba share with the standard Windows SMB protocol but I wanted to try to connect them to an NFS export.  Adding NFS support to Windows 7 was fairly easy but I didn't bother with the name mapping for my simple test.  The add-on does have a lot of options for setting default permissions, character encoding, buffer size, etc.  NFS is not a browsable protocol so you can't just click on the Network icon in Explorer and find servers.  NFSv4 adds a pseudofs option that can be used to create a read-only share that lists all available shares which can be used to connect to the actual shares.  If you know the exact address of the pseudofs export you can use the "\\<servername>\<sharename>" syntax in Explorer's address bar to see them but I wasn't able to connect to the actual shares this way.  I also could not see any way to map NFS shares to drive letters in Explorer.  With SMB shares you right-click on the server's share list and select "Map Network Drive" but since NFS is not browsable at that level you have nothing to click.  I was able to use the Windows "mount" command to map drive letters using either the Explorer \\<servername>\<sharename> or Unix <servername>:/<exportname> syntax but there's no option to reconnect at login.  There are ways to automate NFS drive letter mapping but it is a missing feature regardless.  Being able to map NFS exports to a drive letter is is enough to prove that special folders can be mapped to an NFS share.  There are some issues with extra crud being left behind when some files are copied to the NFS export but I didn't test for it.



^ 11. Sharing Almost Redefined


During the installation, if Windows Setup detects you are connected to a network, you are prompted as to what kind of network the system is connected to - home, work, or public.  Selecting home takes you to the "Homegroup" configuration.  If you click the "Tell me more about homegroups" link the help pops up a blank window.  Actually there's not much more to homegroups than that.  It looks like there are basically five parts:  an extension of network profiles, new special folders for file management and sharing (Libraries and Homegroup), SMB file sharing, SSDP, and a new generic file sharing account (AlphaUser$).  To test this I set up another Windows 7 VM and tested between it and the first.

The network profile extensions are easy to understand.  Take multiple profile support in NetworkManager and extend it to shared directory permissions, control of some network services like Samba, and firewall profiles and you have it.  There are three profiles for home, work, and public with increasing levels of lock-down.  In the firewall they correspond to the profiles private, domain (i.e. Active Directory), and public.  The Homegroups are only active with the home profile.  A relatively obvious and trivial extension of network settings control so it's probably only worth a dozen patents at the USPTO.

With XP the starting point for navigating user files in Explorer was "My Documents".  In Windows 7 the new special folder Libraries is the starting point but it only exists in Explorer and not in the underlying directory structure.  The next level of special folders under Libraries are the media category libraries Documents, Music, Pictures, and Video (more can be created).  Their contents are the aggregate of files contained in the sub-folders which is very confusing if there are duplicate file names.  These also are not represented in the underlying directory structure but do exist as files.  For example, the library Documents file is C:\Users\<username>\AppData\Roaming\Microsoft\Windows\Libraries\Documents.library-ms.  You can delete them from the navigation panel which deletes the library file and hides the sub-folders in the Libraries view but this has no effect on the actual sub-folder directories or contents (some really bizarre architecture here).  The default Libraries can be restored from the context menu of Libraries which causes any sub-folders to reappear.  Under each media category library are two more special folders, a read-only private one and a read/write public one; "My Music" and "Public Music" for example (the missing RIAA's Music is conspicuous.)  In the properties panel of the media category library object you can set which of the libraries within are the default save location.  On Linux this would be similar to changing the settings in the local XDG config.  There is also an option to optimize the library for various kinds of content but I'm not sure what it is for (indexing?  compression?)

Homegroups are a network representation of the Libraries.  It is simply SMB file sharing with a different view and authentication mechanism.  It doesn't work with anything except another Windows 7 system which will annoy a lot of users (see comments at the bottom of the linked page).  This could be another rusty bent nail in the coffin of XP but normal SMB file sharing is still available.  In Explorer's navigation panel there is a Homegroup location.  Under that are objects for users on various systems instead of just systems as you would see when browsing the SMB network.  Under each is the same structure as the Libraries.  Users can select which libraries to share with the Homegroup (adding them to the Homegroup group) or specific users and allowing read-only or read/write access.  If a file is dropped on a media category object it goes into whichever library within that has read/write access with probably some rules for choosing between multiple valid targets.  From the network view each user has a private and a public directory making it easy to keep files organized.  Well it would be if it had been implemented correctly because each user's public directory on the same system is actually the same shared directory C:\Users\Public (another brainless design decision).  If you check the sharing properties of the Public directory you find that sometimes only the current logged-in user (along with Administrators and maybe Homegroup) is shown as having share access when in fact all local users have access but the dialog randomly hides them.  Security through confused obscurity?  Since the Public directory is shared by all users, everyone has a sharing veto.  I can't imagine this scaling well.

The new authentication mechanism includes SSDP for resource discovery, the Homegroup group, and what appears to be a new generic sharing account AlphaUser$.  In a way it acts as a new "network guest" account.  I think the idea behind AlphaUser$ is to use it as a proxy for sharing when a directory service is not available to authenticate users between connected systems.  I think that when a file transfer occurs it's done under the AlphaUser account and then the ownership is changed on arrival.  A password is needed to join a Homegroup and one is randomly generated when they are set up.  SSDP has been used before in conjunction with UPnP.  I accidentally discovered that the auto-complete function in Explorer's address bar showed some of my previous Homegroup locations in an unfriendly path referencing SSDP:

\\Provider\Microsoft.Networking.SSDP//uuid:27088a07-14dd-438f-8433-bc5933be615c


There are other problems with the current system.  One user couldn't access the shared folders in Homegroup on the same system but could access everything on the other system.  I also found that if a user disables sharing on a media category library it sometimes causes the sub-folder to show the wrong sharing icons.  This makes it difficult to understand what permissions are applying to where.  The Library special folder seems to just add unnecessary depth to the tree.  The name for it is odd but I can't think of a better one.  Perhaps the intent is that completed work will be stored there and the desktop used for files that are currently being edited.  I know a lot of users that work that way.  I've never liked Explore and when it was introduced in Windows 95 I still strongly preferred File Manager with two vertical panes.  I tolerate Nautilus on Ubuntu, probably because I have two monitors and it's "Places" side panel is a lot less cluttered than Explorer, especially with the new Libraries and Homegroup objects.  On Windows I always use xplorer².

A default install of Ubuntu doesn't include SMB file sharing although it's easily enabled by adding Samba.  Gnome and KDE (on Kubuntu) do have integrated SMB network browsing that doesn't need Samba.  Sharing between users on the same system is easy; maybe too easy.  One policy I really don't like is every user's home directory is readable by everyone else on the system by default.  This is done to ease sharing but is unnecessary.  Parents don't necessarily want to share everything with their kids like "marital photos".  Each user has their own group (USERGROUPS=y in /etc/adduser.conf) and their home directories are set to be readable by members of their group.  I normally remove read access for others (and newly created users with DIR_MODE=0750 in adduser.conf) because if one user needs read access to another user's home directory that user can be added to the other's group.  This provides a lot more control than with the default settings.  For ad-hoc sharing I have a "local" directory that all local users can write to and a "public" directory that is the same but shared via Samba and NFS.  To get around authentication problems I normally set Samba to treat unidentified users on the network as guests ("map to guest = bad user" in /etc/sambla/smb.conf).  Using a specific account with a password as a proxy (like Homegroups) would make sharing between systems easier and more secure.



^ 12. Epilogue


Windows 7 beta seemed relatively stable but I wasn't really installing much or putting it under continuous use.  I had a lot more problems with the initial release of Vista.  I did manage to crash Explorer a few times without trying but this is a "beta" which is the equivalent to an "alpha" for most other software projects.  This is the reason why many system administrators wait until the first Windows service pack before mass deployments.  Since they're skipping a second beta you might want to wait until SP2.  Of course version numbering and service pack availability is taking on a marketing influence so it's getting more difficult to know when it's safe to deploy.

Some readers may get the impression that I'm against commercial software or closed-source.  I don't have a problem using either actually.  I've used the Linux versions of Perforce and TestTrack Pro, Nvidia's driver for my graphics card, and games like Quake 4.  Given equal functionality I'll take an open-source product over a closed one as it gives me the option to maintain it if the developers abandon it, which reduces risk.  I don't use Linux because it's free.  I could easily use BSD or Solaris.  I use Linux because I won't use Windows unless somebody's paying me to.  Considering how long I've used Microsoft's products that's saying quite a lot.



Practical password security
2009-02-09T17:17:00.011-05:00
A recent security breach at phpbb.com resulted in an intruder obtaining and publishing thousands of member names and passwords.  A design flaw, a.k.a. bug, in a mailing list application was responsible. An analysis of the passwords revealed some interesting facts about the types of passwords people use when creating accounts at web sites.  The most popular ones were "123456" and "password".  A similar pattern was found in passwords exposed by a fake MySpace site in 2006.  While intrusions at non-critical sites like these aren't likely to ruin your life it's a lot more serious if they manage to get access to your account at your bank or credit union web site.  Lets look at the types of password problems I've seen and what you can do make yours safer without a lot of hassle.
First, security is like a chain - it's only as strong as the weakest link.  Even with a secure computer that is connecting to a secure web site using a secure network connection a weak password pretty much defeats the security.  There are three ways intruders can get your password without your direct assistance.  By "direct assistance" I mean you telling them (in other words, lying still works) or by writing it on a sticky note and pasting it on your computer where everyone in the room or those looking through a window can see it.  The remote methods include installing spyware on your computer or the web server your are connecting to, guessing your password based on what they know about you (pet names, phone numbers, favorite foods, favorite cars, etc.), or using another computer to try every possible password (called a brute force attack).  The last one is often used with a method known as a dictionary attack which uses dictionaries of known words to check against.  This works faster because most passwords are words instead of random characters since they are easier to remember.  There are dictionaries for every language.  There are also dictionaries for special categories like scientific fields, entertainment, or industries.  For example, a biology dictionary may contain scientific names of plants, animals, and fungi.  An attacker could include it if they knew you were a biologist in case you used the name of a bacteria for part of your password.
The security strength of a password is directly related to its unpredictability (from the attackers point of view).  If the password is a word in the English language then it's more predictable than random characters.  If it's a word relating to you then the more the attacker knows about you makes it more predictable.  A long password is usually less predictable than a short one.  A password made up of several related words like "big red truck" is weaker than a password made up of several unrelated words like "plastic quickly artichoke".  A password using more types of characters (lower case and upper case letters, numbers, and symbols) is stronger than one that only uses lower case letters.  Intentional spelling errors can make a password stronger but common errors or alternate spellings (including English dialects, Engrish, and Leetspeak) are more predictable and probably in password dictionaries.
Another problem I find with most users is that they use the same password with every account on every site.  If you do this and someone figures out the password for one of your accounts then they have access to all of your accounts.
Another password weak point is the password recovery functions at most web sites.  These allow you to reset your password if you forget it.  They usually require you to enter your account user name or email address and then send a reset web page link to the email account that is registered with the account.  You then click the link and are then given either a temporary password or allowed to enter a new one.  If your email account has a weak password and an intruder gets in then the password reset functions at every site you have an account on can be used by the intruder to set new passwords and get access.
Obviously the best security is to use different big random passwords at each site but these are very difficult to remember.  The solution is to use a password manager.  This is a program that keeps track of passwords.  You could store all of your passwords in a text file or word processor document but if an intruder gets access to your computer they could easily open and read them.  Password managers store passwords in an encrypted file that is itself protected by a master password.  Encryption is a process of scrambling something so that it is unreadable without the correct key.  When you try to open the file with the password manager it asks you for the master password (i.e. the encryption key) and attempts to unscramble the file.  If the file is unreadable then it knows you didn't enter the correct key.  If an attacker gets the file but doesn't have the key all they will find is scrambled gibberish.  There are many encryption methods with varying levels of speed and complexity but they still rely on you to create a strong master password (encryption key) to secure the contents.  With the password manager you only need to remember the master password for the password file - the rest are available once it is decrypted and opened.  You can then copy the passwords for your other accounts from the password manager and paste them into your web browser or other programs as needed.  While there are many different password managers available for all kinds of computers the one I recommend is KeePassX.  It's free and available for Windows, Mac OS X, and Linux (Ubuntu, Mandriva, etc.).  I normally install it on any computer I set up.
A password manager isn't a perfect solution.  If you use it on a computer that has already been infiltrated and has spyware on it, the intruder can get your password manager's master password by reading what you type when you enter it.  But outside of that, it's rather secure with a good strong password.  In fact, with a strong master password, you can make the encrypted password file publicly available and not worry about anyone being able to read it because only you have the key.  You can put the file on a public Internet site or, if you have a web-based email account like Google or Yahoo!, you can email it to yourself so it's stored in your email inbox.  That way you can get at your passwords from any computer on the Internet with the password manager program installed - just make sure they are secure first before entering your master password or even logging into your email account to get the file.  Of course, make backups of your password file by emailing it to yourself or saving copies of it to a USB flash drive.
For creating secure account passwords most password managers have a password generator.  It can create random passwords of varying lengths using numbers, letters, and symbols.  The more variety the better.  For example, if you have a password that consists of a single lower-case letter, there are 26 possible passwords that an attacker will have to try to break in.  They may get lucky on the first try and find it's an "a" or they may have to try them all and find it's a "z".  With two characters there are 26x26 or 676 possibilities.  Add another character and it's 26x26x26 or 17,576.  If you include upper-case letters you now have 52, 2,704, and 140,608 possibilities for one, two, and three character passwords.  Add numbers and you get 62, 3,844, and 238,328 possibilities.  Using every printable character on the keyboard (including a space) you end up with 74, 5,476, and 405,224.  Way to many to try by hand but remember that most attackers on the Internet are using a computer to try each combination and can make millions of attempts every second.  To be relatively safe you should have at least 12 characters.  This makes it unlikely for an attacker to determine your password in any reasonable amount of time (many years) even if they are using thousands of computers simultaneously.  The more complicated your password is the more likely they are going to give up before breaking it and move on to another target.  In general, both the length and number of different characters affect the strength of the password so if you use fewer character types then use a longer password to make up for it.
While the password generator can make complicate passwords and the password manager can keep track of them, you still need a strong master password.  Ideally it should be random and long but it's really hard to remember something like that.  A technique I've used with employee accounts on business networks is to have them create a short secret password consisting of words and a few extra numbers and symbols that they can remember.  Then add several random characters before and after it for the full password.  Then write down the random portions on a piece of paper with a blank line between them signifying the secret part and store it someplace out of sight.  When entering the password they use the paper for the beginning portion, followed by the secret part they didn't write down, and then the rest from the paper.  This is practical because you can generally trust coworkers (or household members) more than anyone on the Internet.  Even if someone finds the paper they don't have the full password while an attacker on the Internet doesn't have any of it.  If you have trouble coming up with random words for a password you can try a technique called Diceware that uses dice and a word list.
One problem you will encounter is that web sites have varying rules about passwords.  Some require between six and twelve characters, some allow much longer.  Some only allow letters and numbers.  Some only allow some symbols while others allow almost anything.  Unfortunately many web sites don't specify their rules entirely so you may have to make several attempts to find one it will accept.  You may find that a web site will accept a 16 character password when you set up a new account but actually only allow 14 characters and chop off the last two without telling you.  When you enter a new password into a web site test for this problem by logging out of the web site and back in again with the new password.  If it rejects it, delete one character from the end of the password and try logging in again.  If you get down to the minimum number of characters the site will allow and you still can't log in, use the site's password reset/recovery function to get access again.  If deleting some characters from the end allows you to log in make sure to note how many characters it accepted in your password manager so you don't end up fighting the site again the next time you change your password.  Another problem you may encounter is a web site that accepts a new password with symbols in it but filters them out, again without telling you.  If you can't log in with a new password and making it shorter doesn't seem to fix the problem, try deleting any symbols in the password leaving only the numbers and letters.  Again, if you can't get access then use the site's password reset/recovery function.
Sometimes you will find a site that will accept a strong password but then does a bad job of keeping it secret.  They do stupid things like confirming your password by emailing it to you unencrypted which means that at every point in the Internet that the email passed through someone could read your password.  Some mailing lists also email a password reminder to you every month again exposing it to the whole world.  There isn't much you can do about these bad security lapses except not use the sites and complain to the administrators.
Using complicated passwords is only part of good password security. The other is changing them regularly.  If an attacker can brute force your weak password in a month and you change it every two months then you have a security problem.  Changing it before they break in makes them have to start over again.  While using a stronger password requires an attacker to take longer to break in, you should change your passwords regularly anyways to limit damage in case your computer or one you are connecting to has an intruder you're not aware of.  How often depends on the strength of the passwords used with each account and how much damage could be caused if someone breaks into them.
While strong passwords reduce the chances of someone finding your account passwords without your help don't overlook the age-old method of social engineering (lying).  Normally if you set up a new account on a web site it will email you a confirmation link to verify your email address.  You will also get an email from the site if you use their password reset/recovery option.  But later, if you get an email from the site requesting that you click a link to respond to a problem with your account, especially if it's a bank or store account, be very suspicious.  These are often used for phishing.  Scammers often send you an email like these with links to a fake web site for you to log into in order to get your passwords.  These often refer to popular companies and are sent blindly to millions of email accounts which is why you sometimes find an "account security notice" in your email inbox for a bank you don't have an account with.  Most financial institutions have a policy of never contacting you by email regarding security problems and will call you instead.  You can also sidestep the phishing by going to the web site directly as you normally would instead of clicking a link in the email.  If there is a problem then you should be alerted when you log in to the web site.  Most web site operators will never ask you for your password as they usually have other methods (like their internal computers) for accessing your data.
While strong passwords can help keep intruders out of your accounts note that there is no perfect security.  What security systems and passwords attempt to do is to add a limited amount of annoyance for legitimate users and multiply the annoyance factor many many times for attackers.  While some security systems and software are better than others often the limiting factor is your willpower versus that of potential attackers.



Windows GUI vs. Linux Command Line Myths
2008-10-26T13:39:00.013-04:00
Undoubtedly you've heard the old cliché that Windows is easier to maintain because it has GUI tools for everything while Linux requires commands lines and a terminal.  Any experienced Windows administrator knows the point-and-click GUI tools don't cover everything. Likewise any experienced Linux administrator knows there are many GUI tools for Linux configuration but terminal shells are available on ANY system regardless of how big or small and the ability to script any action in a platform-neutral way is too useful to give up.  I just again encountered a situation on XP that required a command-line fix and it highlights the ignorance of many fanboys about the reality of Windows system administration.
I recently installed Windows XP Pro from scratch on a dual-boot system.  I normally install Windows first as it doesn't play well with other OSes when it comes to the boot loader.  I was using an original XP OEM CD with SP1 integrated.  After installation I copied over SP3 and installed that as well.  Doing it this way reduces the number of update/reboot/update cycles I have to go through with Windows Update and reduces the risk of an exploit before the process is complete.  After rebooting, I run Windows Update and go through the usual Windows Update update, Installer update, activation, and WGA check.  I then install all of the critical updates.  There are a surprisingly large number of them considering I already have SP3.  Reboot again and run WU again and install NET Framework runtimes, IE7, Media Player 11, and more updates for the updates.  Reboot again and go back to WU again.  Install more updates for the updates and everything else I just added.  Or at least I tried as they refused to install, reporting "failed" for all of them.  I went through the typical diagnostics Windows admins have learned over the years of deleting out temp files, clearing the browser settings, and attempting to install each update individually to no avail.  Some Google searching turned up a blog posting about Wups2.dll not being registered properly if the system is updated through WU and not rebooted before SP3 is installed (KB943144).  Of course this doesn't explain my situation as WU hadn't been used before service pack was installed.  The workaround requires stopping the WU service, manually registering the dll from a command window, and restarting the service.  This fixed my problem.
This isn't an unusual repair process for a Windows system.  Even for Vista there are plenty of examples of command window (cmd.exe) and regedit repair instructions in Microsoft's support pages.  You can ignore all the myths floating around the Internet about never having to use Unixy command lines when administering Windows systems because of the wonderful graphical tools.  On Windows there are many tasks that are impossible to perform with graphical tools or are just a lot easier from a command window.  The only way to avoid command line tools or regedit entirely is to write a custom graphical tool that handles those specific situations (similar to "compiling a kernel" comments from Microsoft fanboys).  The fanboys will point out that regedit is a graphical tool but the reality is that it isn't much more of a "tool" than Notepad (which was used in the pre-registry days with win.ini and system.ini).  An IT manager that hires an admin that doesn't know how to use regedit or command-line tools should themselves be replaced.  When screening job applicants I've encountered many "certified" admins that didn't know anything about maintenance outside of the graphical tools (or even basic hardware troubleshooting for that matter).  Surprisingly, I've also worked with software engineers that had a paranoid fear of even regedit.  It's like they've been brainwashed into thinking that the only "proper" way to work with the registry is to use an API and approved function calls.  Apparently they haven't experienced the "fun" of trying to remove auto-starting malware entries from it.
Because of the emphasis on graphical tools the skill of working at a low level with the Windows OS is a dying art.  While the graphical tools lower the barrier for entry into system administration it also invites fools (with only superficial skill) to enter (and get certified) without low-level skills valuable for troubleshooting.  Graphical tools provide them a flower-strewn path to anywhere they want to go but when a situation calls for them to go off the path they are lost - much to the pleasure of seasoned consultants who will guide them back to safety for a hefty fee.  System administrators are not the types of users that recovery disks were intended for but unfortunately a lot of amateur admins rely on them.
The fundamental limitation of graphical tools is that trying design an interface for every conceivable configuration option, troubleshooting situation, and maintenance function ends up making the tool more complex and time-consuming to use than the task itself.  There are occasions when GUIs are easier than command lines but it's usually a situation involving an over-complicated design of the underlying system than a practical improvement in efficiency.  The hierarchical structure and relationship of keys and values in the Windows registry is relatively simple but the file format makes regedit a necessity.  Typing in a registry key path to a command line application like reg.exe, especially one that includes a GUID, is painful.  On Linux you can experience a similar difficulty when trying to work with an XML configuration file like the one pam_mount now uses.
Graphical configuration tools like regedit are not unique to Windows.  Gconf-editor provides a similar interface to the Gnome GConf settings database.  But the terminal isn't going away anytime soon as it's too powerful and even on Windows the DOS-derived command window is still present.  Windows admins have learned to live with its limitations, switched to higher-level programming languages, or extended it with third-party utilities like KiXtart (which I've used).  The Windows PowerShell is Microsoft's attempt to replace this last remnant of the DOS era and it's legacy syntax.  This may be their admission of the limitations of a GUI or just be a response to the popularity of headless systems in data centers and the need for a replacement to a 20 year old shell.  I haven't tried PowerShell myself as I moved away from the Windows platform before it was released.  With the availability of virtualization I now just use Windows as a bloated runtime for legacy applications and I don't need to do scripting anymore (although I'll admit to playing with batch files in FreeDOS once in a while).



Disabling Suspend and Hibernate Buttons in XFCE
2008-10-24T13:28:00.003-04:00
A few months ago I showed how to disable the buttons in Gnome and KDE.  Occasionally, I use Xubuntu with XFCE on older systems with limited memory available.  Disabling the buttons is easier in XFCE.  Simply go to Applications > Settings > Settings Manager > Sessions and Startup > General and disable the show options for both buttons.
The problem is that this setting is per-user so you have to do it for each user account individually.  The settings are stored in "~/.config/xfce4-session/xfce4-session.rc".  If you want to use them as the default for new accounts then copy it to the equivalent path in /etc/skel and set the ownership to root.  The skel directory is used to create the default home directory structure for new users and the ownership will be changed to the user's account automatically.



Improving Windows XP guest in VMware Player
2008-09-20T13:54:00.010-04:00
I use XP in VMware Player to run some CAD applications on my Ubuntu system.  I don't actually have to use XP for them as they function under Wine but I've been too busy to reinstall them and recreate their configurations.  This setup works more or less but there are a few bugs and performance problems I've had to find workarounds for.
The first thing you need is RAM obviously.  XP will function with 256MB of memory allocated to the VM but you'll need more for any app larger than Notepad.  The host desktop (Gnome in my case) also needs a lot, how much depends on what native apps you have running.  Open up Firefox with a bunch of tabs with Flash, Java, and Acrobat plug-ins active and you can easily use up 1GB total.  For the VM you want to allocate enough for your guest applications but not exceeding what the host can provide else what the guest thinks is RAM will be paged out to disk by the host and it gets incredibly slow.  If your host is a bit short on RAM then try switching to a lighter desktop like XFCE (Xubuntu), LXDE (Ubuntu Lite), Openbox, or even just xterm.  If you're building a new system and plan to use VMs a lot then I recommend getting at least 2GB of RAM.
If you have a lot of memory available then you can improve speed in a XP guest by disabling the paging file.  You'll find a lot of web sites warning against this but it's no more risky then running out of both RAM and disk space with the paging file active.  If your guest apps use more than whats available then you will get an out of memory error either way, except with a paging file active you will often trash the registry in the process as it can't be written to.  To turn it off go to Start > Control Panel > System (or press the logo key + Pause), Advanced > Settings (Performance) > Advanced > Change (Virtual Memory).  In the Virtual Memory window select "No paging file", Set, Ok, then apply the setting (in Windows terms "apply" means reboot).  Afterwards you may notice that the Windows Task Manager (Ctrl-Alt-Delete or logo key + R and then run taskmgr) will still show PF Usage >0 in the performance tab.  Ignore it.  I'm not sure if it's including some other non-memory related temporary file usage or is just an estimate but the important thing is that the hidden pagefile.sys in the root of the drive is gone and the setting can be confirmed in the registry.
Because I do a lot of file management from both Nautilus and the VMware guest I set up a shared folder and redirected the Windows special folder "My Documents" to the share.  To set up a shared folder in VMware Player you need to manually edit the guest's vmx file.  Here is a sample of the entries:

sharedFolder.option = “alwaysEnabled”
sharedFolder0.present = "TRUE"
sharedFolder0.enabled = "TRUE"
sharedFolder0.readAccess = "TRUE"
sharedFolder0.writeAccess = "TRUE"
sharedFolder0.hostPath = "/home/user/vmware/shared"
sharedFolder0.guestName = "shared"
sharedFolder0.expiration = "never"
sharedFolder.maxNum = "1"

You may have to enable it from the Player controls bar under Player > Shared Folders.  The shared folders show up as a network location "\\.host\Shared Folders\shared".  To set My Documents to this location right-click My Documents > Properties > Target then enter the path in the Target box and move your existing files there if desired.  All of your applications should now be able to access the files the same as from the original location.  I emphasize "should" as special folder usage in Windows is a Microsoft "suggested" practice that is not enforced by file system permissions.  Some apps access the My Documents folder directly by an absolute path like "C:\Documents and Settings\User\My Documents" and they will ignore the target change.  It also doesn't affect apps written in the "traditional" DOS programming style with behavior like writing user files to the same directory as the application executable.
This method eliminates having to synchronize files between the guest and host.  You could also set up a Samba share on the host or another system and redirect My Documents to that instead but you will need centralized logins between your Linux and Windows systems to prevent permissions problems.  One behavior you need to be aware of when using using shared folders is that VMware will often lock the files while the guest is running.
One problem I discovered with shared folders is that there is a long delay in accessing them.  I found out the delay is caused by Windows trying to find a system named ".host" on the network and the solution is to define it as local host.  Browse to the directory "C:\WINDOWS\system32\drivers\etc".  If there is no "lmhosts" file present then copy lmhosts.sam to lmhosts.  Edit it with Notepad and add the line "127.0.0.1 .host".  Apply it (reboot again) and the delay should be gone.
I'm using a multi-head (non-Xinerama) monitor setup where I can run a VMware guest full screen on one monitor and have the Gnome desktop (or another VMware guest) on the other.  One problem that I've encountered is that the Ctrl, Alt, and Shift keys would occasionally stop responding and eventually any Linux app I tried to type into would crash.  This is a known bug in VMware and the solution is to run "setxkbmap".  Since even a terminal window can crash when the problem is occurring I found it easiest to create a custom application launcher on the Gnome panel so I can just click it with the mouse.  The fix is instantaneous and doesn't require VMware to be shut down.
Another oddity with Player on my multi-head system is that if it's running on the secondary monitor and I switch to a VT (Ctrl-Alt-F1, etc.) the vmplayer process disconnects from vmware-vmx process and the guest window is gone when I switch back.  It doesn't happen on my primary monitor.  If I run Player again it will reconnect to the running VM if I tell it to run the same one again so it's not fatal, just annoying.
I mentioned above about running two VMs at the same time.  To save time you can use the same base VM and make duplicates of it for each with some minor modifications.  You can also use them on different PCs.  Regardless, there are some caveats to running multiple XP VMs simultaneously.
First is licensing.  Make sure you have a valid license for each one.  It's possible to change the product key after installation which will allow you to re-authenticate with Microsoft if necessary.  This will save you from having to perform a reinstall of XP from scratch on each system.  One thing to remember about VMware - it doesn't virtualize everything (the host's CPU for instance) and some parameters like the VM's MAC address are unique to each guest.  These can be used by Windows Genuine Advantage to identify the system.  Changing them can cause a revalidation prompt.
Second, some things must be different with each guest, the VM directory (obviously) and the aforementioned MAC address.  The MAC address needs to be different in order for your DHCP server to assign a unique IP address to each one.  By default Player uses an auto-generated MAC address which appears to be based on the VM UUID.  Copying a VM will result in the same UUID and MAC address.  To change it you need to change the UUID or set the MAC address manually in the vmx file, either of which can set off WGA revalidation.  VMware also requires a different address range, 00:50:56:00:00:00 - 00:50:56:3F:FF:FF, for static values.  Example:

ethernet0.addressType = "static"

ethernet0.Address = "00:50:56:3F:FF:FD"

Another parameter you may want to change is the displayed name of the VM which is shown in the "recently used" VM list in Player:

displayName = "Windows XP Professional - Testing"
On one of my systems Player would start misbehaving some time after installation.  The guest VM would operate way too fast which would cause XP to freeze during boot probably due to timing problems with devices between the guest and host.  This bug is caused by VMware misidentifying the host's maximum CPU speed due to power management or maybe ACPI problems.  Minor timing problems often show up as creeping RTC (real time clock, aka "time of day") errors.  The solution is to manually set the speed in "/etc/vmware/config".  For a 2.53GHz CPU (cat /proc/cpuinfo):

host.cpukHz = 2530000
host.noTSC = TRUE
ptsc.noTSC = TRUE
There are many other parameters you can tweak in the vmx file.  Some I've found usefull are ide1:0.fileName = "/dev/scd0" which had to be changed when IDE device names changed in recent kernels and the various "present" and "startConnected" parameters for specifying the default state of devices.



Abusing your deb package manager
2008-07-13T20:50:00.005-04:00
Normally all applications should be installed using your distro's package manager in order to set up dependencies correctly (like libraries).  Once in a while you may encounter a problem with either a broken package database or synchronization problem due to hardware faults or naughty user behavior (like deletion of an application's files manually).  The solution to a broken package problem is to first let the package manager try to fix it.  On Debian or Ubuntu systems Synaptic has a menu option to fix broken packages.  The console package manager "aptitude" also has a broken package filter and the command line tool apt-get has an "-f" option.  But there are limits to what kind of a mess they can fix and sometimes you have to tread into the risky world of tool abuse to get the job done.  One of these methods are the apt-get "force" options.  For example, I wanted to reinstall the free Linux version of AVG Anti-Virus as the previous version was failing to update it's virus definition database due to a script or license key problem.  I'm not worried about "theoretical" Linux malware but I do need to check Windows files for viruses prior to using them with Wine.  But I'm using Ubuntu 64-bit on my AMD Phenom system and AVG only has a 32-bit version available which the package managers won't install due to the architecture mismatch.  That's not as serious of a problem as it looks since most any 64-bit distro and CPU can also use 32-bit applications but the package managers take a very narrow-minded view of it.  To get around it I used:
dpkg -i --force-architecture avg75fld-r51-a1243.i386.deb
But this didn't work when I first tried it.  I had been doing some file management with some other manually-installed applications (i.e. not controlled by the package manager) and apparently deleted a link or two that the AVG package scripts looked for when uninstalling an old version.  This may explain while it was failing to update in the first place.  The failing "prerm" script caused dpkg to abort the install and there wasn't any command line option I could find to force it past the problem.  I wasn't worried about breakage as I expected the new package to replace all the files of the old but I didn't want to extract the files from the deb and move them manually.  The Debian/Ubuntu package management system keeps track of package status in "/var/lib/dpkg/status" and it's an easy to read text file.  I searched for "avg75fld" and found this line:
Status: install ok installed
I changed the ending "installed" to "not-installed".  I ran dpkg again and AVG installed without problems since dpkg stopped trying to remove the previous installation first.
I don't recommend abusing your package manager out of habit as it will lead to more problems in the future.  Since a package manager has very significant control over your system and often is what sets distros apart from one another, you should learn how it operates and try to work with it instead of against it.  For Ubuntu and Debian systems see Debian FAQ and Ubuntu documentation.
A word of warning - although Ubuntu is based on Debian the packages in their repositories are not the same so don't mix them.  A stand-alone deb package shouldn't be a problem as long as no other packages depend on it.  Several projects not in the repositores like Webmin and Lost Labyrinth are distributed this way.



Setting up a local repository with debmirror
2008-05-17T00:30:00.022-04:00
I set up a lot of PCs and while I have a fast 10Mbs Internet connection I wanted to utilize my faster internal network bandwidth better.  With a new distro release it's less important as most of what I need is on the CD but as updates are released I end up downloading increasing amounts of data for each install.  I've been doing lazy tricks like copying /var/cache/apt/archives to a network-shared directory but it's sloppy and multiple versions of packages accumulate.  Setting up local repository was the answer for me.
You use debmirror to create a local repository for Ubuntu and Debian systems.  Instead of duplicating an entire repository server you can select by release (feisty, gutsy, hardy), section (main, universe, multiverse, backports), architecture (i386, amd64), and using regular expressions.  An alternative to creating a mirror is to create a caching proxy using apt-cacher.  The advantages of one over the other depends on how similar the package selection of each client is.  Caching is more efficient for serving similar systems, better at handling limited storage space on the server, and often has an earlier data transfer break-even point (the amount of upstream data transfer saved with it versus without).  Depending on what packages are stored locally, a mirror is more efficient with diverse systems but you have to plan out the space requirements beforehand.  The data transfer break-even point will take much longer to reach as many unneeded packages will be transferred unless you are very selective about which portions of the repository to mirror.  With apt-cacher there can be less latency when subsequent requests for a newly published package are received as the initial request retrieves it immediately while debmirror updates are usually controlled via a cron job.  Currently both debmirror and apt-cacher require the clients to be configured to use the new source so there is no administrative savings there.  But apt-cacher does have to potential to support an intercepting (a.k.a transparent) proxy configuration if Debian bug #352140 is resolved which would eliminate client configuration.
Because I had space and rather diverse client requirements I went with the mirror approach using debmirror.  I relied on several sources for information especially BobSongs' How To but he didn't include some of the third-party sources I needed and disabled some of the secure apt checks.  Not every repository uses these security features but I prefer to err on the side of caution.
First, you need to install debmirror using "apt-get install debmirror", aptitude, Synaptic, or Adept.  Next set up a location to store and share the mirror.  I set the root of mine to "/srv/public/linux/distributions/Ubuntu/mirror".  I also have ISOs and other files in this tree which explains the depth.  The "/srv" directory is the FHS standard recommended served data location but you may prefer to dump it somewhere in /var.  Then you select the repositories to mirror and create a shell script to run debmirror.  You can create a cron job run it daily to stay updated.  Finally, to share the mirror you can use anything that apt-get supports.  Refer to "man sources.list" for the options.  I'm not going to duplicate the many HOW-TOs on setting up servers here.
For my systems I needed the Hardy i386 and amd64 versions of packages in the general Ubuntu, Medibuntu, Wine, Google, and Skype repositories.  First you need to set up a key ring for debmirror which defaults to "~/.gnupg/trustedkeys.gpg".  On systems like Ubuntu which disables direct root logins and uses "sudo" instead, I create a special "administrator" account which is in the admin group and has a high-strength password (since the password provides root access).  Within I created the keyring "/home/administrator/keyrings/mirrorkeyring/trustedkeys.gpg" using the following command to import the Ubuntu archive keys:
gpg --no-default-keyring --keyring /home/administrator/keyrings/mirrorkeyring/trustedkeys.gpg --import /usr/share/keyrings/ubuntu-archive-keyring.gpg
To this I added the other keys:
wget -q http://packages.medibuntu.org/medibuntu-key.gpg -O - | gpg --keyring /home/administrator/keyrings/mirrorkeyring/trustedkeys.gpg --import

wget -q http://wine.budgetdedicated.com/apt/387EE263.gpg -O - | gpg --keyring /home/administrator/keyrings/mirrorkeyring/trustedkeys.gpg --import


wget -q https://dl-ssl.google.com/linux/linux_signing_key.pub -O - | gpg --keyring /home/administrator/keyrings/mirrorkeyring/trustedkeys.gpg --import

gpg --keyring /home/administrator/keyrings/mirrorkeyring/trustedkeys.gpg --import rpm-public-key.asc
The last key is for Skype.  Their Linux support is minimal and the repository is kind of a mess.  They moved the key on their server (or lost it) but I had a copy.  The MD5 hash of my key (md5sum rpm-public-key.asc) is 2f595c0efe5d26fb4909f3347670746d and you can get a copy from this link.
Next I created my debmirror-hardy.sh script and put it in /usr/local/bin.  Most of my parameters are explicitly defined on the debmirror command line although you could create a configuration file instead (the default is /usr/share/doc/debmirror/debmirror.conf).  I specify the parameters in the same order as the corresponding directories appear in the repository path.  I used rsync with the main Ubuntu repositories as it is supposedly faster but none of the others support it so they use http.  Notice that the root for an rsync server is specified with a preceding colon (:).  The "md5sums" parameter adds MD5 checking but you may want to skip it to speed up the mirror process.  The "nosource" parameter skips source packages as the only time I need them is when I compile something outside of the distro and even then I only need the headers.  I do compile Wine to perform testing on my primary system but I get it straight from the source tree using git.  The "progress" option shows a download progress meter and I tee everything to the console so I can watch if I'm bored.  It also creates a couple of logs in /var/log and compresses the old ones to save space.
#!/bin/sh
#  debmirror script v1.1 for Ubuntu Hardy Heron
#  Copyright 2008 Jeff D. Hanson (jhansonxi@gmail.com)
#  Released under GNU General Public License version 3
#  v1.1 - added debian-installer section, post chown/chmod
#  fix, size summary, date/time

DEBMLOG=/var/log/debmirror.log
MIRRORDIR=/srv/linux/distributions/Ubuntu/mirror
export GNUPGHOME=/home/administrator/keyrings/mirrorkeyring

if test -s $DEBMLOG
then
    test -f $DEBMLOG.3.gz && mv $DEBMLOG.3.gz $DEBMLOG.4.gz
    test -f $DEBMLOG.2.gz && mv $DEBMLOG.2.gz $DEBMLOG.3.gz
    test -f $DEBMLOG.1.gz && mv $DEBMLOG.1.gz $DEBMLOG.2.gz
    test -f $DEBMLOG.0 && mv $DEBMLOG.0 $DEBMLOG.1 && gzip $DEBMLOG.1
    mv $DEBMLOG $DEBMLOG.0
    cp /dev/null $DEBMLOG    
    chmod 640 $DEBMLOG
fi

# Record the current date/time
date 2>&1 | tee -a $DEBMLOG

# Ubuntu mother lode.  At least it supports rsync.
echo "\n*** Ubuntu general ***\n" 2>&1 | tee -a $DEBMLOG
debmirror --nosource --method=rsync --md5sums --progress \
--host=us.archive.ubuntu.com \
--root=:ubuntu \
--dist=hardy,hardy-security,hardy-updates,hardy-backports \
--section=main,main/debian-installer,restricted,restricted/debian-installer,\
universe,universe/debian-installer,multiverse,multiverse/debian-installer \
--arch=i386,amd64 \
$MIRRORDIR/ubuntu \
2>&1 | tee -a $DEBMLOG

# Canonical's rather lonely partners repo
echo "\n*** Canonical partners ***\n" 2>&1 | tee -a $DEBMLOG
debmirror --nosource --method=http --md5sums --progress \
--host=archive.canonical.com \
--root=/ \
--dist=hardy,hardy-backports,hardy-proposed,hardy-security,hardy-updates \
--section=partner \
--arch=i386,amd64 \
$MIRRORDIR/canonical \
2>&1 | tee -a $DEBMLOG

# Medibuntu fun stuff
echo "\n*** Medibuntu ***\n" 2>&1 | tee -a $DEBMLOG
debmirror --nosource --method=http --md5sums --progress \
--host=packages.medibuntu.org \
--root=/ \
--dist=hardy \
--section=free,non-free \
--arch=i386,amd64 \
$MIRRORDIR/medibuntu \
2>&1 | tee -a $DEBMLOG

# Wine's latest bugs
echo "\n*** Wine ***\n" 2>&1 | tee -a $DEBMLOG
debmirror --nosource --method=http --md5sums --progress \
--host=wine.budgetdedicated.com \
--root=/apt \
--dist=hardy \
--section=main \
--arch=i386,amd64 \
$MIRRORDIR/wine \
2>&1 | tee -a $DEBMLOG

# Our friends at Google.  Including a leading / in the root causes failure.
echo "\n*** Google ***\n" 2>&1 | tee -a $DEBMLOG
debmirror --nosource --method=http --md5sums --progress \
--host=dl.google.com \
--root=linux/deb \
--dist=stable \
--section=main,non-free \
--arch=i386,amd64 \
$MIRRORDIR/google \
2>&1 | tee -a $DEBMLOG

# Skype's half-baked linux contribution.  Located in a half-baked repository.
echo "\n*** Skype ***\n" 2>&1 | tee -a $DEBMLOG
debmirror --nosource --method=http --md5sums --progress --ignore-release-gpg --ignore-missing-release \
--host=download.skype.com \
--root=/linux/repos/debian \
--dist=stable \
--section=non-free \
--arch=i386 \
$MIRRORDIR/skype \
2>&1 | tee -a $DEBMLOG

echo "\n*** Fixing ownership ***\n" 2>&1 | tee -a $DEBMLOG
find $MIRRORDIR -type d -o -type f -exec chown root:root '{}' \; \
2>&1 | tee -a $DEBMLOG

echo "\n*** Fixing permissions ***\n" 2>&1 | tee -a $DEBMLOG
find $MIRRORDIR -type d -o -type f -exec chmod u+rw,g+rw,o+r-w {} \; \
2>&1 | tee -a $DEBMLOG

echo "\n*** Mirror size ***\n" 2>&1 | tee -a $DEBMLOG
du -hs $MIRRORDIR 2>&1 | tee -a $DEBMLOG

# Record the current date/time
date 2>&1 | tee -a $DEBMLOG
This works very well so far but it took a lot of time to figure out.  One thing I noticed is that apt-get handles some repository structures better than debmirror.  Google's repository had an oddity, possibly due to a redirect, that caused debmirror to not find the Release file or detached *.gpg signature unless I left out the preceding / from the root parameter.  Skype's repository has a Release file but not where debmirror could find it.  They don't sign it either.
UPDATE:  I've made some changes to the script.  I've been having fun with PXELINUX and performing Ubuntu installs by netbooting.  This required the addition of the debian-installer portion of the repositories.  I also added time/date timestamps and a final size check (about 37GB for everything so far).  One problem I haven't found the solution for is that when I put the script in /etc/cron.daily it doesn't run.
UPDATE2:  Thanks to the comment by sq5nbg I figured out the problem with cron.daily.  The crontab entry for it uses run-parts to run the executables in the directory.  According to it's man page it is picky about the file names it will accept and a period is not a valid character.  You either have to rename the file or symlink to it.  The run-parts utility is in debianutils and bug #38022 reports this issue.  It's marked as a wishlist item since the restriction is documented in the man page.  I added a note about this to the cron page in the Ubuntu Wiki.
I need to point out that the script should be edited to use a server nearest (in Internet terms) to you instead of the ones specified.  This especially applies to the Ubuntu mirror (us.archive.ubuntu.com).  Use the Ubuntu mirror list page to find one that has the packages and protocol you want.  This reduces the load on the primary servers.
UPDATE3:  You can use your local mirror with the Minimal CD to install Ubuntu on systems that don't support network booting.  First, set up a server that provides access to the mirror directory.  I used Apache2 to serve them via http and put a link to my debmirror directory in "/var/www".  If you are using an http server you should be able to navigate the debmirror directories using any web browser.  If you can't see them then the installer won't either.  After setting up the server, boot the CD and proceed as you normally would through the boot settings and locale selection.  After specifying the network configuration and hostname you will see the "Choose a mirror of the Ubuntu archive" screen where it wants you to select the "Ubuntu archive mirror country".  Hit the Home key to jump to the top of the list and select the "enter information manually" option.  For the "Ubuntu archive mirror hostname" enter your servers hostname, FQDN, or IP address.  Do not specify a protocol prefix (http://) or any directory path on that screen.  I'm not sure if the installer tries all protocols, defaults to http, or guesses based on a specified port number but I didn't have to tell it what to use.  On the next screen, enter the "Ubuntu archive mirror directory" with the full server path to the directory containing the dists, pool, and project directories.  If you do it wrong it won't be able to find the "Release" file and you will get a "bad archive mirror" error.



Penguicon 6.0
2008-04-22T01:19:00.003-04:00
I attended Penguicon 6.0 over the weekend.  It met my expectations which were rather high but I figured it would.  More importantly, two friends of mine also enjoyed it.  One had been there last year and the other was a n00b.  Another had attended last year and would have returned but unfortunately had family matters that took priority.  I did advertise a bit around the Alpena area but I'm not sure how effective it was.  Next year I think I'll try to get enough people to fill a van or maybe a small bus and reduce the fuel cost of the trip.
I did help out with the installfest but only saw a half-dozen attendees that took advantage of it.  I fixed a networking problem on a laptop of a WLUG member which was simply an issue with a manually set network configuration which was made by another member.  It was the direct result of WCC not having a DHCP server on their WiFi.
The other was a volunteer that wasn't able to get any distro to install on an old HP laptop with a 500MHz K6 CPU.  The problem was obvious - 64MB of RAM.  I put Puppy 4 beta on it which seems stable and looks much better then v3.  I didn't have time to try setting up the network but at least he had something to start with.  He was planning on installing more memory now that the problem was identified.  It would be a good candidate for either Puppy or Xubuntu.
The only major issue at the conference was the Hilton's overloaded WAN connection.  By Saturday night it was unusable.  There was a small LAN party set up in a large and mostly empty ballroom although there may have been more players in the guest rooms.  It looked like it was a local game so it probably wasn't affecting the WAN.  I think a LAN party is kind of a waste of space as most attendees can play games online at home and there are better things to do at the conference.  Especially since they were Windows systems.  They did have some nice large LCD and HDTV screens.  The computer lounge had a proxy with ISOs and repository mirrors and there didn't appear to be a large number of users on the public PCs so not much load there either.  A virus on some Windows system somewhere could have been the culprit but there's no way to tell.  Next year the conference is moving to a different hotel as its outgrown the Hilton so it will be a different environment.



Laptop problems with Ubuntu Hardy Beta
2008-04-01T16:50:00.003-04:00
So far my Hardy experience has been good.  I've had a few crashes with Firefox 3 but it's session recovery works well so it's been only a minor annoyance.  I have mixed feelings about the new live searching in the address bar as I'm used to it only picking up URLs as in Firefox 2.  When it hits on page titles and other data it's distracting.
Some of the bugs I encountered in alpha 5 have been fixed.  SCIM can now be disabled via System > Administration > Language Support > Enable support to enter complex characters (unchecked).  The Gnome display properties applet and Gnome settings daemon no longer crash.
The Java plug-in problem may have been my fault.  The correct package is sun-java6-plugin and I may have installed a different one by mistake or it wasn't in the repo at the time.
The problem with X freezing the system during a VT switch or restart/shutdown is still present.  According to bug #204603 it's an Intel driver issue which requires Option ForceEnablePipeA "true" to be specified in the xorg.conf Device section.  This may be a repeat of a previous bug.
The battery monitor bug only occurs with one account and only with the Power Manager applet.  The Battery Charge Monitor applet shows the correct state and other in other accounts on the laptop they both work correctly.  Minor config problem somewhere.
A major usability problem I've noticed is that the drive mounting/browsing options that were in the "Storage" tab in "Removable Drives and Media" has been moved to Nautilus preferences.  The menu entry hasn't been renamed however which is confusing because it no longer affects removable media.  Bug #210499 was already filed about it so I'm not the only one who noticed.
One glaring omission in new Hardy features is the lack of networking support in Friendly Recovery - the menu you now encounter after selecting recovery mode from the Grub menu.  Unless your system is configured to use static IP/gateway/DNS you don't get any network access so apt-get is rather limited.  On a DHCP network like most broadband and wireless connections you have to manually start dhclient first.



Living on the edge with Ubuntu Hardy Alpha
2008-03-09T19:01:00.009-04:00
I was having some problems getting Wine to compile with OpenGL support in Ubuntu Gutsy on my laptop.  The config script could never find the OpenGL dev libraries.  I had been messing around with learning package management and other things and I broke something somewhere and couldn't find it.  Instead of reinstalling Gutsy I decided to test Hardy alpha 5.  This way I could report any bugs I encountered so they could get fixed before the final release and also fix the Wine problem.  The laptop is a Toshiba M35X-S114, a Celeron 1.3GHz with an Intel 855GME chipset.  It's Linux compatibility is better than average but not spectacular.  I expected breakage with Hardy and was not disappointed.  Alpha testing is living on the edge as packages are constantly being updated so system reliability can vary wildly from one update to the next.  I often would start it up in the morning, install a dozen updates, a few minutes later reload the package lists and find more updates.
The biggest immediate issue I encountered was X locking up randomly with the Intel driver.  It would also lock the system whenever I switched to a virtual terminal.  I've encountered the latter problem before when the framebuffer drivers used by the kernel for the splash screen conflict with the X drivers.  I edited the Grub menu (/boot/grub/menu.lst) and changed the "splash" entries to "nosplash" which made the system usable.  It now locks up rarely during use but fairly often during logouts, shut downs, and reboots.  No errors show up in the logs.  I'll have to set up an SSH server on it so I can monitor it remotely the next time it hangs assuming it's just X and not the kernel itself.  If it is the kernel then the SSH server will die immediately and I probably won't see any error messages.  In that case the solution would be to use a serial console but the laptop doesn't have a serial port and I don't have a USB serial adapter.  The advantage of a serial console is that the kernel handles it directly and "out of band" so networking issues don't affect it.
Another possibly related graphics issue is that the Gnome display properties utility crashes whenever I attempt to use it (bug #198951).  I use this to correct the screen resolution when a game or Wine application aborts and doesn't reset it back after changing it.  I can do a "xrandr -s 1024x768" in the terminal for now.
Another constant annoyance is the crashing of the Gnome settings daemon at every login (bug #197153).  There was an update that came through and fixed it but two days later another one caused the problem to return.
There is also an issue with the battery monitor always reporting a 51% charge, hal setting the cdrom (/dev/sg0) to group root instead of cdrom and complaining it can't unmount the volume when the drive's eject button is used, and I can't disable the SCIM language control which keeps changing my keyboard language randomly (bug #199314).
Firefox v3 beta works more or less.  I like the new smart bookmark which lists recently used or bookmarked links.  It crashes once in a while but I haven't determined if an addon or plugin is responsible.  I installed the Sun Java plugin but it can't seem to detect it and is not listed in about:plugins.
In spite of these problems the system is usable and 3D in Wine now works.



Re:  Krazy Kubuntu Annoyances
2008-02-14T23:43:00.008-05:00
Carla Schroder (tuxchick) recently posted a Kubuntu critique on LinuxPlanet titled Krazy Kubuntu Annoyances.  I saw this on both Linux Today and LXer.com but I'm replying here as commenting is annoying on both sites because they strip out most html tags.
I've been setting up an advanced home office network to replace my existing haphazard mess.  I haven't been posing much lately because this it taking a lot longer than planned due to changing design, learning the details of systems I only had superficial experience with, and Ubuntu bugs.  Originally I didn't have a server or central authentication and was just using Samba to share access the lazy "map to guest = bad user" in smb.conf way.  I have IPCop on an old box for firewall, DNS, and VPN.  Internet access is via an Ethernet connected cable modem and there's five subnets for my office, household, wireless, public servers (unused), and an isolated one. The isolated network allows me to retrieve user documents from a malware infected system (Windows of course) without it being able to access the rest of my network or the Internet.  This arrangement worked for a while but my primary workstation was running out of disk space and I decided to set up a proper network and server to make it easier to control access and synchronize my data remotely.  I also plan on building a Debian/Ubuntu repository mirror, public wifi with a "usage agreement" access page like many places have, and a public game server for some Linux games like Tremulous.  Nothing really ridiculous but a lot of work as I've found out.
So far about half of my time has been spent diagnosing various problems that often are Ubuntu Gutsy bugs.  Not always server-related but it's often hard to tell where the problem is when you're in the middle of a major restructuring and there are a large number of variables.  About half have been data storage problems and the other networking.
The storage issues are the result of me hitting the limit of Ubuntu's alternate installer.  It's fine for basic setup with RAID or LVM but I had to make my life difficult by mixing PATA and SATA drives with RAID+LVM+LUKS/dm-crypt+pam_mount including encrypted root and swap.  It didn't handle that well.  I could go on for hours about the problems and other bug reports I filed but more on that later.
Networking-wise I encountered many problems including some of the ones Carla mentioned.  Not all were bugs but just design decisions whose basis is hard to understand or track down.  The ones we've encountered are by no means all as there are many bug reports at launchpad and on various forums about networking problems that don't occur with Knoppix or other distros.  I'm using Ubuntu on most systems so some of my problems are Gnome-specific but many are not.
Network Manager seems to be the cause of a lot of complaints.  First, just having the wrong ethernet chip will cause Network Manager to disconnect it for you even if it was working at boot.  Or maybe it will refuse to shut down because of it.  Then there are the general IPv6 issues.  Of course you still have to deal with the normal industry-wide problems like not being able to resolve host names when using a laptop with multicast-DNS (Avahi) active on a Windows network using a something.local domain.
The the link local auto-config emulates Window's behavior but I'm not sure if it's good or bad.  On Windows networks I always used a class C network address and if a PC ended up with a class B then I knew that it wasn't seeing the DHCP server and had auto-configured one.
I've had some issues with the host file not being set up properly which seemed to cause a major slowdown in Gnome.  It took me a while to straighten it out.  The odd 127.0.1.1 entry confused me but apparently it was requested because of compatibility and historical reasons.
For printer configuration I use the CUPS web interface if the graphical utilities don't suffice.
The Bluetooth support seemed to be driven by phone integration usability concerns but it was low priority according to the Gutsy blueprint.  The Hardy blueprint has an entry for networkless installation fixes but I don't see anything regarding my drive setup problems, only post-installation management.



Disabling hibernate and suspend buttons
2008-01-20T17:57:00.000-05:00
On systems with broken hibernate and suspend its a good idea to disable the option in Gnome and KDE to prevent inadvertent usage and potential lock-ups.
In Gnome use the gconf-editor.  On systems using sudo to prevent direct root login launch it by "gksu gconf-editor".  Browse the configuration tree to:
/apps/gnome-power-manager/general
Uncheck "can_hibernate" and "can_suspend".  Right-click them and select "Set as Mandatory".  Users will probably have to relogin before it takes effect.
In KDE, create a file:
/usr/share/kubuntu-default-settings/kde-profile/default/share/config/power-managerrc
In this file add:
disableSuspend=1
disableHibernate=1



Find tricks
2008-01-01T15:49:00.000-05:00
These are some find examples with moderately complicated regular expressions that I've used for administration tasks.  Note that regular expressions used by find, grep, and other programs have some variants with both the old "basic" form and the newer "extended" forms.  Find defaults to the extended version based emacs but some of it's tests like -name use "shell patterns" instead (see the sh man page).  In the regex man page the "(!)" identifies some of the syntax and behaviour that may not be compatible with other regular expression implementations.
The first example cleans out the Unreal Tournament 2004 cache from the home folders of all users.  The purpose of cleaning out the cache is that every time the client connects to a server that is using a map, vehicle, or other add-on that it doesn't already have locally it downloads it to the cache.  The cache on a system of an avid on-line gamer will easily exceed many gigabytes and can run their home directory out of space on smaller drives.  On some distributions, it is impossible to log in if there is no home space available.

find /home -regex '.*/\.ut2004\(/Cache\|/.*/Cache\)' -exec rm -rf {} \;


Breakdown:
. = metacharacter implying any single character

* = any quantity of the previous character (in this case any quantity of any character because of the "." metacharacter)

/ = detect the slash to set a reference to a directory when combined with the next part.

\.ut2004 = the backslash escapes "." so that it is treated as a period and not a metacharacter.  Combined with the previous ".*/" it limits the results to hidden /.ut2004 and not /..ut2004, /xut2004, or any other directory.

\(...\|...\) = This sets up a pair of branches with alternation as indicated by the vertical bar.  The parenthesis define the range of expressions that make up each branch.  The vertical bar is escaped to keep the shell from thinking it's a pipe and the parenthesis are escaped to indicate they are not being searched for.

\(/Cache\|/.*/Cache\) = The combined alteration limits results to .ut2004/Cache and any other items named Cache inside the .ut2004 directory.  The parenthesis are important - without them find will return /.ut2004/Cache and anything anywhere with a subdirectory named Cache (like in .mozilla).

The entire regular expression is protected by single quotes so find knows they belong together.  You could also limit the results to directories by adding the "-type d" test.

-exec rm -rf {} \; = This tells find that for every item it returns it is to execute rm with the parameters -rf (to delete directory trees) followed by the path with which will dynamically replace the {}.  The command line is terminated by an escaped semicolon.

Here are some other find examples I've found useful.

Find any user's Mozilla/Firefox cache:

find /home -regex '.*/\.mozilla/.*/Cache'

Find any user's hidden trash directory:

find /home -regex '.*/home/[^/]*/\.Trash'

When installing updates to applications in Wine you will occasionally encounter duplicate file and directory name problems, sort of a reverse name collision.  It can occur because Linux names are case sensitive but Windows is not.  This means it is possible to have two files, "readme.txt" and README.TXT" in the same Linux directory but not in a Windows one.  If an application update is in the form of a executable or self-extracting archive, Wine will resolve capitalization differences and ensure that a replacement file from an update that has a upper-case name will correctly overwrite a target file with a lower-case name.  But if the update is from a zip or other archive and contains names with different case, then you just can't extract them and copy them on top of the installed application's directories as duplicates will result.  In order to work around this you either have to install a Windows archive utility like 7-Zip and use it to extract the files and take advantage of Wine's name resolution function, or manually change the names to match.  Since Windows applications generally don't care about file or directory name case, another option is to rename everything to lower case.  You can do this by combining the find command with the rename command:

find <directory or file name> -depth -execdir rename 'y/A-Z/a-z/' {} \;

By default, find has some optimizations that will speed up searches in large directory trees.  One of these is to skip checking subdirectories by assuming that two less exist than the total becasue of the "." and ".." entries.  This will cause find to miss directories on file systems that do not have hard links for these like CD-ROM and vfat.  To prevent this from occurring, use the -noleaf option.



Resetting a terminal
2007-12-26T23:10:00.000-05:00
Sometimes while troubleshooting a system I end up with scrambled terminal settings resulting in odd colours or graphics characters instead of text.  This usually happens when I accidentally send binary data to the stdout and the terminal emulator (getty in Ubuntu) detects an escape code and switches modes.  To fix it, enter "setterm -reset".  This sends a reset character string through the terminal which usually sets it back to the default settings.


Root ending up in the wrong home with sudo
2007-12-24T17:30:00.002-05:00
On Ubuntu and some other distributions direct login to root is disabled and sudo has to be used.  The idea is to discourage running as root as this is bad for security (just like running as administrator in Windows).  It is possible to get to a root shell but there is a right way and wrong way to do it.  The wrong way doesn't change the environment variables like $HOME which will cause programs like Midnight Commander to save their config files to the originating user's home directory but with root ownership.
This is the WRONG way which leaves $HOME still set to the originating user's home:
sudo -s (or) sudo /bin/sh
This is the correct way which sets $HOME to /root:
sudo su (or) sudo su - (or) sudo -i
To experiment, enter the above commands followed by "echo $HOME".  Enter "exit" to return to the original shell.



Compiz on multi-head setup = multi-problems
2007-12-23T15:38:00.000-05:00
I rebuilt up my primary desktop system with Gutsy and tried out Compiz with my non-Xinerama multi-head setup.  It didn't handle it well and after a few hours of trying to fix it I switched back to plain-old Gnome.  I reported my findings as bug #178341.  There were many bug reports about most of the problems I experienced but it seemed like there was some fundamental problem that was causing them.
First, no window borders (ie, decorations) were present on windows in the second head.  I had to reload the decorator with "gtk-window-decorator --replace" to get them to show up.  Windows can be manipulated using mouse + keyboard but it's annoying.
Second, the Quit button on the second head would show an invisible power/logout window.  I could cancel it with Esc or log out with Alt-L.  Other people who have encountered this one mistakenly thought it was a desktop freeze and killed X.
Third, the long default menu pop-up delay was really irritating but it only was present on the primary head, not the secondary.  Setting gtk-menu-popup-delay=0 in ~/.gtkrc.mine didn't have any effect on either one.  The excessive delay makes the system seem slow like Vista.  Regardless, it should be the same on both heads and this leads me to believe that the config files are not being read properly on the second head.
Fourth, the workspace switcher applet didn't function correctly.  If I changed the number of workspaces on the second head it would affect only the primary head.  It didn't get along with Compiz either but it's a known limitation.  Minor issue but further proof of something being broken within.
Other than some Nautilus errors in .xsession-errors there was nothing useful in the logs.
I may have to wait until Hardy for Compiz to be usable on my system.  It wasn't an easy decision for the Ubuntu dev team to include Compiz because of the known issues with drivers and stability but it will mean more stability for the Hardy LTS (5-year support) version.  The problems I encountered are hopefully specific to multi-head setups as they are not typical for most systems; it would create an unfavourable impression on new users.



Nautilus bookmarks and Places problems
2007-12-12T23:32:00.000-05:00
I rely rather heavily on the bookmarks/places sidebar in Nautilus (Gnome's file manager).  I have a very large archive of freeware, evaluation software, reference material, and documents files sorted by OS and application type, category, or customer project number.  It takes a lot of clicking to drill down into and the bookmarks allow me to jump quickly to the directories I use the most.  I also have bookmarks to SMB shares on various subnets and across a VPN.  Its worked rather well but I noticed that I couldn't sort the list the way I wanted.  Normally you can just click-and-drag a bookmark and drop it where you want.  But when I tried it didn't work correctly and ignored the action or replaced it with a different bookmark like it lost track of which one I was moving.  I decided to resolve the problem by deleting several bookmarks I thought it could be having trouble with like the SMB shares.  But it still didn't work.  After some searching I figured out that the bookmarks are stored in the hidden text file ~/.gtk-bookmarks which is easy to edit manually.  I found several bookmarks that Nautilus was not displaying which pointed to old targets that no longer existed.  I deleted them and killed and restarted Nautilus and it seems happy now.  It might be a design flaw or a bug.  I'll have to investigate more after I upgrade to Gutsy.


Keyboard LED flashing panic
2007-12-10T18:53:00.000-05:00
I was in the middle of replying to an email from my home-made desktop PC when it suddenly stopped responding to anything.  The only activity was the Caps Lock and Scroll Lock LEDs flashing at about a 1 second (1Hz) interval.  At first I thought a PS/2 connector had come loose on my D-Link DKVM-4 KVM or on the system but the USB mouse was also not responsive.  After some Google searching with my laptop I figured out that the kernel had probably panicked and the LED activity was caused by the panic_blink function which is basically an idiot light.  According to what I read the flashing can also involve the Num Lock LED but mine may have been off because the Num Lock was active at the time of the panic.  While there is no information conveyed by any of them it does let you know that something serious happened, not just a loose connector or X hanging.  There was a kernel patch submitted by Andrew Rodland back in 2002 to blink in Morse code but I don't think it was ever included in the main tree.  In my experience, once a Linux system has been stable for a while a panic usually indicates a hardware problem.  After a few hours of troubleshooting I determined that my 1GB PC-3200 DIMM had failed, probably because it wasn't seated in the socket properly.  I'm surprised it lasted as long as it did.


Tedious Engineering
2007-12-03T11:29:00.000-05:00
Spent last week doing more design work on a company's latest project for one of their automotive customers.  Mostly working with AutoSketch 7 for creating basic 2D machining drawings for enclosures, Adobe Illustrator 10 for creating overlay labels for the enclosures, and Altium Designer 2004 (aka DXP 2004) for PCB design.  I don't really like any of them but it's what the company uses.AS7 is primitive but adequate for simple 2D drawings.  It is useless for graphical design of labels because it's font handling is inadequate (although v9 may be better) and label manufacturers can't import it's SKF files.  I usually export a DXF and import it into IA10 but its tricky and there are usually odd round-off errors.  I suspect AS7's DXF format is the problem because it will often export DXFs it can't later import.  Qcad can't read it at all, SagCAD shows some of the objects but endpoints of many lines are set to the sheet origin, but the VariCAD viewer shows most of it.  I would consider buying the full version of VariCAD but I would have to be able to convert drawings to something AS7 likes which is unlikely.  AS7 can be installed in Wine but there is a problem with the format of numbers in dimensions which may be related to bug #4348.  I'll investigate more later.  In Vista it functions but is unstable and often crashes during panning or messing with printer settings.
IA10 is annoying to use if you are from a drafting background.  I'm used to drawing everything on a grid with precise absolute or relative locations of everything and IA10 is difficult to use that way.  But the label graphics and cutouts need to match the enclosure cutouts so precision is required.  It functions in Vista but I haven't tried it in Wine yet.  The AppDB reports indicate mediocre results.
Altium DXP looks a lot nicer in a brochure than reality.  It is a perfect example of a product from an acquisition-focused developer in that lots of new broken toys are merged in with every release.  It works as "exceptionally" on Vista as it did with XP.  It still throws exceptions all over the place but they only occur half as often on Vista.  This is not from Vista's improvements but because Vista is half as fast as XP and it takes twice as long to encounter one so effectively the error density has gone down (along with productivity).  The new feature Vista provides is that doing anything with the output job BOM report generator (and reportedly print previews) will kill it every time.  I have to run it on XP in VMware.  I've been meaning to try it on Wine but I need to free up some space and want upgrade to Gutsy first.  According to the AppDB reports version 6 works partially.  The company will probably upgrade to the new version at some point in order to take advantage of new features and bugs.
While this part-time contract work pays the bills I find it generally boring.  Having to do network administration on Windows systems doesn't improve it any.  At least it's a small company that doesn't require centralized e-mail and document sharing so I can continue to ignore the SharePoint and Exchange services.  There are a few benefits in that I occasionally get obsolete office computer junk to play with and I have learned about a few useful tools.  One in particular is Perforce, a cross-platform version control system.  I don't know what benefits it brings to management of software development as compared to other solutions but it is rather useful for keeping track of my design-related files.



My Linux-Aspected Black Friday Assault
2007-11-24T00:56:00.002-05:00
I was thinking of ignoring the Black Friday sales this year as it's annoying to have to get up really early, stand in line freezing for hours, fight my way into the store along with a few hundred other people, get 1/10th of what I wanted, then spend another hour in the check-out line.  But then I spent Wednesday night stuck at my office because a large snowstorm had blocked all the roads and they weren't be plowed out until the following afternoon.  So to pass the time I started checking out the leaked BF advertisements to see what was available and I decided to try my luck at the Staples store in Alpena.  Although they are known as an office supply store they have a lot of computer tech on sale, especially on Black Friday.  I was interested mostly in Linux-compatible cameras and printers which meant skipping the Canon and newer Epson models.  They carry Lexmark but none were on sale, not that it mattered as I wouldn't take one even if they were free.  The best deals are offered during the "Early Bird" sale which was held 6:00 to 10:00 although other good deals were available throughout the weekend.
The next day, after Thanksgiving dinner, I head for bed early.  I wake up at 4:00 and it's probably around 15°F (-9°C) outside.  I put on long underwear, snow pants, a sweater, two coats, heavy boots, and a stocking cap then get in the car.  I live out in the country and normally the roads are vacant this early but this time there were several cars in front of me, all heading into the town.  I get to the store at 5:15 and there are many cars in the parking lot and about 50 people in line already.  I suspect that many of them are holding places for others sitting in vehicles keeping warm so I expected the number to double when the store opened at 6:00.  The manager came out some time later and handed out sales flyers and a store map.  I found out they were not using item reservation tickets like some stores do so that meant that it was going to be a free-for-all assault.  At 5:55 the crowd increase wasn't that much and some store staff blocked some line-cutters and sent them down to the end.  I'm not sure where the end was at actually - the line was way around the corner.  I'm guessing 200 or so people based on crowd later in the store.  I check my list and map and wait impatiently - I can feel the excitement in the air.  But when the doors finally opened it was more of an orderly surge instead of a stampede.
I took a chance and grabbed a cart.  Grabbing a cart is not a small decision when it comes to BF shopping.  It costs time because it is difficult to maneuver in a crowded store which potentially means losing out on popular items with limited inventory.  But printers are too big to carry and I wasn't sure what else I would grab.  I headed over to the likely area on the simple map and found it was a little inaccurate.  The items were not on the shelf but on tables in the aisle towards the rear of the section which now jammed full of shoppers.  I was worried that the minute or two of disorientation would cost me the HP camera I was after but I managed to get one.  Conveniently they had bundled it with a 1GD SD card and 4x6in HP photo paper.  This was about 6:03 and items were going fast so I then started grabbing anything even remotely desirable.  My normal BF tactics are based on credit card power.  I buy everything I can get because comparing items takes too much time.  I then take them home for evaluation, offer rejects to friends and family, and return the leftovers.  This has worked well in the past and it worked very well this time.  I think my success was helped by three factors - the cold limited the competition, there was a large
amount of inventory on hand, and I wasn't after the high-demand items.  About a third of the shoppers were lined up for some laptops and GPS units on sale towards the rear of the store.  I didn't need a laptop and decided to pass on the TomTom ONE 3rd Edition GPS and concentrated on the printers and other targets of opportunity.  After I obtained everything else I could think of I did wander back to the now vacant rear section to see what was left.  The TomTom was sold out so I grabbed one of the last Navigon 2100T.  I could barely keep stuff from falling off the cart when I got in the check-out line at 6:15.  I finally got out of the store around 7:00, loaded my car and headed to my office for assemsent of the haul.
First, I double-checked everything on the receipts, rebate forms, and my shopping list.  A last-minute item I grabbed for the camera was a 2GB CF card that someone had discarded on a shelf next to where I was waiting in line.  But the camera need SD so I returned it for a 2GB SD card for the same price.  I decided a 2x1GB PC2-5300 CL5 SODIMM kit for a laptop wasn't much of a bargain at $47.98 and I could get it later (and with lower latency) when I actually had a laptop that could use it.  The Navigon didn't make the cut either.  Reviews gave it only passing marks against the TomTom and other models and it didn't have a USB connection.  At $99.99 it was cheaper than the TomTom ($124.99) but I didn't have a strong need for it anyways.  It didn't help that it used Windows CE 5 while the TomTom used Linux and had a SDK available.  I did go back later and pick up a SanDisk “Ultra” SD card to see if the camera would save faster than with the economy card but it didn't seem to improve much.  The camera doesn't have an optical viewfinder but I got used to it quickly.  Interestingly, the camera case has a magnet to hold the flap closed instead of Velcro.
Here's the score after the initial returns:

 
  
   Brand
   Model
   Description
   Base $
   Tax $
   Rebate $
   Sub-total
   Normal $
   Linux?

   Brother
   MFC-440CN
   Inkjet Printer AiO with Ethernet
   $79.98
   $4.80
   $20.00
   $64.78
   $135-145
   Mostly
  

   Brother
   PT-1010
   Portable labeler
   $19.99
   $1.20
   $10.00
   $11.19
   $43-45
   N/A
  

   Executive
   EPS-1200X
   12-Sheet crosscut shredder
   $49.99
   $3.00
   $30.00
   $22.99
   $80
   N/A
  

   HP
   C5180
   Inkjet Printer AiO with Ethernet
   $149.98
   $9.00
   $50.00
   $108.98
   $135-145
   Perfectly
  

   HP
   M737
   8MP digital camera
   $129.98
   $7.80
   $30.00
   $107.78
   $147-162
   Yes, PTP
  

   HP
   Q7906A
   100sht 4x6in glossy photo paper
   $12.99
   $0.78
   

   $13.77
   $11-17
   N/A
  

   Logitech
   QuickCam Orbit MP
   1.3MP Webcam with pan/tilt
   $104.99
   $6.30
   $85.00
   $26.29
   $78-100
   Yes
  

   PNY
   P-SD1G-RF3
   1GB SD Flash
   $14.98
   $0.90
   

   $15.88
   $14-17
   Yes
  

   Samsung
   CLP-300
   Color laser printer
   $249.98
   $15.00
   $150.00
   $114.98
   $214-246
   Mostly
  

   Samsung
   ML-2510
   B&W laser printer (w/$20 gift card)
   $119.99
   $7.20
   $70.00
   $57.19
   $101-114
   Perfectly
  

   SanDisk
   SDCZ6-4096-A10RB
   Cruzer Micro 4GB USB Flash drive
   $27.98
   $1.68
   $10.00
   $19.66
   $55-70
   Yes
  

   SanDisk
   SDSDB-2048-A11
   2GB SD Card
   $14.98
   $0.90
   

   $15.88
   $20-35
   Yes
  

   SanDisk
   SDSDH-2048-901
   2GB SD Card
   $17.98
   $1.08
   

   $19.06
   $47-53
   Yes
  

   SanDisk
   SDSDQ-2048-A11MK
   2GB microSD kit w/adapters
   $17.98
   $1.08
   

   $19.06
   $35-60
   Yes
  

   Staples
   554638
   500 sheets 8.5x11in 24lb paper
   $4.98
   $0.30
   

   $5.28
   $7.49
   N/A
  

   Staples
   648177
   60sht 4x6in glossy photo paper
   $6.98
   $0.42
   $6.98
   $0.42
   $6.98
   N/A
  

   Staples
   674535
   600 sheets 8.5x11in 20lb paper
   $4.99
   $0.30
   

   $5.29
   $5.99
   N/A
  

   Western Digital
   WD4000JSRTL
   400GB SATA 300, 7200RPM
   $89.98
   $5.40
   $30.00
   $65.38
   $130
   Yes
  

   

   

   Total:
   $1,118.70
   $67.12
   $491.98
   $693.84
   

   


It will be a while before I get through testing all this.  Then of course there are all the rebates to submit.  Thankfully Staples has on-line submission for most of their rebates.



FontPage on Wine
2007-11-21T22:34:00.000-05:00
I was trying to get capella reader to operate on Wine but it couldn't find it's custom music notation fonts.  Definitely a compatibility bug there but I also encountered a problem with Wine 0.9.49 not finding fonts in the windows/fonts directory.  To get them to show up at all I had to stick them in ~/.fonts and relogin.  To verify that they were registered correctly by Wine I used an old Windows freeware program, FontPage.  It worked perfectly.  The only issues were that it required the Visual Basic 6 runtime and threed32.ocx wouldn't load.  The VB6 runtime will not install but if you run it with "vbrun60sp6.exe /C" it will extract the dlls and you can copy them to the windows/system32 folder (don't overwrite anything).  The threed32.ocx problem I solved by looking at the Wine messages when I tried to start FontPage from a terminal.  It needs mfc40.dll to be installed in windows/system32.


SMB browsing in Xubuntu/Thunar
2007-11-21T00:24:00.000-05:00
The biggest usability problem in Xubuntu is that the file manager, Thunar, doesn't have integrated SMB browsing for Windows networks.  This solution works but doesn't scale well beyond a single user because of the network directory location and permissions.  I came up with a configuration that works better by locating it in each user's home directory instead.  Optionally it can be auto-configured for new users when they are added.  First you need to create a script to mount the network directory with fusesmb for the current user:


#! /bin/sh

fusesmb $HOME/network




Save to /usr/local/bin/auto-fusesmb, owned by root, and set execute permissions for everyone.  Then create a xdg menu file to launch it:



[Desktop Entry]

Encoding=UTF-8

Version=0.9.4

Type=Application

Name=fusesmb

Comment=SMB network browsing

Exec=auto-fusesmb

StartupNotify=false

Terminal=false

Hidden=false




Save this to ~/.config/autostart/fusesmb.desktop  so it will lauch the script at login.  This will show up in the Applications > Settings > Autostarted Applications dialog.   Obviously you can create this using the dialog but now you know where it is stored.   Then create the directory "network" in the user's home folder.  Only the user needs any access permissions to it.



To create this setup automatically for newly added users, copy the .config/autostart/fusesmb.desktop file and network directory to /etc/skel and set the permissions the same as for the existing files (.profile, etc.) but change the owner and group for everything to root.  When a new user is added, this skeleton directory will be the initial default so the ~/network directory will exist and the fusesmb.desktop file will cause auto-fusesmb to mount it whenever the user logs in.


blog start
2007-11-21T00:19:00.000-05:00
It's created.  Now I have to remember to post regularly without ranting.

Brand	Model	Description	Base $	Tax $	Rebate $	Sub-total	Normal $	Linux?
Brother	MFC-440CN	Inkjet Printer AiO with Ethernet	$79.98	$4.80	$20.00	$64.78	$135-145	Mostly
Brother	PT-1010	Portable labeler	$19.99	$1.20	$10.00	$11.19	$43-45	N/A
Executive	EPS-1200X	12-Sheet crosscut shredder	$49.99	$3.00	$30.00	$22.99	$80	N/A
HP	C5180	Inkjet Printer AiO with Ethernet	$149.98	$9.00	$50.00	$108.98	$135-145	Perfectly
HP	M737	8MP digital camera	$129.98	$7.80	$30.00	$107.78	$147-162	Yes, PTP
HP	Q7906A	100sht 4x6in glossy photo paper	$12.99	$0.78		$13.77	$11-17	N/A
Logitech	QuickCam Orbit MP	1.3MP Webcam with pan/tilt	$104.99	$6.30	$85.00	$26.29	$78-100	Yes
PNY	P-SD1G-RF3	1GB SD Flash	$14.98	$0.90		$15.88	$14-17	Yes
Samsung	CLP-300	Color laser printer	$249.98	$15.00	$150.00	$114.98	$214-246	Mostly
Samsung	ML-2510	B&W laser printer (w/$20 gift card)	$119.99	$7.20	$70.00	$57.19	$101-114	Perfectly
SanDisk	SDCZ6-4096-A10RB	Cruzer Micro 4GB USB Flash drive	$27.98	$1.68	$10.00	$19.66	$55-70	Yes
SanDisk	SDSDB-2048-A11	2GB SD Card	$14.98	$0.90		$15.88	$20-35	Yes
SanDisk	SDSDH-2048-901	2GB SD Card	$17.98	$1.08		$19.06	$47-53	Yes
SanDisk	SDSDQ-2048-A11MK	2GB microSD kit w/adapters	$17.98	$1.08		$19.06	$35-60	Yes
Staples	554638	500 sheets 8.5x11in 24lb paper	$4.98	$0.30		$5.28	$7.49	N/A
Staples	648177	60sht 4x6in glossy photo paper	$6.98	$0.42	$6.98	$0.42	$6.98	N/A
Staples	674535	600 sheets 8.5x11in 20lb paper	$4.99	$0.30		$5.29	$5.99	N/A
Western Digital	WD4000JSRTL	400GB SATA 300, 7200RPM	$89.98	$5.40	$30.00	$65.38	$130	Yes
		Total:	$1,118.70	$67.12	$491.98	$693.84