Two more utilites for DansGuardian Users

I reduced the DansGuardian user account and blacklist maintenance hassles with my previous two utilities but while working on whitelisting I found the need for a few more.

In DansGuardian (DG) terms a blacklist bans something, a greylist allows something (overrides blacklisting) but still filters it, and an exceptionlist allows something without filtering (overriding greylists and blacklists). The "something" can be URLs, IP addresses, server names, etc., depending upon the specific list type. Blacklisting a site is easy but blacklisting a specific type of content is very difficult and error-prone. It works the same way as anti-malware utility definitions - if the undesirable items are on the list, and they match a particular requested target, then it's blocked. If not, then it gets through. It's a big Internet out there and trying to block all the bad is rather difficult. "Bad" is also relative and what is bad for one person/religious group/company/government may not be bad for another. Whitelisting has the opposite problems in that you gain strict control over what is available but trying to predict where the user wants to go, determining if that is a safe destination, and maintaining the lists is also difficult.

I found that I needed to use both blacklisting and whitelisting. I use whitelisting with younger children and blacklisting for older. Older children won't put up with strict constraints and will either figure out how to bypass them or simply go somewhere else to browse the Internet. Younger children are easier to keep happy but you still have to spend time figuring out all the web sites they will want access to, preferably with the initial configuration so they're not whining every five minutes about another toy/game/whatever site they can't access.

With DG a "whitelist" configuration is basically a blacklisting of all sites with a "**" in the bannedsitelist file with entries in greylist and exceptionlist files to bypass it. The exceptionlist file entries will enable site access but this is not what you want for allowing a user to browse a particular site because it disables all filtering. Use greylist files instead. This way if there is an offensive part of a site that you didn't know about (or it gets defaced by black hats) then you still have the filters to rely on. The exception lists are useful for sites that are not normally browsed but may trigger the filters inadvertently such as Linux distro repositories using http.

One of the problems with whitelisting is that the user won't necessarily know where they can go on the Internet. To solve this problem you need an index page of some sort. This is the problem I encountered when creating my greylists and I came up with a solution.

I didn't want to maintain an index separately from the greylists so I figured out a way to embed the data in the lists. DG recognizes a # in the list as a comment. I added a comment at the end of each list entry with a Wiki-style link after it. This isn't all that unusual as Debian/Ubuntu did something similar with the menu.lst file in Grub. The comment hides data that isn't relevant to DG but the defined format allows extraction of the data to create an index. Soon after I started adding the links to the list entries I figured out two things - it was a lot of typing and was going to be a very big index. To organize the index better I added a category tag on the end which could be used in the index. The final format is:

<exception link> #[<URL><space><label>][Category:<category text>]

The brackets are required characters. The parsing is somewhat whitespace tolerant but in the Category tag don't leave any spaces between the colon and the category text (sed and regex expressions can be tedious). Example:

gutenberg.org # [http://www.gutenberg.org Project Gutenberg][Category:Books]

To save some typing I wrote add-exceptionlist-url-comments which creates a default URL comment. First it pads the end with tabs (up to 5) to keep it pretty. The default link is made by slapping an http protocol prefix on the exception entry. It then uses wget to try to fetch the default web page and scrape the page title to use as a default link label. This works for most pages and redirects but not those that are using a meta refresh. It finally adds an undefined category tag at the end. Anything in the list that starts with a # is ignored. Note that not every entry will need a link. Some sites you don't want may serve data to a site you want. A lot of USA government sites that are kid-specific will link to media on the main government sites which aren't of interest to kids and just clutter the index. Some web stores also use third-party search services which will need exceptions but not links. In many cases you'll want a link that points to a specific part of the site, not just the server root, so you'll have to edit the defaults.

To create the index page I wrote exceptions-index-page-generator. It looks for the bracket-formatted URLs in the input files. It also builds a list of category tags, assigning a default tag (defined in the script) to any that are missing. It then creates a basic html file with entries separated by category. If a category has more than a certain number of entries (default 5 as defined in the script) it makes two columns to reduce the page length. It doesn't try to normalize category names so they must match in the entries exactly in order to be combined. It ain't pretty but it works. These are both command-line utilities but are rather easy to use.

UPDATE: I updated exceptions-index-page-generator. Version 1.1 adds a category table of contents to the top of the page. It will also make two columns of these if the number of categories exceeds the column threshold.

You can use my greylists to test with and as a base for your own lists for younger children. I haven't performed in-depth checking of these but they look relatively safe. Some of the entries may seem odd but they're intended to aid holiday gift buying. You will also notice that I used html entity codes in the labels for some punctuation as they didn't display correctly in Firefox.

A pair of utilities for DansGuardian users

Content filtering is a requirement of the home desktop system configuration build I'm working on. Young children are part of the client base so it's mandatory. DansGuardian is basically the only free option available. It's a server daemon so it has command-line configuration only. Once it's running parents don't need to mess with the basic settings but they need to be able to set filtering controls for children without a lot of hassle. On Ubuntu it doesn't come with any blacklists but third-party lists are available. Shalla Secure Services has one of the most comprehensive list that's free for home use but installing and updating it is also a hassle. I wrote a pair of scripts to solve both of these problems.

There are a few options for DansGuardian GUI. Some firewalls like SmoothWall have plug-ins for it. Two popular stand-alone ones are DansGuardian-GUI from Ubuntu CE and WebStrict from Saliby. Unfortunately they both rely on Tinyproxy which has a bug with DansGuardian that prevents many pages from loading. They also drag in FireHOL which I don't need.

Since remote administration is a requirement for my desktop configuration I installed Webmin. A plug-in is available, DansGuardian Webmin Module, which allows easier control than straight command-line methods including a semi-automatic configuration for multiple filter groups. There's one bug with the latter that I had to fix first and the default DansGuardian binary location in the module's configuration was incorrect for Ubuntu (it's at /usr/sbin/dansguardian) but that's all.

When working with multiple filter groups the goal is to have DansGuardian automatically apply the correct filter based on the user account. Correlating user port activity to filter groups is tricky. Since my targeted desktop systems are stand-alone and won't have multiple simultaneous users I chose the Ident method using Ident2. I tried Bisqwit's identd (bidentd) but the version on Ubuntu 10.04 (Lucid Lynx) has a nasty looping bug that is triggered by local queries. Getting this to work only requires activating the ident authplugin and creating the filter groups.

While the module makes configuration easier for the admin, it's still not that friendly for a parent. The filter groups make it easy to set user restrictions based on group membership but DansGuardian filter groups are completely separate from system groups. They can only be changed from the command line or with the Webmin module. I wanted parents to be able to use the standard desktop user administration tool, users-admin (System > Administration > Users and Groups) to assign users to special DansGuardian groups that could then be converted to filter group memberships. There once was a patch for DansGuardian that integrated the two but it's not included upstream. So I came up with a system group naming scheme and wrote dg-filter-group-updater, a GUI tool that automatically creates the filter group list (/etc/dansguardian/lists/filtergroupslist by default) from the system group membership. Installing it is easy. Just copy the script to "/usr/local/sbin" with root ownership and 755 (rwxr-xr-x) permissions. Download this desktop file and put it in "/usr/local/share/applications" with root ownership and 644 (rw-r--r--) permissions which will cause a menu entry to appear in the System > Administration menu. This is for Gnome as it uses gksudo to get root access by you can convert it for KDE by changing the "gksudo" to "kdesudo" or "kdesu" then changing the "Categories" entry for KDE (look at other KDE desktop menu files in /usr/share/applications). For this script to be useful you have to set up the required system groups first and assign users.

DansGuardian references group filters by an index number. The first group is "filter1" which corresponds to the configuration file "dansguardianf1.conf" and is the default. Typically in a multi-group configuration this filter is set to disable Internet access with a "groupmode = 0" setting. By "Internet" I mean "http" as DansGuardian can't really help with "https" (TLS/SSL) or much else. The rest you have to block with firewall rules or a filtered DNS like OpenDNS. The module's multiple group tool is the one named "Set Up Lists&Configs For Multiple Filter Groups" on its main page. Before using it, backup the "/etc/dansguardian" directory as this option only works once and then locks itself out. Restoring the directory is the only way to revert. When you use this tool you will have a few options to chose from. The scheme is up to you (I used separate). I recommend selecting "Use of Default Group" and "To Set Aside Unrestricted Group". I used four groups:

#1 "No_Web_Access" default (filter1, groupmode = 0)
#2 "restricted" (filter2, whitelisted with groupmode = 1 in its conf file and ** in its bannedsitelist file)
#3 "filtered" (filter3, filtered with groupmode = 1 and nautynesslimit = 100)
#4 "unlimited" (filter4, groupmode = 2)

The idea here is that unassigned accounts are automatically blocked by filter1, young children are sandboxed with filter2, older children are filtered with filter3, and adults unrestricted through filter4. Since the restrictions are more about maturity than age the groups don't have names that refer to the latter.

The dg-filter-group-updater script requires system group names to have a specific format of "dansguardian-f#..." where # is the corresponding filter number. Anything after the digits are ignored so you can create more descriptive group names that a non-technical user can recognize in the users-admin tool when assigning members. These groups should be created as system groups (GID < 1000). I created my groups with addgroup:

addgroup --system dansguardian-f2-restricted
addgroup --system dansguardian-f3-filtered
addgroup --system dansguardian-f4-unlimited

Obviously you need to have a "sudo" before these or get a root terminal with "sudo su" first. Since filter1 is the default you won't be assigning users to it and don't need a matching system group. Next you just need to assign users to each group. If you assign the same user to more than one, DansGuardian will use the lower numbered filter in the resulting filter group list. Afterwards just launch the script via the menu item "DansGuardian filter group updater" and enter your admin password. First it will read through the dansguardian.conf file. The file location is set by the "dg_conf" variable in the script and is the only hard-coded value you need to worry about. From the conf file it locates the filter group list file and the number of filter groups. It then starts a new filter list group file (overwriting any existing one). Next it reads through /etc/groups and looks for the "dansguardian-f#..." groups, extracts the users for each, and adds them to the filter group list file in "username = filter#" format. It then restarts DansGuardian. So all a parent needs to do is assign users to groups with users-admin and then launch the script from the menu item to apply the changes.

The script is based on the same code I used for webcam-server-dialog so it will work with any dialogging program installed. Other than that it only uses basic text manipulation tools including grep, sed, and cut. If it doesn't start then launch it from a terminal window or do a "tail ~/.xsession-errors" to see any messages it put out (including those from DansGuardian when it restarts). Most error messages are displayed in a dialog box.

While dg-filter-group-updater solves the basic user administration problem, the lists for filtering (filter3 in my example) still need to be configured. The Ubuntu package only includes basic advertisement-blocking blacklists. Adding third-party blacklists is complicated as you have to merge them in with "Include" statements in the main lists. The lists are organized by categories so you can pick and choose what to filter. Annoying but you only have to do it once if you're using simple filter groups like mine. The problem with blacklists is that they have to be updated often. Shalla Secure Services has some update scripts but they didn't impress me much or did what I wanted. My policy with third-party anything (clipart, CAD libraries, templates) is to keep them separate as references and use other files for customization. To that end I wrote shalla-bl-update. It downloads the list and creates a MD5 file to track the installed version. When it is executed again it checks the MD5 published on the web site against the installed version and downloads the list again if it differs. It has some fault tolerance included as it will retry if the file fails to download or the downloaded file fails a MD5 check. The lists are located in "/etc/dansguardian/lists/shalla" by default. Just download the script from the link and put it in "/usr/local/sbin" with root ownership and 755 (rwxr-xr-x) permissions. It's designed to be started by cron. To have it run daily do "ln -s /usr/local/sbin/shalla-bl-update /etc/cron.daily/shalla-bl-update". It produces no output as cron will Email root whenever anything it runs does. It has a debug mode you can enable by editing the script if you want it to fill your mailbox. It will restart DansGuardian after a successful list update.

Update: I updated shalla-bl-update to v1.2 which adds an optional check for empty system groups. The idea here is that if there are specified system groups used by dg-filter-group-updater, and these groups use the Shalla lists, then these groups should have members. If they don't then there is no point trying to update the Shalla lists. You need to edit the script and set the system_groups variable to the names of the system groups used by dg-filter-group-updater. The grep expression it is using will find partial matches. You can specify "--force-update" to override the check with empty groups.

Update: I've released v1.3 of shalla-bl-update and the link has been updated. Changes: --force-udpate now sets debug=true and clears existing md5. Retries can now be aborted interactively in debug mode. Because of this, the script now uses bash because of the reqirement of the timeout capability of the "read" command. Added "test" parameter for use with Ubuntu Recovery Mode. DansGuardian is not restarted if RUNLEVEL=S (single mode, essentially Recovery Mode). Added --help parameter.

Note: There is a patch by Philip Allison that integrates DG with system groups but the lead developer, Andreas Büsching, has been too busy to integrate it or keep up with maintenance.

UFW application profiles

Uncomplicated Firewall (ufw) is a front-end to iptables. One of its features are "application profiles" which are INI-style files that contain profile names and ufw settings. This allows packages to include their own firewall settings and make them available to ufw when installed.

Using profiles is relatively easy. To see what profiles are on your system, go to a terminal and enter "ufw app list" to see the names. The profiles are located in the directory "/etc/ufw/applications.d" and the names referenced are the "[section names]" in the files. Note that ufw also references the services list in "/etc/services" for rules. If a section name conflicts with an entry in the services file then the latter takes priority (and ufw warns you every time you use it).

There doesn't seem to be any documentation on the file format and the example files mentioned in the docs don't exist on my Karmic or Lucid systems but the existing files for OpenSSH server and Apache are good examples to determine it from:

[section name] (The identifier that ufw references)
title= (shown in "ufw status")
description= (doesn't seem to be used anywhere)
ports= (the port list)

This is the profile for OpenSSH server:

[OpenSSH]
title=Secure shell server, an rshd replacement
description=OpenSSH is a free implementation of the Secure Shell protocol.
ports=22/tcp

Multiple protocols are specified as "80/udp|80/tcp" with a vertical bar separating them. If just "80" was specified then both udp and tcp are assumed. The port can be a comma delimited list (80,443) or a range with a colon (81:82) or combined (80,443,81:82/udp|8080/tpc). If a range is specified then a separate entry for each protocol is required (81:82/udp|81:82/tcp).

I've been working on a Ubuntu 10.04 deployment configuration for my clients and one of my requirements is a user-friendly firewall for mobile users. While ufw is a command-line application GUIs do exist. Gufw is rather basic and doesn't support application profiles. My clients don't know much about network protocols but they can pick an application by name out from a list. It does list some applications but they seem to be hard-coded. Another GUI is ufw-frontends (ufw-gtk) which does support them. My only complaint with it is that when a profile is used there isn't any way to see what ports it affects - all you see is the profile name. In many cases the title and description are more informative than the profile/section name so I hope the tool shows them in future revisions.

With my deployment configuration selecting the firewall GUI was the easy problem. The hard one was the profiles themselves. Application profiles are easy to make but few packages include them. Many of my clients are gamers and most of the best games have online multi-player capability. This isn't just a Linux problem as all of them want to play games on Wine also. Most of these games are client/server and they need ports unblocked when hosting a private server. The profiles are easy to write but finding out which ports need to be forwarded can be very frustrating. Many gamer-oriented web sites provide aggregated ports lists but most of these are unverified and usually specify way more ports and protocols than are necessary. Developer sites either don't list them or list them without specifying the protocols (TCP/UDP or both) or traffic direction. With home users generally you only care about incoming connections to the server - not outgoing. Since most of the ports used are unofficial and not controlled by IANA many games have port collisions with other games, often because they are based on the same engine (like those from id Software and Epic Games) or the use the same API (DirectX/DirectPlay and GameSpy Arcade). It's very rare that you find an list as accurate or concise as that of Novalogic. Several open-source applications only document their ports the old-fashioned way - in the source code. With some I had to install them and use "netstat -nap" to figure out what was used (which sometimes conflicted with the documentation). Another complication is that several games, like Quake 3, require a different port to be opened for every simultaneous client.

I couldn't really avoid the task so I spent several days writing profiles. You can download them all from here. These are intended to be used as incoming exceptions to a "deny all" rule. Just extract and copy them to "/etc/ufw/applications.d" with root ownership and rw-r-r (644) permissions. Start ufw-frontends and click the "Add Rule" button. In the Destination/Port section select the Application radio button and choose the profile from the list. For applications like Quake 3 that have many possible port configurations I created a few different ones which should cover most situations. Unfortunately the ambiguous profile names in some files are going to be confusing. On a few I tried to make them more readable but fixing ufw-frontends so that it shows the title would be a better solution. Unavoidably there are several duplicates and overlaps with other applications which shouldn't harm anything unless the conflicting servers are both operating at the same time. Many servers can be configured with alternate ports but my profiles only specify the common defaults.

Both ufw and ufw-frontends have limitations that I hope will be addressed in the future. Support for port triggers, dynamic configuration based on the network connection, or just warning when profile port ranges overlap would be helpful. If you add all my profiles to ufw the first thing you will notice with ufw-frontends is that it doesn't handle large numbers of profiles well. To help address the problem I've added a new parameter to the profile file format that hopefully ufw and ufw-frontends can utilize in the future. This is easy to do because INI files don't have much of a standard and ufw ignores everything other than the original ones. The parameter I added is "categories" for classifying profiles. This will allow users of ufw and related GUIs to quickly filter large profile lists. I put in a wishlist bug report about it for ufw. I didn't want to bother creating my own standard from scratch so I used the freedesktop.org menu spec categories since they're already used for organizing desktop menus. I had to break the standard a bit by mixing main categories, usually "Network" with "Game", but this shouldn't be a problem.

The second parameter I added was "reference". This was due to the ridiculous amount of research I had to go through in finding port numbers for each application. Multiple "reference" parameters can exist for each profile, each listing a one-line item. The references indicate the basis for the profile configuration, like "netstat -nap|grep python", to indicate how the port was determined. Mostly these are web site references with a link specified in [URL label] wiki format.

Obviously there are many more servers and daemons to add but generic ones like DirectX, GameSpy, and GGZ Gaming Zone cover many. This brings up a possible optimization - a "meta" or "prerequisite" parameter. Because a lot of games share the same ports due to underlying common code, it would be simpler to define a profile that simply links to other profiles to specify ports. This way a profile could be specified for every individual program but not add a lot of duplicate rules to keep track of.

I only encountered one problem with ufw's profile implementation. It happened when I created a profile for 0verkill. Apparently ufw doesn't allow section names to begin with a digit but I can't imagine why this limitation would exist. Besides that and the way ufw-frontends handles huge profile lists this firewall configuration works well. I don't know if all of the profiles are correct as I didn't have time to test everything. Some may open more ports than a game or application requires and some may not open enough. Feedback is welcome.

Note: The NFS profile (nfs-kernel-server) requires static port mapping. The references in the file will lead you to articles on how to configure NFS this way but I changed the common 4000:4003 ports to 4194:4197 as these aren't in Wikipedia's list or used by anything I could find with Google. There may be a Netfilter module that handles the NFS random port usage better as one exists for saned (nf_conntrack_sane) but I'm unaware of one.

Update: I updated the profiles package to v1.1 which includes a bunch more Linux games and some corrections. NFS profiles have been split into three different files representing the common address ranges. Using static ports for NFS is kind of a hack and I reported bug 688446 about possible solutions.

Update: I released v1.2 of the profiles which made some changes to http due to bug #694894 (which is mostly the fault of Debian and IANA). I also added another parameter, "modules", which specifies the connection tracking module (nf_conntrack) for some protocols. Currently in ufw-gtk some of these can be enabled under "Edit > Preferences > IPT Modules". Having a separate dialog for them doesn't make a lot of sense as they are specific to a particular protocol and you usually need them enabled. The modules act similarly to "port triggering" but are more intelligent as they understand the handshakes of their respective protocols and can identify which additional ports have been negotiated between server and client. I also found out that with NFS v4.1 that the dynamic port problems are being eliminated.

Update: I released v1.3 of the profiles. This is a minor release that only adds Skype, toribash, and webcam-server. The download link has been updated.

A script for auto-configuring saned network connections

Host-connected image scanners can be shared through saned (part of sane-utils in Ubuntu). It can be run continuously as a daemon or on-demand through Inetd. Basic configuration for either mode is simple and generic but adding the network address to the saned.conf file in CIDR notation is not. When you are setting up systems for multiple clients on different networks and IP ranges, this is a bit of a nuisance. To automate this I wrote saned-subnet-conf which will automatically add an entry for whatever network the host connects to through Network Manager or the ifupdown utilities directly.

Whenever a network connection is made or broken, scripts can be triggered. These scripts need to be located (or linked to) in "/etc/network" in specific subdirectories, the choice of which determines when they execute. Variables are passed to them that can be used for changing behavior based on the network interface, address assignment mode used (DHCP, static, ppp, etc.), and other values. See the interfaces man page for some hints. Network Manager executes these scripts with "/etc/NetworkManager/dispatcher.d/01ifupdown" which uses the run-parts utility. Network Manager does not trigger the "pre" directories due to a design decision.

To install the script, first download and extract the script, then put it in "/etc/network/if-up.d". You'll need to use sudo or have a root terminal for the copying (and most of the rest of the commands). Make it owned by root:root with rwxr-xr-x (0755) permissions. Whenever a network interface is brought up by ifup or Network Manager the script will execute. It uses scanimage to look for scanners and if any are found it will then use the ip command to get a CIDR version of the network address and produce an entry for saned.conf if one doesn't already exist. The last part is important as the script will add an entry for every network the host connects to. If you want to block a particular network, let the script add it to saned.conf and then comment the entry out with a # as the script won't add it again if it finds it anywhere in the file. Make sure you restart saned anytime you edit saned.conf (see below). If you want to keep the script from adding entries in relation to a particular network interface you'll have to edit the script and have it exit based on the IFACE variable. Look at the "$METHOD = loopback" entry for a rough idea. If you enable the VERBOSITY=1 entry the script will generate a log file in /tmp that includes all the variables. Currently the script only supports IPv4 addresses as my network doesn't use IPv6 so I can't test it.

Setting up saned is rather easy. During installation you have the option of running it as a daemon. To enable this later use "dpkg-reconfigure sane-utils" and indicate "Yes" to the standalone server option, or just edit the "/etc/default/saned" file and set "RUN=yes". The server daemon will start automatically at boot but you can start (or stop, restart) it manually with "invoke-rc.d saned start" or "/etc/init.d/saned start". To see any messages from saned use "tail /var/log/daemon.log".

To have saned start automatically when a client connects, indicate "No" to the standalone server option or set "RUN=no" in the default config file. Then add (as per the man page) the required entry to "/etc/inetd.conf" if it doesn't already exist. You can use a text editor but a safer way is with the update-inetd utility with "update-inetd --add "sane-port stream tcp nowait saned.saned /usr/sbin/saned saned". If you watch the log (tail -f -n 20 /var/log/daemon.log) you will see saned start and stop automatically whenever a client connects. Don't run a daemon and have an Inetd configuration active at the same time as they will conflict over network port access (6566 by default). To disable the Inetd entry use the command "update-inetd --disable sane-port".

To configure clients to use the server just add the server IP address or host/domain name to "/etc/sane.d/net.conf" and start whatever scanning program you want to use. You can get a list of available scanners with scanimage -L but note that neither saned or scanimage supports scanners connected via a parallel port.

On Ubuntu 10.04 (Lucid Lynx) and some earlier versions access to scanner devices isn't handled correctly for anyone other than standard users (UID=1000+) on the host. As a workaround you can use my Scanner Access Enabler to correct the permissions until reboot. In the future, scanner network access may be handled by Avahi but it doesn't work with Karmic or Lucid due to another bug.

Update: Forgot to mention that scanimage is used to look for scanners first before adding a saned.conf entry.

Scanner Access Enabler

There is a problem with scanner device permissions on Ubuntu. Regular users (UID>999) can access libsane applications like Xsane and Simple Scan without problems. Linux Scanner Server, which is running in Apache as www-data, can't access them without a chmod o+rw on each scanner device. Nobody seems to know how the permissions work so this has to be fixed manually in a terminal. This is not n00b friendly so I created a GUI application that automatically changes the permissions of every scanner device.
The application relies on scanimage and sane-find-scanner utilities to identify scanner device ports then simply does a chmod against all of them. It supports USB, SCSI, and optionally parallel port (-p parameter) scanners and has been tested against the same ones I used for my LSS patch. It uses the same universal dialog code as webcam-server-dialog so it should work with almost any desktop environment.
To install first download the archive and extract the contents. Move the script to "/usr/local/bin/scanner-access-enabler" and set it for root:root ownership with rwxr-xr-x (0755) permissions. Copy the destop menu entry to the /usr/local/share/applications directory with root:root ownership and rw-r--r-- (0644) permissions. You may have to edit the desktop file as it uses gksudo by default. On KDE you may want to change the Exec entry to use kdesudo instead. If you specify the -p option on the Exec line you may have to quote everything after gk/kdesudo. If you don't have one of the GUI dialoger utilities installed and plan on using dialog or whiptail then you need to set "Terminal=true" else you won't see anything.
On Ubuntu the menu item will be found under System > Administration. If you want users to be able to activate scanners without a password and admin group membership, you can add an exception to the end of "/etc/sudoers" file. Simply run "sudo visudo" and enter the following:
# Allow any user to fix SCSI scanner port device permissions
ALL ALL=NOPASSWD: /usr/local/bin/scanner-access-enabler *
While you can use any editor as root to change the file, visudo checks for syntax errors before saving as a mistake can disable sudo and prevent you from fixing it easily. If you mess it up, you can reboot and use Ubuntu recovery mode or a LiveCD to fix it.
Update: I released v1.1 which adds filtering for "net:" devices from saned connections. This didn't affect the permission changes but made for a crowded dialog with both the raw and net devices shown.
Update: v1.2 adds a non-interactive/silent mode activated through a "-s" parameter.
Update 20120708: v1.3 cleaned up the code a bit but broke detection completely as I recently noticed.  I updated it to v1.4 which actually works now. :D

A patch for Linux Scanner Server v1.2 Beta1

I just spent several days testing, fixing bugs, and adding features to Linux Scanner Server v1.2 Beta1. LSS is an easy way to share a non-networkable scanner through a web server. While the interface doesn't allow cropping like Xsane or Simple Scan it does support multiple file outputs, printing, and OCR through Tesseract. Development has stalled with the beta and I encountered some bugs when testing it on Ubuntu 10.04 (Lucid Lynx). Instead of complaining about it, I fixed them.

Bugs fixed/features added:
Noise in Apache logs caused by unquoted variables and non-critical stderr outputs from ls and rm.
Adding scanners would fail if the scanner name included a forward slash.
Multiple scanner support broken due to a lack of newlines between entries in the scanner.conf file.
No support for scanners connected via parallel-ports.

I wanted to try to break the beta before deploying it to my clients and I did - as soon as I connected a second scanner. I decided to fix the bug even though none of my clients have more than one attached to any given system. I just happen to have a bunch of them on hand and, thanks to a local tech recycling center, I added a few more. Sane supports scanners connected to parallel ports but LSS doesn't so I decided to fix that, well, just because. Yes - I went out and paid money for more scanners including an obsolete parallel port Mustek model just to fix LSS.

The deciding factor in doing this was that LSS is based on a shell script and a lot of sed scripts. Shells scripts are about the only programming language I know to any depth (and Applesoft BASIC). Some of the regex/sed stuff still throws me but I had help from some of my LUG mates. These are the scanners I tested with (and tested simultaneously):

AGFA SnapScan 1212U (snapscan:libusb:002:003)
Brother Industries MFC-440CN (brother2:bus4;dev1)
Hewlett-Packard ScanJet ADF (C7190A, identified as 5200C) (hp:libusb:004:002)
Hewlett-Packard ScanJet 4470c (rts8891:libusb:004:003)
Hewlett-Packard ScanJet 6100c (C6260A but identified as C2520A) (hp:/dev/sg5)
Microtek ScanMaker E3 (microtek:/dev/sg3)
Mustek 600 III EP Plus (/dev/parport0)
UMAX Vista-S8 (umax:/dev/sg4)

Fixing the multiple scanner support was a pain. LSS relies on scanimage for all scanner functions. Getting scanimage to provide a newline at the end of the device list was trivial but the message printing function for the web page doesn't tolerate them and they all have to be converted to HTML breaks. Forward slashes in the model names from scanimage also required escaping but not anywhere else (like in the device paths). This got into sed loops which are really hard to do.

Adding support for scanners on parallel ports was also difficult. They have to be defined manually in the sane config files (/etc/sane.d/*.conf) but scanimage doesn't report them regardless. The sane-find-scanner utility does find them and will indicate what brand is on which port but no additional details like the model name. Since sane can use auto-probing to find which parallel port the scanner is on there is no deterministic way to use the information from sane-find-scanner and the sane conf files to indicate a specific model. The only solution I could come up with is to manually specify parallel port scanners in a separate "scan/config/manual_scanners.conf" file and then merge it after the rest are detected. The format is the same as for scanners.conf but the value for ID needs to be specified as %i (same as the device entry in the format line for scanimage). The modified LSS index.cgi script will replace it with an auto-incremented value when merging. The NAME= value doesn't matter but forward slashes have to be escaped with backslashes and anything longer than 30 characters will be truncated in the pull-down list on the Scan Image page.

Setting up a parallel port scanner is a bit confusing. The Mustek model I used was configured in /etc/sane.d/mustek_pp.conf simply by uncommenting the line "scanner Mustek-600-IIIEP * ccd300". The second parameter is the name. The third is the port with an * indicating autoprobing which in my case became /dev/parport0. The last is the actual driver. With scanimage the device is not specified by the port but rather the backend driver and then the name. With the settings I used it became "mustek_pp:Mustek-600-IIIEP" (also specified for the "DEVICE=" value in manual_scanners.conf). If only the backend is specified scanimage will default to whichever is enabled in the conf file. I only have the one parallel port scanner (the ScanJet 4470c has USB and parallel but there's no driver for the latter) so I don't know how it handles multiple ones configured in the same file/backend.

There are still bugs in LSS. The most obvious one is a fault with the "Print_Message" function. There are several page updates that don't occur, mostly the "Please wait" ones that are supposed to display during scanner detection and image scanning. I don't know enough about the interaction between javascript and the browser to identify if it is a bug in the code or an architectural problem with the page design.

Another bug is with the scanner names. As you can see from my list above, some of the scanners are not named correctly. It may be that the model reported is the base one that the actual model is compatible with and sane just isn't more specific than that. LSS just uses whatever scanimage reports. This isn't a major problem as most systems will only have one scanner.

A third problem is with the scanner driver options that LSS specifies - basically none. Some scanner/driver combinations require specific options to be specified else the scanner has problems. The only one I encountered with the models tested was that the default resolution of 200 was unacceptable to one of them so it was downgraded to 150. These errors show up in the Apache logs (/var/log/apache2/error.log) but only refer to index.cgi and not a specific point within the file. I'm not sure how this bug could be fixed. Parsing the options out of the sane conf files may work but different versions of the same base model may require different settings.

Future enhancements that would be nice are cropping and internationalization support but that's more than I'm going to take on. My LUG mates also suggested using anything other than shell scripts.

To use the patch, first download and extract LSS into your web server data directory (/var/www/scan). Then download the patch archive, extract it into the "/var/www" directory and apply with

patch -p1 --directory /var/www/scan --input=/var/www/scan_1.2_Beta4.patch

Just reload any browser window that has the old version loaded to make the new one active. Restarting the server is not necessary.

LSS is GPL 2 but it's not clear in the package as the author didn't follow the recommended method for applying the terms.

Note - there's a nagging problem with Ubuntu in that LSS can't access any scanners due to device permissions. It runs as user www-data but the old scanner group no longer exists so devices need chmod o+rw applied manually. For regular users (UID >1000) it seems to happen automatically but nobody seems to know how that works. I wrote Scanner Access Enabler to solve the problem.

Update: If you also have saned configured for scanner sharing then duplicates may be detected from both the raw devices and saned shared versions. If you don't want the ones from saned then comment out "localhost" in the "/etc/sane.d/net.conf" file and restart saned.

Update: I and pqwoerituytrueiwoq have been making more improvements to the beta. You can follow along and download updated files from this thread at the Ubuntu forums.

20110207 Update: I and pqwoerituytrueiwoq made a bunch of fixes and I've released 1.2 Beta 4 of the patch. The links and instructions above have been updated. It is a recursive patch so it will affect several files. You still need to add the favicon for it to "/var/www/scan/inc/images". I performed a feasibility study of adding proper preview, settings selection, and cropping. I found that the difficulty of adding them to the existing code base is extreme, even though some are needed to get LSS functioning correctly. For example, the Brother MFC-44CN doesn't scan because the modes that LSS uses (like "Color") are hard-coded in the html and don't match up with what the Brother driver offers. Because of these problems (and my lack of time) I've ended my involvement with the project. For my needs Beta 4 functions adequately. I also found another scanner project, phpSANE, that seems to have a better code base on php although it has many limitations otherwise.

webcam-server-dialog: A basic front-end to webcam-server

I'm working on a new Ubuntu configuration to deploy for friends and family. One of the capabilities I wanted to add was simple remote webcam viewing. I saw an article about webcam_server. While it's old and only does images it met my requirements. The primary limitation is that it's a command-line application. It can be launched from a XDG desktop file and will default to /dev/video0 but on systems with more than one video device a terminal is needed. I did spend some time testing streaming with VLC but it doesn't show a device list for selection either and it's streaming configuration dialogs are confusing at best. To deploy webcam-server I had to make it more friendly which meant making a video device selection dialog for it. My current programming hammer is Bash shell scripting (Applesoft BASIC was the other option) but it's not enough for GUI design. To add that capability I turned to what I call "dialoger" utilities that can produce GUI dialogs and provide feedback to command-line applications.

When I started this project the only dialoger I knew about was dialog which is text-based and I needed something that would run in X. From researching alternatives to dialog I found xmessage which led to gxmessage and eventually Zenity. Plenty to choose from and all different. Now I had another problem - which one to use with each desktop environment? I normally use Gnome but some of my clients use XFCE or LXDE. There is also the possibility of a KDE user in the future. While Ubuntu includes Zenity by default, which would work with XFCE also, a GTK application isn't the best choice on KDE. I could use kdialog but either I had to make custom versions of the script for each environment, select the dialoger with a script parameter, or try to select it dynamically. In the great tradition of overdesign I chose the latter.

After spending several days solving a 5-line problem with 300+ I ended up with webcam-server-dialog. It supports dialog, whiptail, Xdialog, xmessage, gxmessage, kdialog, and Zentiy. Because it's rather generic it can be expanded to support more without a lot of effort. It looks for processes that indicate a particular desktop environment or window manager, uses a built-in priority list of dialogers for that environment, checks for availability, then uses the best available to provide the GUI. The core of this script is nothing more than "ls /dev/video*" dumped into an array but the focus here was eye-candy. Getting this to work with all of them was difficult as they all have different command-line parameters, behaviors, and bugs, even between those that are supposed to be clones of each other. Some use exit return status for indicating button presses, some stdout, some both depending on the mode. The fun ones were dialog and whiptail which use stdout for drawing the screen and stderr for indicating list choices. There are a lot of comments and commented-out debug lines if you want to use this with your own projects (GPL v3). It could be useful as a front-end to other programs that lack selection lists like xawtv.

To use it, install webcam-server then put the script in /usr/local/bin with root ownership and rwxr-xr-x permissions. I also made an XDG destop menu item for it. Put webcam-server-dialog.desktop in /usr/local/share/applications. For an icon I used the one from camorama and just copied /usr/share/pixmaps/camorama.png to /usr/local/share/pixmaps/webcam-server-dialog.png (make the directory if it doesn't exist). You can add parameters to the Exec line in the desktop file but the run dialogs always reference port 8888 (I'm too tired of working on this to make the text dynamic) and quotes don't pass through well so the caption format had to be hard-coded in the script.

In addition to direct web-page access a Java applet is included. It works well since it can automatically reload the image at a selectable interval. It has one really bizarre limitation - it will only connect from localhost. Apparently the developers decided to limit it that way instead of implementing remote authentication but you probably could use it from within an ssh connection. This security feature has a bug in it's implementation when resolving the hostname so it requires the IP (127.0.0.1). I created a custom PHP web page that works around this. Just rename it to index.php and put it in the client directory on your web server (/var/www/client by default) along with applet.jar from the /usr/share/doc/webcam-server directory and link to it from your default home page. You'll need PHP support installed (php5 metapackage).

I also tested it in Mandriva 2010.0 which worked but with one problem - the webcam-server executable is named webcam_server so just replace all instances of one with the other in the script. You'll also need v4l-info which is in the xawtv-common package. If you are using the firewall you have to add "8888/tcp" to the allowed ports for remote access.