Category: Linux

Posted on

Best practice tips for Gentoo sysadmins

Currently, there are some critical ebuild dependency issues in Gentoo’s portage tree that might seriously hurt your box. Symptoms: When updating your system, portage might display an error message similar to this one:

[ebuild     U ] sys-fs/e2fsprogs-1.41.2 [1.40.9] USE=”nls (-static%)” 4,263 kB
[ebuild  N    ] sys-libs/e2fsprogs-libs-1.41.2  USE=”nls” 479 kB
[blocks B     ] sys-libs/ss (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] <sys-fs/e2fsprogs-1.41 (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] sys-libs/com_err (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] sys-libs/e2fsprogs-libs (is blocking sys-libs/ss-1.40.9, sys-libs/com_err-1.40.9)

The important thing: DON’T unmerge ss or com_err, as it will break wget and other essential parts of your system! Portage thus won’t be able to download e2fsprogs-libs-1.41.2 which is required to replace the removed ss and com_err libraries (which are part of e2fsprogs-libs starting with v1.41.2).

Solution: Either wait until this issue gets resolved by the Gentoo core dev team or read through the following posts and Gentoo bug reports:

If you really know what you do, you might want to try this suggested quickfix (Important disclaimer: Looks reasonable and fine as a quick workaround to me, but I haven’t tried it yet. You apply it at your own risk, as usual! Note that this workaround doesn’t solve the real problem.)

As serious issues like these are quite common in Gentoo, here are some best practice tips for Gentoo sysadmins that help prevent some of the potential problems:

  • First of all, try to use stable ebuilds only. If this is not possible for some reason, try to minimize the number of unstable ebuilds (~amd64 etc.) on your system.
  • It’s rather tempting, but DON’T setup a cronjob to do automatic emerges! Portage only catches the most evident issues, but emerging new ebuilds is never without risks (not updating your system is risky too, however). The best approach would be testing any updates on a test box first before installing them on a production system. The second best approach is probably doing a monitored, manual update in small, incremental steps with immediate testing afterwards. This helps isolating problems, should they occur (it’s difficult to isolate a problem that was detected after an automatic update of hundreds of ebuilds).
  • Automate ’emerge –sync’ by putting it in your daily crontab in order to refresh your portage tree regularly. That’s neither particularly safe nor unsafe, but it guarantees that you don’t emerge that weeks-or-months-old broken ebuild that has been fixed in the meantime.
  • Regularly fetch new source packages by setting up a cronjob for ’emerge -uDN –fetchonly world’ (or -f). Like this, portage uses some additional hard disk space for the package sources (always make sure you have enough free space and properly setup partitions/volumes!). It makes sense though as one day, you’ll use most of these source packages anyway and having a source package locally can be very helpful in a situation like the one described in this post. IOW: If you aren’t able to download anything anymore due to a severely broken system, chances are, that you can still solve the problem on localhost, if you have source packages at hand.
  • Append “buildpkg” to the FEATURES variable in /etc/make.conf. Like this, portage will additionally create binary packages in /usr/portage/packages/All when emerging new ebuilds. This will require some spare free space on your hard disk again, but having a prebuilt, binary package at hand can be very helpful if there are any problems with the gcc toolchain or any other compiler chain needed. If you don’t like to enable this feature permanently, you can use the -b or –buildpkg option when executing emerge.
  • If you haven’t used the “buildpkg” feature so far, you can create binary packages of all the installed ebuilds on your system using the “quickpkg” utility and my quickpkg_all bash script.
  • Keep old, compiled kernel images in /boot and listed in your /boot/grub/menu.lst. Booting a new manually configured and compiled kernel is always a bit of an adventure (unless it was tested on an identical box before), and it’s good to keep previous kernels that are known to work. Even if it doesn’t work perfectly, it can take you to a console login prompt at least.
  • Instead of doing things the regular “remove old packages first, then install new packages” way, get used to the Gentoo way of doing things: “install new packages first, then remove old packages (if at all)”. Avoids serious problems that can occur when accidentally deinstalling an old, seemingly no longer used package that other important packages depend on and don’t work without.
  • When merging new configuration files, use dispatch-conf instead of etc-update. dispatch-conf uses CVS to create backups of old config files (which can be a helpful source of information in some situations). See the value of the “archive-dir” setting in /etc/dispatch-conf.conf.
  • Add files and directories to CONFIG_PROTECT, if in doubt. It’s better to have one ._cfg0000_XXX file too much than an important configuration file accidentally overwritten by portage.
  • Use emerge’s -D option for improved (deep) dependency checking.
  • Do regularly use revdep-rebuild to check for broken dependencies and to remerge the according ebuilds.
  • And of course, create automated, incremental backups of your systems regularly. You’ll sleep better, believe me ;)
  • Monitor your systems for errors. I do it with some custom bash scripts I wrote, but there are many full-fledged monitoring solutions for general purpose health monitoring.
  • As a fallback for (some) DNS problems with DHCP-based systems, I regularly send a heartbeat of a DHCP system to a box in another network, revealing the DHCP system’s last known/assigned IP address.
  • Not limited to Gentoo sysadmins: Having a (hardware) remote console accessible via a different IP address is worth a lot in case there are serious troubles with the operating system or the hardware.

Feel free to add other helpful best practice tips for Gentoo sysadmins!

Posted on

Google Chrome from a business and “techie” view

If Google will really deliver what it promises with its new Chrome browser plans (Google blog) (personally, I have no doubts about this), the line between web applications and standalone applications will further blur and hereby enable a better, seamless user experience and probably a whole new class of powerful applications.

Some thoughts:

  • From a technical point of view, Google’s Chrome will be the WebOS others have been dreaming about for a long time already. It basically offers memory management, process management, markup renderers, a GUI and a VM with a JIT compiler (V8).
  • It will finally unify the ideas behind the WebOS, “The network is the computer”, cloud computing, SaaS, RIA and virtualization.
  • Actually, it’s astonishing it took so long for someone to come up with something like this. The concepts as such are not new at all, but the combination of all these different concepts is what makes the thing cool. It’s typical for a good idea that, once you’ve heard of it, you almost can’t imagine living without it anymore, as it seems all so natural.
  • Detachable tabs on top: Not a new idea either. For example, I remember that the Fluxbox window manager actually offered the same feature back in 2001/2002 (or even earlier) already. I remember it as I used it myself too (and I liked it a lot, despite of its “suboptimal” scalability), as illustrated in these animations:
    Fluxbox Window Grouping Feature (2002) 1/2 (small animated GIF screenshot)
    Fluxbox Window Grouping Feature (2002) 2/2 (large animated GIF screenshot)  

    I guess there were other window managers and GUIs that had the same features even before fluxbox had them.

  • With this move, Google will be gradually taking control and power away from traditional Desktop OS manufacturers such as Microsoft and Apple. Being open source, Chrome and its components like V8 will be the “Linux of the web” and thus a big threat particularly to Microsoft that still generates most of its revenue with Windows and standalone applications like Office.
  • The ongoing process, that (desktop) operating systems are becoming commodities more and more, will further be accelerated. Will there be an “unsacred” alliance between Apple and Microsoft to fight these tendencies or will they shift their businesses further into the “web” application (SaaS), content (music, videos, TV, e-books, multimedia etc.) and lifestyle (design, hardware, ethics) spaces?
  • Of course that’s in the best interest of Google (as their business is data/content and webapps/SaaS). I wouldn’t call this move an evil move, but it’s definitely not a friendly move in the eyes of the competition.
  • From a “techie” point of view, this move will enable many interesting applications in the future. As the framework will be open source, the dev community will potentially be as vital and dynamic as in other high-profile OSS projects (like Mozilla, Linux)
  • What about the Mozilla, Safari, IE, Opera camps? They will have to adapt themselves to the concept and try to top it. IE (and perhaps also Safari) might try to take the “embrace and extend” route.
  • With the birth of the WebOS, there will probably be a need of an open, standardized webapp GUI toolkit and webapp GUI guidelines soon (and there’s a big potential for conflicts here). Who will provide these? What will be the roles of the current big players? Also, standardized, open specs for user authentication and user data exchange will be required – here, there’s already some progress with OpenID, OAuth etc.
  • I like that Google communicates its plans using an easy-to-follow cartoon and that they give credit to individual internal and external contributors and players (though I assume there were much more people involved in the process than those mentioned)
  • The thing that disappoints me a bit is that when talking about V8, they only talk about targeting JavaScript. I’d prefer a more generic approach providing a VM and JIT for various languages (similarly to a CLI VM – why not re-use/extend Mono, for example?). Maybe that’s what V8 actually provides and they just don’t emphasize it at this point in order to not confuse or upset end-users, devs, big players etc..
  • Taking a look at the big picture, it seems that there’s a very pragmatic driver behind this whole development: It’s the laziness of us end-users (just as a fact, not meant in a negative sense – being “lazy” is usually quite rational). Or in other words: The information takes the line of the least resistance. And so far, that line for the “Network OS” happens to be the web, i.e. basically HTTP, despite of its known shortcomings.

[UPDATE 20080902: Corrected a typo. And here’s a statement regarding the Google Chrome news by John Lilly, CEO of Mozilla Corp.]

Posted on

Setting up Procmail and Postfix with maildir for mobile usage

When I’m on the road, I can check my private and business e-mail using my very handy Nokia e61i smartphone (which I like a lot, apart from its rather unstable web browser). However, as I’ve never been far away from a computer for a long time so far, I’ve only used to check mail that was delivered to my inbox, and not to any of its about 200 subfolders (I once tried to subscribe to all folders – the mail client of the e61i simply crashed ;). Now, I figured I’d need to do something about this in order to not miss any of the mails that are automatically moved to a subfolder by the server-side procmail mail filter.

The solution: Simply create a new procmail rule that copies (that’s what the c stands for) any incoming message to a special folder (I named it “mobile”; note that my postfix uses the maildir format to store messages):

:0 c
.mobile/

Preferably, this rule should be placed after the spam filtering rules (you don’t want to pay for spam on your mobile, do you? ;) and before the normal procmail mail filter rules (which move every incoming mail to the appropriate subfolder). You can also use a dedicated rules file for every group of similar rules. For example, I include the files spam.rc, mobile.rc, lists.rc (with their according rules) in the main .procmailrc config file.

Now every mail that wasn’t filtered by the spam filter gets copied to the “mobile” subfolder of your inbox prior to getting moved to the appropriate subfolder (by the rules in lists.rc).

On my e61i, I hence only need to subscribe to the “mobile” subfolder in order to catch all incoming messages. As you can’t unsubscribe from the inbox subfolder itself, you should set the maximum number of caught new inbox mails to 1 (which is the minimum) in order to avoid fetching things twice. You can further configure that only headers are fetched to minimize the amount of data[1] that is automatically downloaded with every mail check (also: switch off polling if you don’t need it).

Note that another advantage of this solution is that you won’t tamper with any of the original messages (that are either in your inbox or in any of its subfolders). Some might call this a disadvantage though.

That’s it, have fun! :)

[1] I pay 0.10 CHF per 20 KB, i.e. 5 CHF (5 USD) per MB (which is incredibly expensive as I currently don’t have any flat rate data plan -> please don’t send me a mail bomb just for fun, else I’ll have to write a more sophisticated procmail rule that only copies messages < 100 KB to the mobile folder)

Posted on

Gentoo: Emerge errors after a Python upgrade?

If you encounter seemingly “strange” Python errors when emerging apps after a Python upgrade, execute:

# python-updater -v

This will rebuild packages that are broken due to the Python upgrade (Note: You can also manually emerge the packages python-updater lists)

Posted on

Gentoo: Troubles with Trac and mod_python-3.3.1?

If Trac doesn’t run anymore after an ’emerge -uD world’ (i.e. there’s a “500 Internal Server Error”) and Apache complains about “ImportError: No module named trac.web.modpython_frontend”, here’s the solution:

Gentoo Bug 230211 – www-apache/mod_python-3.3.1 – connobject.c:142: error: request for member ‘next’ in something not a structure or union

(IOW: Emerge mod_python-3.3.1-r1 from your local portage overlay and restart Apache)

Posted on

Gentoo: Pebble 2.3.1 on Tomcat 5.5 and Tomcat 6 (using JDK 1.5)

In Gentoo, a couple of manual configuration steps are required in order to make Pebble run on Tomcat 5.5, using JDK 1.5. So, apart from the obvious (like emerging Tomcat, a JDK, fetching pebble-2.3.1.zip etc.), I had to do the following:

  • In /usr/share/tomcat-5.5/lib, add the following two jar libraries from the pebble-2.3.1.zip archive (they’re located in the lib subdirectory):
    activation.jar
    mail.jar
    Further create symlinks to these two jars in /usr/share/tomcat-5.5/server/lib:
    o2 # cd /usr/share/tomcat-5.5/server/lib
    o2 lib # ln -s ../../lib/activation.jar .
    o2 lib # ln -s ../../lib/mail.jar .
  • Make sure these two jar files are in Tomcat’s CLASSPATH. Astonishingly, placing them in the above directories is not sufficient in Gentoo, one needs to explicitly add them to the CLASSPATH too. Thus, in /etc/conf.d/tomcat-5.5, edit the CLASSPATH to make it look as follows:
    CLASSPATH=${CATALINA_LIBDIR}:${CATALINA_LIBDIR}mail.jar:${CATALINA_LIBDIR}activation.jar
  • In /usr/share/tomcat-5.5/common/endorsed, add symlinks to xalan.jar and serializer.jar (in order to get rid of the “javax.xml.transform.TransformerFactoryConfigurationError: Provider org.apache.xalan.processor.TransformerFactoryImpl not found” error message):
    o2 # cd /usr/share/tomcat-5.5/common/endorsed
    o2 endorsed # ln -s /usr/share/xalan/lib/xalan.jar .
    o2 endorsed # ln -s /usr/share/xalan/lib/serializer.jar .
    These two files are part of Xalan (’emerge xalan’, if necessary)
  • On my Gentoo server, I had to explicitly set the dataDirectory property Pebble uses, as by default, the variable ${user.home} seemed to point to /dev/null in a secured Tomcat environment. So, make sure the property file ./WEB-INF/pebble.properties in the pebble.war file defines
    dataDirectory=/var/pebbledata
    (or whatever directory you want pebble to store the actual blog data in) instead of
    dataDirectory=${user.home}/pebble
    Important: Make sure this directory (here: /var/pebbledata) exists (create it, if necessary) and is writable by Tomcat (i.e. user tomcat or group tomcat)
  • I’d suggest you rename pebble.war (from the pebble-2.3.1.zip archive) to something more generic, e.g. blog.war or news.war before deploying it to Tomcat. Like this, the relative URL for accessing the blog will automatically start with /blog or /news accordingly (instead of /pebble). The naming is a matter of taste though. To deploy the war file, simply copy it to /var/lib/tomcat-5.5/webapps and restart Tomcat (which might not even be necessary, actually):
    o2 # /etc/init.d/tomcat-5.5 restart
  • If you want to use the Tomcat Manager web application in order to monitor, start and stop webapps like pebble (you probably want to do this), make sure you define a user with the role ”manager” in /var/lib/tomcat-5.5/conf/tomcat-users.xml (this file is empty by default). E.g. add the following user:
    <tomcat-users>
    <role rolename=”manager”/>
    <user username=”tomcatmanager” password=”t0tallySecretPassw0rd” roles=”manager”/>
    </tomcat-users>

For Tomcat 6, the required changes are analogous. ASAP, I’ll verify these steps with Pebble-2.3.1 on Tomcat 6 using Sun JDK 1.6. Stay tuned..

Posted on

Setting up postfix for virtual mail hosts

After setting up name-based virtual web hosting for Apache, I’ve just set up virtual mail hosting for my new company, Printscreen GmbH, using postfix – and it was amazingly simple! :) For some basic virtual mail hosting using normal UNIX/Linux user accounts, all you need is a plain-text file (‘/etc/postfix/virtual’) that maps virtual mail addresses to the actual user accounts and two additional lines in main.cf:

/etc/postfix/virtual:
postmaster@example.com postmaster
# Uncomment entry below to implement a catch-all address
# @example.com jim

/etc/postfix/main.cf:
virtual_alias_domains = example.com
virtual_alias_maps = hash:/etc/postfix/virtual

Then just execute

# postmap /etc/postfix/virtual && /etc/init.d/postfix reload

Voilà!

The nice postfix readme file for this and some more complicated virtual mail hosting configurations:

http://www.postfix.org/VIRTUAL_README.html

Another, possibly helpful howto (search):

http://www.freebsddiary.org/postfix.php

For setups using a MySQL database as a backend for storing the mappings, see Google. I currently don’t need this and hence don’t like to introduce any unnecessary MySQL dependency to my postfix server – remember the KISS principle ;)